There are wide and long discussions on how you can test programmers who apply for a job with you/in your company. Most of developers have some experience on that (because they apply for jobs themselves), few of them are actually on “the other side” and have to somehow pick the best from the best. The most skilled from skilled.
I am not going to elaborate on what is better, to freely talk to candidates, to order them to write code online, write code offline or to let them answer a dozen of specific questions to score points for correct answers. All these have their pros and cons and, based on my own experience and from what others say, it all depends on the specific profile, the size of the company, the kind of the job, the amount of candidates etc.
What works best for me (since I am responsible for the technical part of the recruitment procedure) is to ask candidates to write a simple but complete application focused on a precise specification. It could be a simple SQL query generator for POCO classes or a core of an IoC container. It does not really matter whether or not a candidate recognizes the notions from the specification, for example people implement perfect IoC containers based solely on my specs with no prior knowledge of what an IoC container is. They are allowed to work offline for few days (although for someone experienced the implementation should not take longer than few hours) and then we meet to discuss the details of the implementation. This is where I ask every candidate to write something fast, on demand, using only a pen and a sheet of paper. This could be “a method to sum an array of ints” or “a method to reverse a string”. People usually do some subtle mistakes so I test how they adopt to “I am a test engine and your implementation seems to throw NullReferenceException which is not expected, please correct this”.
After both these tests I know that:
a) the candidate can read, understand and follow a written specification even if he/she does not recognize some notions
b) the candidate can work under a pressure and knows the programming language well enough to implement a simple algorithm without a compiler at hand (from which I conclude that the real knowledge of the language, with the compiler at hand, is much deeper)
For some time, however, I am thinking of making some changes to the way the process is organized. Instead of a pen, sheet of paper and a consise, short specs of a tiny method, I am thinking of a virtual web-based environment, containing a text editor and a compiler so that the candidate could follow a more complicated specification, write, compile and test the code he/she submits as the proposed solution.
There are two great examples of how this could look like:
1. Codility. It’s a web-based environment where candidates are presented with a concise and precise specs of an algorithmic task and they can invoke a compiler (which sits on the server) to actually compile and test the solution. The engine performs some tests on the server-side so it not only detects that the implementation misses the specification (I wonder if Codility tests are hard-coded so that it’s possible to write a code which only solves the test cases by hardcoding the answers for test cases?) but also detects the complexity of the solution (so for example if a linear solution is expected, you get a lower score for a solution with quadratic complexity).
2. Pex for Fun. It’s actually pretty smart. Pex is an engine which follows all the control flow paths in given code and generates a complete set of test cases. Based on that, Pex for Fun has a “secret” implementation of a simple method (like “take x and return x * x”) but you are not allowed to see the implementation. Instead, you have to provide your own implementation for which the engine generates a set of test cases and feeds the secret implementation with it (and vice versa). You are then presented with a list of test cases both your and secret implementations and if there are some differences in the output, you have to change your implementation and follow the validation procedure until your implementation is semantically equivalent to the secret implementation (all test cases produce the same output for your and secret implementations). Beside being extremely fun, this one not only checks if someone writes a correct code but also – to what extent he/she can adopt to changing conditions (the answers of the Pex comparing engine change each time you ask it for comparision, as you change your implementation).
While implementing a custom solution which somehow follows the idea from Codility seems easy, I wonder if presenting “Pex for Fun”-style task for a candidate would be as fun for a candidate as it surely would be for me watching people who try to solve such puzzles :)
