Are buffer overflows acceptable from a graduate developer? Are we setting the bar too high? What are the expected capabilities of graduate/junior engineers?
We are currently recruiting for a Junior Developer position working mainly in C on Linux.
As part of the process, we require candidates to complete a code test at their leisure in C.
So far we have rejected two candidates on the basis that their code, although readable and in one case rather idiomatic, suffered from buffer overflow errors due to unbounded buffer writes.
- We explicitly ask for error-checked, production quality code.
- We provide a test & build framework for the candidates
As a result of this thread, and conversations we have had with other developers in person, we are changing the way we carry out code tests and who we target with our recruiting.
We decided that a candidate being unable to fix or understand a buffer overflow means that he would be unsuitable for the work we carry out, in particular he would take more mentoring than we are comfortable with. We will therefore still reject candidates that cannot eventually submit a robust code sample.
However, we have put in place some measures to make the recruitment process more productive for both us and the candidates.
- We make our expectation more explicit, with clear explanation of what we mean by production quality, and a warning that the code is expected to be robust with respect to input and errors.
- We now link candidates to resources on defensive programming and the C standard library in the description of the code test.
- We changed our target audience from Junior developers and graduates to target people with some relevant experience.
- In case the submitted code fails in some way but would otherwise be accepted, we now provide a minimum test case that causes the error condition and give the candidates a chance to correct their mistakes (unless the code is rejected for some other reason). We'll also point out problematic lines/functions if appropriate.
- The goal of the tests itself has now slightly changed from a front end filter to a chance to build a better picture of the candidate, in particular it will inform our phone discussion. That said, we are still willing to reject based solely on code.
[Update 2015-07-09]: Andy Davis from Nujob has written an interesting and relevant article on the use of a code test from the candidate's perspective, and the article is worth looking at. Find it here.