What standards can be expected from graduate/junior engineers? [closed]

Question

Are buffer overflows acceptable from a graduate developer? Are we setting the bar too high? What are the expected capabilities of graduate/junior engineers?

Context:

We are currently recruiting for a Junior Developer position working mainly in C on Linux.

As part of the process, we require candidates to complete a code test at their leisure in C.

So far we have rejected two candidates on the basis that their code, although readable and in one case rather idiomatic, suffered from buffer overflow errors due to unbounded buffer writes.

[Edit]:

• We explicitly ask for error-checked, production quality code.
• We provide a test & build framework for the candidates

[Update]:

As a result of this thread, and conversations we have had with other developers in person, we are changing the way we carry out code tests and who we target with our recruiting.

We decided that a candidate being unable to fix or understand a buffer overflow means that he would be unsuitable for the work we carry out, in particular he would take more mentoring than we are comfortable with. We will therefore still reject candidates that cannot eventually submit a robust code sample.

However, we have put in place some measures to make the recruitment process more productive for both us and the candidates.

In particular:

• We make our expectation more explicit, with clear explanation of what we mean by production quality, and a warning that the code is expected to be robust with respect to input and errors.
• We now link candidates to resources on defensive programming and the C standard library in the description of the code test.
• We changed our target audience from Junior developers and graduates to target people with some relevant experience.
• In case the submitted code fails in some way but would otherwise be accepted, we now provide a minimum test case that causes the error condition and give the candidates a chance to correct their mistakes (unless the code is rejected for some other reason). We'll also point out problematic lines/functions if appropriate.
• The goal of the tests itself has now slightly changed from a front end filter to a chance to build a better picture of the candidate, in particular it will inform our phone discussion. That said, we are still willing to reject based solely on code.

[Update 2015-07-09]: Andy Davis from Nujob has written an interesting and relevant article on the use of a code test from the candidate's perspective, and the article is worth looking at. Find it here.

Answer 1

I don't think you've set the bar too high, I think you might need a different bar.

I think code tests are useful for determining the competency of a candidate, but they shouldn't be pass/fail. You should use the results of the code test to start a dialog with the candidate.

If you see mistakes that they've made (especially if they're junior developers) point them out and ask them what they'd do differently or if they understand why there is a problem.

Answer 2

I think the junior qualifier is what makes all the difference here. Juniors shouldn't be tested for competence, they should be tested for learning ability, curiosity, passion, ethic, and definitely humility. The assumption with a junior should be that they are not competent, it's your job as the senior to make them so.

Obviously they should be able to write basic code like fizzbuzz and have a general knowledge of concepts; if you pointed it out to them and they didn't even know what a buffer overflow was, then I would say it's a no go, but I don't expect a junior to write more than 5 lines of code without a bug.

The day you trust your junior's competence is the day yours should be questioned, juniors should be treated with a lot of mentoring and a healthy dose of "trust but verify". I was a junior once, and to all those who were there at the time: I'm sorry. Remember the terrible things you did as a junior? (Please don't tell me it was just me; you'll give me a complex..)

Answer 3

I see a few issues here.

The first is assuming that an average computer science graduate knows, well, anything. They don't. Quite frankly, I'm pleasantly surprised when I see a computer science graduate that knows how to install and setup Visual Studio. Heck, I recently worked with a guy claiming to have more than five years experience on the Microsoft stack writing .NET code that couldn't figure out what TFS was or how to connect.

The second is your very limited pool. We also have candidates do a programming test. There are five separate "programs" they have to write. They do it at home and ship the code over. The tests are extremely simple (no database, no external dependencies) and can easily be done using the Express version of Visual Studio. The tests themselves are easily completed by a senior guy in about 30 minutes. Note that we generally only do these for recent graduates on up to three years of verifiable work experience.

What we've quantified is that around 70% of those given the test simply never get back to us. Roughly 15% turn in stuff that won't compile, usually due to blatant syntax errors (for example, missing ;). Another 10% compiles but fails to perform the required actions.

This leaves a whopping 5%. At this point, we aren't even considering conditions such as entering an alpha character as input when a numeric one is required. It's purely on given a very limited set of X as input the application does the appropriate output. Also, these numbers come from about 500 candidates over the past four years: we kept statistics because we wanted to know.

If we were to look more at code structure and defensive coding techniques such as properly disposing unmanaged resources or usage of try .. catch statements then almost no one would pass.

The question, of course, is why?

Why can't a kid with a degree in this field from a four year university accomplish what are simple programming tasks? The answer is that colleges are completely out of touch with business needs and lag many years behind what we consider state of the art. 10 years ago coding standards were such that security was something you did after the fact; and unit tests weren't even in vogue yet. Whereas today security better be at the forefront of your mind with every feature or enhancement. Just remember: most professors have either never actually worked in this field OR haven't worked in it for a long time. Once you know that, then you start to understand why they are so far behind. Worse, some of those professors spend too much time on a particular technology (Java, PHP, whatever) and fail to discuss serious foundational issues such as code structure or acceptable approaches (and WHY!).

Just a side example. A recent graduate was telling me about some of his experience writing a mobile OS for one of his classes, but he couldn't explain, even in basic terms, how a web server worked. He simply didn't know. 15 or 20 years ago was probably the right time for understanding how to make an OS. Today... not so much. Yet this was a required class when a class on defensive programming would be so much more useful to them AND the outside world.

So, what do we do?

Out of those 5%, we will interview a bit further to get an idea of their personality and fit. Then we pick the "best" ones in full knowledge that we are going to spend around six months "reprogramming" them to get rid of the crap their professors filled them with.

Answer 4

I think you are looking at the problem the wrong way. What you should ask yourself is this: What do we require from a candidate for them to be able to do the job? You should properly evaluate the position and what it entails. Below are some suggestions on when to hire a junior developer and when not to.

When to hire a junior developer: - If there is an overflow of easy work to be done, which would be a waste of time for a more senior developer. - If you are willing to mentor and train this person over the next few years. - If you are trying to grow the company and want someone who will stay for a long time. A junior developer who stays for only a year would be a waste of company resources, they would do little to produce anything, and most of the results would be in their own personal growth. - When you feel like spending money on someone's growth. Since again, most of the benefits will be what they learn.

When not to hire a junior developer. - When the work is too complicated. In this case it is just out of their league. - When you want to save money. A senior developer should complete the same tasks in a fraction of the time with better quality results, and thus should always be cheaper. - When the work is outsource-able or isn't enough to keep an employee busy. In such cases it would be better to offload some of the work to a private contractor.

One last important point. Don't hire a junior developer because "that is all we can afford", or "that is all we are willing to spend" when they are not suited to the job. In the end all you will end up doing is flushing money down the toilet. Besides, once they acquire those skills they will request the big bucks anyway.

About me:

• Degree in physics with almost no formal training.
• Two years of work experience. So I know what the learning process is all about.
• Start software developer. I have done very demanding work and have seen all different skill levels from various individuals. Many of them can't handle a lot of what I do.

Answer 5

As others mentioned, junior positions may have little experience with C. I personally only briefly was taught about buffer overflows in C and even if I could watch out for them, it is likely I would still introduce some (especially if given an assignment that lends itself to creating buffer overflows). Likely, many junior developers will be a similar situation where they may know of buffer overflows, but they have not been prepared to identify and handle them in any extensive manner.

Given this, I would think the appropriate response is to have the issue brought up in the next possible interaction and ask them what they know about buffer overflows to test their overall knowledge. After that, tell them that you found one in their supposed production ready code. This would give you a good window to judge how they would react to correction and instruction.

Isn't the common thought that a junior developer who knows less but is willing and able to learn and improve is more valuable than a junior developer who knows more but who cannot or will not improve?

That being said, in one of your comments you mentioned you handed them tests that would have pointed out the buffer overflow in their code if they had used them. So perhaps the bigger question is why didn't they run the tests (of if they did, why did they turn in buggy code)?

Answer 6

A buffer overflow is an absolute no-go. You could have a corresponding code question. In a what all is wrong (can go wrong) with this piece of code the candidate should be able to pinpoint the problem. The question is, whether the issue is irrelevant, as you are running clint anyway.

However in an artificial free-form code test I would be mild on a violation like sprintf. Too little time (assumed), mind hyperactive, too large an urge to present something working.

Answer 7

I think you're asking the wrong question. There is no objective bar to be a professional programmer. If there were, there'd be a standard programming exam, but there isn't. Your individual bar just has to be set based on how picky you can afford to be, how much you can afford to pay, how much blunder your software can afford to accept, and how much time you can afford to spend teaching.

In this case, I'm guessing that a buffer overrun probably means that this code, which the candidate presented as a sample of exemplary work, crashes with a segmentation fault instead of doing what you asked. So, should you accept a coder who writes code that crashes with a segmentation fault instead of doing what you asked for? Ask:

• Is your organization capable of attracting anyone who can write working code?

• Is your development process robust enough such that someone who can almost write code could write working code with the help of peer review and testing support?

• Are you capable to teaching sort-of-programmers how to be a programmers, and are you willing to expend that effort and wait maybe a few years and hope that the candidate's inner talent will come to fruition?