What is the best way to evaluate new programmers? [closed]

Question

What is the best way to evaluate the best candidates to get a new job (talking merely in terms of programming skills)? In my company we have had a lot of bad experiences with people who have good grades but do not have real programming skills. Their skills are merely like code monkeys, without the ability to analyze the problems and find solutions.

More things that I have to note:

• The education system in my country sucks--really sucks. The people that are good in this kind of job are good because they have talent for it or really try to learn on their own.

• The university / graduate /post-grad degree doesn't mean necessarily that you know exactly how to do the things.

• Certifications also mean nothing here because the people in charge of the certification course also don't have skills (or are in low paying jobs).

We need really to get the good candidates that are flexible and don't have mechanical thinking (because this type of people by experience have a low performance).

We are in a government institution and the people that are candidates don't necessarily come from outside, but we have the possibility to accept or not any candidates until we find the correct one.

Answer 1

Regarding candidate selection, I usually go with a three-strike plan :

• Regular test with FizzBuzz-like coding questions and many knowledge questions where they have to give coded examples. Depending on the position, it can be OO principles, SQL design principles, etc. I increment the difficulties of questions across the test to see how far they can go. The idea is not really to have all the questions answered (if they do, the better), but also to see if they can acknowledge when they don't know something. Trust is essential, and I don't want to have someone lying to me in my team.

• Return on the test with the candidate, and discussion around the answers. Possible extension of the questions to reach the candidate's limits. This can be extensive, and the more extensive it is, the better.

• Last part but not the least, The Code Review. I ask the candidate to bring a piece of code (I generally space the previous test/discussion and this review by a few days, to let them write and polish one piece of code). Then we do a regular code review of it with two people : one person that will directly work with the candidate and the person that reviewed the test with the candidate previously. Regarding the code review you can read this article from JohnFX.

At the end of all this, you should be able to decide if you want this candidate to be part of your team or not.

Answer 2

Start with giving them FizzBuzz to solve. That should weed out the worst of them.

Then something a bit harder - for example, how to reverse a string without built in library functions. Ask them to talk while solving in order to see what their thought process is.

You can keep giving harder problems if they find these very easy, until you are convinced they can walk the walk and not just talk the talk.

Answer 3

Just look for passion about the job.

To quote Joel, look for people who are "Smart, and get things done."

The rest doesn't matter

Answer 4

Based on my 25 years of programming (which, admittedly includes only a 5 or 6 instances of hiring other programmers) :

Positive indicators:

• Passionate about technology

• Programs as a hobby

• Will talk your ear off on a technical subject if encouraged

• Significant (and often numerous) personal side-projects over the years

• Learns new technologies on his/her own

• Opinionated about which technologies are better for various usages

• Very uncomfortable about the idea of working with a technology he doesn’t believe to be “right”

• Clearly smart, can have great conversations on a variety of topics

• Started programming long before university/work

• Has some hidden “icebergs”, large personal projects under the CV radar

• Knowledge of a large variety of unrelated technologies (may not be on CV)

Negative indicators:

• Programming is a day job

• Don’t really want to “talk shop”, even when encouraged to

• Learns new technologies in company-sponsored courses

• Happy to work with whatever technology you’ve picked, “all technologies are good”

• Doesn’t seem too smart

• Started programming at university

• All programming experience is on the CV

• Focused mainly on one or two technology stacks (e.g. everything to do with developing a java application), with no experience outside of it

In addition, I'd suggest:

• The FizzBuzz test (or something like it to test basic ability to write an algorithm.
• Harder version of the FizzBuzz test (to get them to the failure or near-failure point.)
• Discuss their code and see if they are willing to be self critical and look for improvements (which they probably didn't have time do do in a short on the spot test) such as: - good variable names (I've had very experienced skilled coders use variables in production like "flag" (WTF??) - modularization. - Anticipating problems and doing "defensive coding"
• A willingness to see "flaws" as opportunities for improvement. I think the best coders always look unflinchingly for flaws in their previous code. They are not so egocentric as to think that finding a flaw their is a personal affront. They see it as an opportunity to do better. (Those that can't look at flaws unflinchingly either are overwhelmed by seeing a flaw (and become super underconfident or, to avoid just that, they ignore the flaws.
• Can they debug?
• Can they Unit Test? (I've talked to way too many programmers who say "QC does that". I'm not talking about Testing, I'm talking about testing: you write a function, does it work? Does it make reasonable efforts to deal with likely problems (NULL input, etc.) ? If you can't do that, how do you know when you're done?
• Do they have good communication skills? (at a minimum: good comprehension and self knowledge about when they do and do not understand and a willingness to say "I don't understand, please explain it again".

Much of the summary above is from How to spot a good programmer, which is a great article, focused a bit more on longer range indicators. It definitely confirms my intuitions and experience. It's also a lot of things (like "passion") that aren't normally mentioned in a checklist of "what's a good programmer".

Answer 5

Evaluating programming intelligence is a form of Turing Test. Thus there are (currently) no closed form evaluation procedures that are guaranteed to work. It takes intelligent programmers to recognize other intelligent programmers, but only with some reasonable probability.

Your chances will be better if you have interviewers on your team who can smell snow jobs, and instinctively dislike working with stupid people (even the ones who are good looking, have impressive looking resumes, and can spout all the usual canned solutions from memory).

(One possibility methodology that would help the quality of stackoverflow as a side-effect is to dig up old stackoverflow questions, related in some way to your job requirements but that in your opinion have inferior answers; ask the interviewee how they would answer, and have them post it if it's a good answer. Similar to a recapcha for crowd-sourced OCR.)

Answer 6

Give them a problem, preferably one associated with the problem domain they'll be working on, and ask them to discuss how they would approach it. You can have them just discuss, pseudo-code or write bits of actual code depending on how confident you are in their skill level

For example, if your organization did conferences, ask them to outline how they would code a secure online registration system. They should be able to cover some of the basics and ask good questions about exactly what needs to be implemented. As you interact, you should be able to determine if they'll be a good fit for your organization and the role you need them to fill.

I'm not a big fan of programming trivia tests and brain teasers. While they can be fun for some people, they can also annoy and/or stress out other people, including people who might just be the best fit for your team. Plus, info on many such tests are readily available online and will encourage cramming for the tests and other tactics that would blunt their viability to gauge programmer ability.

Answer 7

Reading this question and some of the answers it has received prompted me to write an article which I feel might be of interest:

Odd recruitment practices when hiring software developers

Ok, so the article title is rubbish, but the article gets to the heart of the problem. It's not the candidate's problem that you've chosen to interview them no matter how inappropriate they may be for the role you have in mind. If you haven't managed to define a well-factored hiring procedure to allow you to find the gems in the rough, then you're going to just have to live with the consequences, and yes, this means getting a few candidates that could never meet with your expectations. Filtering your candidates based on their letters and resumes requires you to first, ask your applicants to write a letter about themselves and what they want from the role, and then look at how the resume is written. If you've only got one or two potential candidates to interview, then you've probably done the pre-screening properly. If you can't decide between your candidates at this stage and you still have a hundred applications, then you've probably either set your expectations too low, or you haven't been aggressive enough in your filtering process.

When you eventually do find the 1 or 2 candidates that you consider actually worth your time, don't simply ask a handful of inane tester questions, but instead invest the time to get to know these people, and to engage in open discussions about software engineering in general. You'll learn more from a casual approach about the candidate than you ever will in the traditional (and somewhat adversarial) interview situation. Also, don't simply settle for a single interview, but instead treat your key candidates to several meetings where open discussion is used, and where the candidate can meet with their prospective colleagues. The time is never wasted, as inappropriate candidates will not thrive very well in a highly technical discussion, and will show their flaws very quickly as they drop their guard. If you spend the time and still don't have a hire, you've had an opportunity to learn more about what your needs are, and can continue to improve your interviewing process based on what you learned from the 'failed' interviews.

Answer 8

I suggest you go with a FizzBuzz question and hire the first one that passes. Further tests tend to be flawed as not every good programmer will approach a problem like you, or handle an interview without stuttering, or know the languages you want or care about or silliness like exchanging integers without a third variable (who needs that anyway? I mean, since RAM exceeded 128 bytes?).

Think about it. If the FizzBuzz question eliminates 199 out of 200, then it just eliminated hundreds of interviews. Were you really going to interview hundreds of prospects?

Just seems like diminishing returns after FizzBuzz. That is assuming that 199/200 is even approximately close. And I presume YOUR time is valuable too...

Answer 9

You haven't said for which language, but it is fairly easy to test someone's knowledge. It also depends on the level you are looking, but there is a fairly large pool of questions regarding the interview questions.

However you decide to hold your interview, don't asks those "lateral-thinking puzzle" interview questions.

Answer 10

I'm not sure if this is a comment or answer but basically what Matthieu said. You want stupid easy questions that take a minute or two (but no longer than 5) minutes to do and they should be about different areas.

Such examples of stupid easy question is a question about recursion such as you have a list and you must print it reverse order without using a loop. A simple regex question if regex is normally done in your development. A question about bits and bytes if using C++ (write a template that accepts char to long and prints out the binary representation. Specialization isnt required, just use sizeof() to figure out the bit length)

It should take you about <=3minutes per question

Answer 11

Ask them about the most interesting programming challenge that they have ever tried to solve but could not, what approach did they use while solving it, why they could not solve and what other approach they think can solve it.

This is enough for me to judge a programmers abilities as a programmer.

Answer 12

1. Can they defend what they claim they know? They put it on the resume/CV as a skill or something they did on another project. See how in-depth they can go on the subject.

2. Can they learn something new? Talk about a high-level aspect of the technology you're using or something specific to the business domian you work in and see if they can grasp the subject. Do they ask intelligent questions? Can they come up with an analogy? Is it similar to something they've done in another industry or technology?

3. Would they rather be programming? It doesn't have to be number one on their list, but they have to show a preference to writing code. And I mean actually writing code and making something, not sitting around and talking about it or drawing on the board all day long. Not to minimize planning or to promote cowboy coding, but you gotta have code eventually. Avoid those who avoid the keyboard. This isn't a managment position.

You can do some scoring on a scale of one to ten thing or just rely on being able to smell your own kind.

Answer 13

If it makes you feel better bad programmers exist in pretty much every country. How to weed them out is the problem.

First weeding is the resume. One thing I look for is a lot of claimed language experience and nothing to describe what they did in that language. I've seen resumes that pretty much claim they know every language ever invented and yet their experience shows they've only actually worked with Access and Visual Basic. Those go right in the trash. 10 page resumes go right in the trash (especially ten page resumes from people with less than 2 years of experience which I have gotten). From recent college grads with little experiece, you have to be really picky about how they present themselves. The best candiates are careful with their resumes, they don't have errors. Are you really looking for someone who cares so little he didn't bother to proofread his resume?

Professionally prepared resumes go in the trash too. Once you've read hundreds of resumes, you can pick them out as they use the exact same phrasing. You can't trust the content in a professionally-prepared resume and you know the person didn't do his own prep. This is the kind of person who will rely on others to solve his problems for him, do you really want that in a programming position?

Look for things that make the person stand out for the ones you pick. That's harder of course with the ones just out of school, but look for accomplishments, contributions to open source, etc.

The next weed out is the phone interview. Ask about basic concepts that relate to the actual job you have. If people don't have basic knowledge of concepts you need them to have, they aren't worth bothering to bring in to a personal interview. The young often think this is unfair becasue they can look up everything on the Internet, but the truth is I have never met a good programmer who had to look up everything on the Internet. You should have some knowledge of your profession that you don't have to look up each time.

After the phone interview you should pick the best 4-5 candidates and interview. Of course if you only have 1-2 good candidates, don't bother interviewing people you already eliminated. Now you are going to ask the hard questions and get a feel for how they approach problems. I would never use the fizzbuzz test because it is too well known so the answers tell you nothing. Instead make up some problems from your own code base. I might give them a requirement and a piece of code and ask them if the code meets the requirement and if not why not and what they might to do make it meet the requirement. I would ask them to describe the most difficult programming problem they have had to solve and what steps they took to find the answer. I would ask some more in-depth technical questions. Remember you are trying to get a feel for their technical competence, their problems solving and debugging ability and their ability to fit with your existing team. I also ask questions that they probaly don't know the answer to to judge how well they handle stress, it's a stressful job, I don't want someone who folds in the interview because the stress of the job is greater than the interview stress. I look for strengths in areas we are currently weak in and ability to work in teams and to present themselves to the customers (our devs deal extensively with users), your list may be different.

Answer 14

The candidates must be given a real world problem to solve with the freedom to use any technology.

If she comes out in flying colors, she's In !