82
votes

In the setting of an interview: What is the best way to reliably identify when somebody is an excellent programmer. By this I mean he is one of those that is 10-15 times more efficient / rapid / better than his peers towards the lower end of the spectrum.

Many of us have heard of the FizzBuzz Problem as a way to weed out the weak ones. Certainly, taking 5-10 minutes to solve that problem is a serious indicator that an applicant is a weak candidate. I posit that a good indicator is being able to solve that as quickly as you can write. This doesn't seem sufficient, though.

Is it maybe something like giving him a moderately complicated buggy program, and seeing how fast he can grok it and identify all the problems with it?

3

18 Answers 18

65
votes

My apologies to anyone who doesn't care for lengthy answers, but I do think it's pretty important to qualify your candidates before you hire them. Anyone who's conducted a significant amount of interviews in this industry knows that most candidates won't last through the first 15-30 minutes of an interview, so most of this list won't be necessary. Just keep in mind how expensive it is (both financially and emotionally) to fire someone before you dismiss my list as overkill. I've tried to list my interview topics here by order of importance.

General Intelligence (brain teasers/logic puzzles)

Computer Science Knowledge

Programming exercises

Knowledge of object-oriented programming techniques and common design patterns

Algorithm analysis (run-time O(n) complexity and storage requirements)

Usage of tools and methodologies

Knowledge of common security vulnerabilities and attacks

Basic Mathematics

  • Numeral systems (convert from one base to another)
  • Probability Theory
  • Distance between two points on Cartesian plane (Pythagorean theorem)
  • Square root (Heron of Alexandria, successive approximation)

Cryptography

  • Public key cryptography
  • Symmetric-key cryptography
  • Hash functions
  • Cryptographic protocols (secret sharing, zero-knowledge proofs)

Discrete Mathematics

  • Logic
  • Set theory
  • Graph theory
  • Information theory
  • Combinatorics
  • Proofs (like existence of irrational numbers, infinite primes)

You might also want to take a look at the book Programming Interviews Exposed. It's a good reference on the topic.

13
  • 10
    Phew, that must be some long interview. Oct 27, 2008 at 13:47
  • 8
    Today I attempted to solve "Crossing Bridges" with my ACM Programming competition teammate. The only difference is that we have to solve it for any number of people. It took us about 30 minutes to solve for any number of N people.. but in an interview setting I feel like puzzles are a poor metric.
    – Simucal
    Nov 1, 2008 at 0:39
  • 52
    In an interview, puzzles suck because the interviewee is nervous, not thinking straight, etc. In addition, many puzzels are a-ha! type things that really don't tell you anything about the candidate.
    – Robert C. Barth
    Feb 2, 2009 at 20:30
  • 11
    I find these mathematical problems of high difficulty. After you've finished your Computer Science degree for 10 years, how can you remember how to prove irrational numbers and such?
    – lb
    Dec 4, 2009 at 10:04
  • 14
    The trouble with puzzles is that most people haven't solved them; they've just seen the answers before. So the most savvy candidates will pretend not to have seen them and haltingly work out the answer they already know. If your goal is to hire smart people, not deceitful people, then puzzles are a bad choice.
    – Kyralessa
    Jan 13, 2010 at 4:34
28
votes

Ah, the eternal question.

I did a lot of interviews this year (I have two candidates scheduled tomorrow), and in my experience, hiring is more about gut feeling and people skills, and less about technical knowledge.

  1. Take your time with CVs. Some CVs can be rejected in seconds, some take half an hour. Sometimes I think about candidate based on CV a lot longer than I interview him. A few times I've prepared interview questions specifically for that candidate, even though I typically don't have prepared questions.

  2. Technical knowledge - there is a minimum I want, and this is usually pretty easy to tell. If in doubt, during the interview, talk about projects he mentioned in CV, and go as deep as you need to. This is usually more than enough to tell you what he knows and what makes him tick. Education is not important, previous jobs matter, possible personal projects score high.

  3. Ask about what he wants to do and where he wants to go with his career - do you need what he has, and can you provide what he wants? Also, near the end of interview, I usually ask about a preferred salary. If he's out of my range, or if I won't pay that much for what he knows, that's where we end the interview.

  4. Most importantly, candidate must fit into team, and I must feel confident that we'll be able to work together. I don't need to like him, but I must be able to handle him, and he needs to be able to handle me. If that's not the case, I'll pass, because I won't be able to use his technical knowledge. On the other hand, if this is the case, and if he's a quick learner, his lack of technical knowledge won't prevent me from hiring him.

I've trained girls from HR to pass me any CVs as soon as they get them; I schedule interview personally, as fast as I can (ideally day after tomorrow after receiving CV for good CVs). Then he gets half-an-hour or one hour interview with me and at least one co-worker (usually my boss or team-member), where I get to know him and answer any questions. Even if I reject his application on the spot, he gets 20-30 min tour of the company and I talk about what we do and how we do it. Then I send him to HR for psycho test and a bit of really really basic paper coding/SQL. Both tests almost never play a significant role in my decision, it's more of a verification that I judged correctly in the interview. After results, it's 15-min talk where I make him an offer, and if we negotiate the terms we're both happy with, he's hired.

This is a process I had to fight for through company bureaucracy, after missing a couple of great candidates, and which works because I'm the one that decides about hiring (although I do listen the advice from both HR and co-workers, my word is final). More decision-makers, longer process. The longer the process, the more you have to be Google to get top of the crop.

As soon as I'm certain that is a no-match, I end the interview, he gets the company tour and it's over. This might be as short as two minutes on the phone while scheduling interview. Even if you reject a candidate, sell the company. If you did a good job, good hire can come through word-of-mouth from rejected candidate.

Also, one tip. Do send rejection letters (or e-mails) for each application you get. In my current company I usually leave that to HR (apart from those I tell during the interview), but at one point it was priceless getting delighted response from rejected candidate in the lines of "THANK YOU! You're the first company which actually responded instead of leaving me wondering if they'll reply one day!"

5
  • Psychological testing?
    – Ink-Jet
    Jun 9, 2010 at 22:00
  • 5
    @Ink-Jet: nope, psycho testing was correct - the candidate is asked to write code which will be maintained by a violent man who also knows his/her home address.
    – Grundlefleck
    Jun 9, 2010 at 23:50
  • That's what I read it as the first time, to be honest.
    – Ink-Jet
    Jun 10, 2010 at 13:24
  • @Grundlefleck - yep, that's about correct. :)
    – Domchi
    Jun 10, 2010 at 22:07
  • 2
    If I got your rejection letter I would thank you. I've been rejected through silence after getting as far as a phone interview, and it's nerve-wracking.
    – 01d55
    Jan 7, 2012 at 3:42
24
votes

This answer is a little outside the box, but I think it's a valuable point.

The very best programmers rarely interview. They don't have to. If your company is particularly world-changing, or excitingly shrouded in secrecy, or several programmers they respect have gone there, then they might apply, but ordinarily great programmers get jobs through their network of associates, not by sending in résumés.

So: the best way to tell an excellent programmer in a job interview is that he's not there.

2
  • 2
    so true... great point. :) May 24, 2009 at 12:29
  • 5
    So.... it's "who you know" rather than "what you do"? The truly horrid programmers also get their jobs through friends and family. Oh, I'm sorry "network of associates".
    – Philip
    Mar 5, 2013 at 19:58
17
votes

Any answer must include code samples. Hiring a programmer without seeing his or her code is liking hiring a chef without trying his or her cooking.

11
votes

Possibly an "excellent" programmer isn't coming to you for an interview. You probably have to steal him from someone else.

1
  • doh! this answer seems to be becoming popular. Just as I have to start going out and applying for jobs ...
    – interstar
    Feb 2, 2009 at 23:42
9
votes

One way to tell the passionate programmers from the "I just want a job" programmers is to ask them what book they are reading this week. Then ask them about the books they have read in the past weeks.

I have found that the passionate programmers are ALWAYS reading, and usually the list will include a few programming / Comp. Sci. books in the recent list.

It's not just about keeping "up with the profession" - passionate programmers have a desire and love for programming, and tend to devour material on a variety of topics - not just the whatever language they are using now, but methodologies, other languages (especially new or "weird" or ancient ones), other aspects of IT (maybe robotics, or AI, or games, or...)

If they don't have a recent booklist at all, then they are probably not much of a programmer, in my experience.

Cheers,

-R

2
  • 8
    My recent book list is almost always fiction. My recent technical reading is almost entirely online, because that's more up to date.
    – Darron
    Nov 25, 2008 at 19:13
  • 1
    Better yet, ask them what book they wrote this month. :)
    – brian d foy
    Jun 17, 2009 at 5:51
7
votes

There are different time scales at which someone can be "fast": some smart people can solve difficult puzzles in seconds, but some smart people produce a lot of good code in a month, even though they might not be that fast at interview questions.

Ask the candidates if they are active in any open source project where you could review some of their code, and spend some time reading those projects' mailing list archives and commit logs. That will tell you a whole lot more than anything the candidates can demonstrate in an interview. (Of course this cannot replace the interview, since not all good coders do open source work.)

7
votes

The book "Smart and Gets Things Done: Joel Spolsky's Concise Guide to Finding the Best Technical Talent" may help to find an answer.

Table of content:

  • Introduction
  • Chapter 1: "Hitting the High Notes"
  • Chapter 2: "Finding Great Developers"
  • Chapter 3: "A Field Guide to Developers"
  • Chapter 4: "Sorting Resumes"
  • Chapter 5: "The Phone Screen"
  • Chapter 6: "The Guerrilla Guide to Interviewing"
  • Chapter 7: "Fixing Suboptimal Teams"
  • Appendix: "The Joel Test"

The Article by Joel "The Guerrilla Guide to Interviewing (version 3)" can be helpful as well.

And the article "Done, and Gets Things Smart" by Steve Yegge on the topic.

4
votes

Ask them a series of questions which require them to code, and have the questions get harder. If they seem to enjoy the challenge, you probably have a live one.

If they can't answer the first easy question, like "write a for loop" or something stupidly easy, then you know this person can't code.

4
votes

Have them code on a whiteboard. Only way you'll be able to tell if they know how to write code.

3
  • Don't know why this got downvoted. If a programmer can't write code on a whiteboard, what makes you think they'll be able to write it on a computer? Aug 18, 2009 at 17:04
  • 3
    @Kristopher: If a programmer can write good code on a computer, what makes you think they'll be able to write it on a whiteboard? Those are considerably different environments. Jun 9, 2010 at 21:59
  • The "whiteboard test" is not intended to simulate actual coding. It's a chance to see how the candidate thinks, whether the candidate can describe what they are doing, how quickly the candidate forms a solution in their head, etc. Someone who just stares at the whiteboard with no idea what to write will probably have the same problem at a computer. Jun 10, 2010 at 16:58
3
votes

You should mainly judge the work that they've already done. Any code or ideas that somebody generates during an anxiety-ridden interview are a poor proxy for what they can actually produce on a team.

To do coding challenges, use IM with something like codepad.com and let them do it from the comfort of their own house. Do you write much of your code on a whiteboard in front of your boss, with a 30 minute deadline and your bonus on the line? I don't.

So, is the interview pointless? No, but the emphasis should be on them explaining what they've done and exactly what they've contributed.

You are also going to be subject to all kinds of psychological biases once you meet someone face to face. Don't accidentally hire a programmer because they made better eye contact or are taller than someone else. To route around these, I would conduct as much of the interview as possible over IM/email before you meet them face to face.

1
  • You can reverse this effect by looking at other peoples psychological biases in the candidates hiring history. Short people who've had senior positions and achieved a things are probably really really good. Tall people with the same history will be, on average not so good and getting by on halo points. Jun 23, 2011 at 6:14
2
votes

The language doesn't matter. Logic does. I mean IDE's and compliers are so good these days that any good programmer can pick up any language (ok maybe not assembler) in a week; become decent in a couple of weeks and be very good in a couple of months.

It's his (her's) brain you need to confirm. And you do that my talking. I ask them to solve simple problems. Not by writing code but by stepping me through their logic to come to a solution.

But I admit, if he can't write a simple loop counting 1 to 10, you got troubles.

1
vote

First of all, there's one way you can get an idea before the interview even starts:

If they have a blog or contribute to one or more open source projects, just look at the code and articles they've written. First of all, if they've done either of these then they have initiative to get things done. Also, you can compare these things to the work experience they have listed on their resume and get an idea if they go home and learn more after work, or if they go home and forget about work after 5pm.

Essentially, Do they have passion about programing or not? That's the real question.

1
vote

Having a good programmer present in the interview is best in my opinion.

Only an expert can judge if the applicant just knows a lot of interview questions or if he is actually thinking about the problems and can go into detail. Remember, you don't want to hire people to solve interview puzzles, you want to hire them to get actual work done.

Puzzles are to exclude people who don't get the basics right. If you want to test for skill, prepare a few things you (or your "good programmer") can go into detail about and focus on the one the applicant has to think about for a while. How does he approach problems that he doesn't immediately know the solution of?

1
vote

I don't think that you should talk about passion on the interview. Frankly, it sounds like a company that looks for 'passion' really means 'working for no money for the idea'.

Passion doesn't even warrant excellence. I mean I spend almost all my life programming, reading about programming, learning crazy languages like Erlang or Clojure and I don't get paid for any of it. Yet I suck at programming.

I think excellent programmer should have a track of successful projects they have been actively involved in. Thus making a programmer write anything above basic FizzBuzz is unnecessary in the interview. Talk about their past projects and what they did. Are you hiring programmers to solve Rubik's cubes and count marbles or work on long and big and exhausting software projects of more than 50 lines of coe?

1
vote

http://www.inter-sections.net/2007/11/13/how-to-recognise-a-good-programmer/

From the article:


The criteria in bullets

So, in summary, here are some indicators and counter-indicators that should help you recognize a good programmer.

Positive indicators:

  • Passionate about technology
  • Programs as a hobby
  • Will talk your ear off on a technical subject if encouraged
  • Significant (and often numerous) personal side-projects over the years
  • Learns new technologies on his/her own
  • Opinionated about which technologies are better for various usages
  • Very uncomfortable about the idea of working with a technology he doesn’t believe to be “right”
  • Clearly smart, can have great conversations on a variety of topics
  • Started programming long before university/work
  • Has some hidden “icebergs”, large personal projects under the CV radar
  • Knowledge of a large variety of unrelated technologies (may not be on CV)

Negative indicators:

  • Programming is a day job

  • Don’t really want to “talk shop”, even when encouraged to

  • Learns new technologies in company-sponsored courses

  • Happy to work with whatever technology you've picked, “all technologies are good”

  • Doesn't seem too smart

  • Started programming at university

  • All programming experience is on the CV

  • Focused mainly on one or two technology stacks (e.g. everything to do with developing a java application), with no experience outside of it

1
  • would you mind explaining more on what it does and why do you recommend it as answering the question asked? "Link-only answers" are not quite welcome at Stack Exchange
    – gnat
    Jun 9, 2013 at 10:17
0
votes

An excellent programmer will be able to work with those lower-spectrum peers too. As long as they can do the test and don't wallow in their ego then you have a good candidate, no?

That fizzbuzz test is kinda funny though. The solution I can think of uses the modulo operator. Which I only know from working out character-sheet mapping coordinates (never mentioned in school or college). Would the average programmer even know about that or have I had crap education?

7
  • I am surprised that you did not come across the modulo operator. I was introduced to it in the different languages I have learned over the year.
    – David Segonds
    Oct 27, 2008 at 6:47
  • 2
    If you majored in CS in college, Modulo operator is Programming 101
    – Simucal
    Oct 27, 2008 at 6:55
  • surprisingly stuff like bit-shift and modulo gets skipped over in college
    – Claudiu
    Oct 27, 2008 at 13:08
  • I think it depends what kind of things they are trying to teach you at college. I don't think I've ever used modulo in a real-world problem, nor taught it explicitly. But it's very common in this kind of exercise (and in exam questions).
    – interstar
    Oct 27, 2008 at 16:40
  • 2
    Actually, they're both generally taught in elementary school; at that stage they're referred to as "the remainder" and "multiplying by 10".
    – intuited
    Sep 23, 2010 at 9:00
0
votes

One criteria that I use is to see the 'kind' of languages and tools he has worked on, either in academic or in professional projects and what exactly he achieved. Has he always worked at the application level using standard libraries (always a C# or VB6 guy?) Or has he done a project using C in Linux dealing with hardcore stuff like pointers, memory management, recursion, process synchornization, mutual exclusion, events etc. If he has always used these core and fundamental concepts under some abstraction layer, I will be doubtful.

This is obviously in addition to making him write code. Nothing is a substitute for that. I do however cater for the fact that some people can write code faster than others, and people have different response time when in an interview limelight.

2
  • recursion, process synchronization, mutual exclusion. Those technologies are equally important whether you work with C#, VB.NET, C or assembly language.
    – erikkallen
    Dec 27, 2008 at 9:43
  • -1 - This is dead wrong. 'Kind' of languages and tools are 100% 'irrelevant'. Sep 12, 2011 at 19:58

Not the answer you're looking for? Browse other questions tagged or ask your own question.