How to Interview Programmers

I’ve posted a follow up with more specifics on “How to Interview Programmers Part 2”

Google has finally come clean that they figured out what many of us have been repeating for years: brain teasers in interviews don’t lead to measurably better employee outcomes.

I’ve never interviewed at Google but I did interview at a company that asked me a “brain teaser” question. It wasn’t one off of the shelf, but based on natural conversation (I mentioned I was a frequent-flier, so I had to calculate some aviation-related things that were very similar to “Guess the # of piano tuners in NYC”.). It was very difficult to resist the urge to blurt out: “Do you have any idea what you are doing?”

These aren’t just wacky – they’re stupid. Not only do these questions bear little resemblance to anything we do in software development, it’s lazy to use brain-teasers. “Let’s spend 30 seconds thinking up a few puzzles then have the candidate spend a few hours answering them.” Recruiting should be your number one priority! Treat it like it’s more important than the semi-annual flossing you do the the day before your dentist appointment, ok?

I have gotten too many job offers after inadequate interviewing to think that interview incompetence is anything but prevalent in our industry. Luckily for them, I’m excellent at what I do, but plenty of other startups lose the hiring lottery. You don’t need to be one of them!

Over the years I’ve refined my process to weed out people from the very start and provide repeatable, scalable interviews that organizations can trust.

Preconditions

Hiring is the most important thing you can do as an organization. Yes, more important than building software, selling it, and cashing fat checks. Without your people, your organization is nothing. You’ll need to trust them to act independently with minimal oversight.

The interview process starts long before candidate XYZ lands on your jobs page.

Before you hire your first person

Figure out your culture. Most startup CEOs/Founders/CTOs that I talk to have no discernible idea what their culture is. How can you hire for cultural fit if you don’t know what it is?
Make sure everyone on the team interviews every candidate. If you’re large enough, include people from other teams.
Make sure everyone on the team knows how to interview, based on a checklist/plan. Instead of Bob asking the same SQL questions that he came up with in 1993, have a guide that each person can consult. Have each person write down what they are going to talk about and go through it as a group. What are you trying to look for? How do you know if you found the answer?
Don’t ask the same thing twice. I’ve been in day-long interviews and had to “tell [person X] about [myself]?” six times. I didn’t accept an offer at that company: clearly they have communication and organizational problems.
Conduct mock-interviews to level up your hiring capability.
Have your developers take the coding challenges so they can provide intelligent guidance (and make sure that the challenge works / is still relevant. I’ve been in an interview where the coding challenge had an unintentional bug and wouldn’t work as stated.)

During the interview tips

Be open and honest. Power distance in interviews is heavily in your favor; be the first to offer salary and equity ranges for the position. Don’t ask them “what’s your bottom line salary?” – you should have a well-defined technical ladder that outlines the career path of a developer within your company. No tricks, just fair, equitable, and transparent progress.
Tailor the interview to the candidate. If someone has spent 30 years writing compilers, don’t ask them “What is a pointer?”. For someone fresh out of school, ask more CS-y questions. Many companies have the same set of Q&A for every candidate and it shows both a lack of effort and turns candidates away. If you’ve gone to the trouble to bring a candidate into the office, don’t waste your and their time – researching the candidate shows that you are truly interested in them and what they have to offer.
Don’t “lead the witness”. I’ve sat in depressingly many interviews (on both sides of the table) where someone asked a question like: “So, do you perform Test Driven Development or do you simply ‘Cowboy Code’”? Any smart candidate, regardless of whether or not they’ve even heard of TDD, will read between the lines and tell you exactly what you wanted to hear.

Ask topgrading-style questions and be careful not to let your biases be used against you.
Have your shit together. Make sure that the person who is helping the candidate out gets everyone in the right place at the right time. The candidate is interviewing you and your organization as much as you are interviewing them. The best candidates are interviewing at companies that get this.
Make lunch reservations at a nice sit-down restaurant. The best teams eat lunch together.

Post-interview

Everyone must give a thumbs up or thumbs down (and why, based on the above). The rationale is just as important as the rating. Why do or don’t they fit the culture? Why will they raise the collective skills of the team (you do measure that, right?). If you don’t like the color of the t-shirt they wore to the interview, tough. That’s not a reason to not hire someone.
If they are going to be issued an offer, send it ASAP.
If not, thank them for their time and explain they will not be moving forward. Do this as soon as possible.

Pre On-Site Interview

Initial phone screen (30-45 minutes)

Just a few background questions (why are you looking for a job, what sounds interesting about what we are doing, what is your current salary, etc.). (This should take about 30-45 minutes.) The goal is to make sure the interviewee meets your “deal-breakers” (you should have a list) and vice-versa, and minimize the amount of people who slip through the cracks that later don’t take an offer (or you don’t issue one) because of knowable conditions that wouldn’t allow them to be successful in the role.

Be sure to talk about the company culture, how the company makes money (being profitable helps), the fundraising situation, how large it is, etc. – these answers will help a candidate self-select to move forward in the process. If they don’t want to work at a 3 person startup – why waste your and their time by continuing with phone screens and on-site interviews?

For example, if you don’t allow work-from-anywhere, be up front and explain that in this interview. Most programmers expect it nowadays (and you’re silly if you don’t offer it). Why spend thousands of dollars to fly them out only to have them turn down your offer because of that?

Second phone screen (1-2 hours)

After the deal-breaker conversation, at least you and the candidate are not immediately disqualified. Now, let’s figure out if they actually can program. Ask a bunch of topgrading questions then a few FizzBuzz-style questions to make sure they can actually program.

For the coding challenges, I use some sort of screen-share program like TeamViewer. I ask them to open a REPL in the language of their choice and write a series of simple programs (I construct a special URL that has the instructions they can use so I don’t have to read it over the phone).

They are not fizz-buzz (because folks have figured out they need to know it) but similar time/space challenges.

On-Site Interview

If they pass these, then bring the person out for a day of in-person interviews (flying first-class, provided it’s not prohibitively expensive, picked up at the airport in a nice car, all that jazz).

If the candidate is at all good, they will be fielding many offers – if you can take care of them from the start of the process it will help you stand apart from the crowd. Don’t go overboard – a driver in a tux sends a strange message – but show that you care.

Day’s events

Tour the office, show off the fancy espresso machine in the break room, etc.
Explain the day’s events (basically, show them this list)
A few programming challenges (of increasing difficulty) that involve stuff the candidate would use on the job, extracted from, or related to, our problem domain.
Lunch
Then pair programming for the rest of the day to make sure we can all get along with this person (continuing through the next day if possible).
Fancy car ride back to their hotel (not the airport). Try and give them more than a day in your city; if you have the budget, give them the whole weekend.

Programming Challenges (2-4 hours)

You need to see if this person can solve challenges that are harder than FizzBuzz but won’t take them 3 weeks. They should be difficult enough that they can’t solve it by memory but not require much esoteric knowledge (unless you’re looking for that). It’s “open-book” – Google, Stack Overflow, etc.

In the initial interview I ask their preferred development environment (OS, editor, etc.) and setup a computer thusly, so that they should feel more-or-less at home. Alternately, if it’s specialized enough, I have them bring in their own laptop and work off of that.

I have a Github repository that is already setup with a basic test suite and a bunch of problem sets that they can choose from. If you are setting out to build this repo from scratch, I highly suggest something like RubyQuiz or maybe CodeQuizzes.

For example, you might have them implement a local script that takes either a Zip code or city name and prints out the current weather.

The goal is to see if they can solve a real-world problem that either we have solved or is representative enough of what we do on a daily basis to be a good predictor of future performance (weather involves dealing with/parsing an API, and how do you automatedly-test a 3rd party API?).

Lunch

Self-explanatory. Have lunch, break bread with them. Everyone on the team attends. This is the proverbial “have a beer with them?” test. Let them lead, ask them open-ended questions.

Pair Programming

If you had the applicant sign an NDA that grants the company ownership of the code, (you don’t want them to work on production code, then come back later and claim that you owe them some equity or compensation) have each developer prepare some tasks on the real code-base that are complicated and require some problem solving. Often I find some bugs that I think are tricky, track down the solution (but I don’t fix it), then I can have them try it. Debugging requires building up complex mental models of how the code works, and it also tests how they might react when presented with difficult problems.

If they haven’t/won’t sign an ownership agreement (I’ve never had someone do this) then take some open-source project you use and have them work on adding some new functionality to it that you need. Contributing to Open Source Software should be a core part of your business – and the licensing means that your business is not put at risk.

Reference Checks

July 2 update. An anonymous commenter asked about reference checks. They completely slipped my mind, but in my experience the value of the reference check is pretty variable. I’d be curious to see if Google has any data that suggests it’s a good predictor of future success.

At bigger companies, legal has beaten into everyone’s heads that reference checks are sources of liability and should stick to factual information only: Did such-and-such work here, for the terms stated (the role and time-frame), and at what salary, etc.

I’ve never been able to extract actual meaningful information, but sometimes you can read between the lines.

At smaller companies, you’re much more likely to get the founder/CTO on the line and chat about their work habits. Of course, the candidate hand-picked these people because they are going to give a rosy picture – all reports you get must be taken with a grain of salt.

If you get a reference that gives a negative picture, I suppose that is a good reason to cut and run, but I can’t imagine any candidate worth hiring would make that sort of mistake. For me the only question I need answered is: “Would you hire XYZ again?” – if it’s a “no”, then I tend to run.

I’ve heard some folks do friend-of-a-friend – ask people who no longer work there about candidate XYZ. Personally I’ve never done it but I can see that might be more diagnostic, but I think the ethics are probably pretty dicey.

(What is better – candidates should interview former employees. Really interesting. Same grain of salt applies; I’ll follow-up later in a “How to interview companies” post.)

Longer-Term

If possible, put a new hire on a 1 month contract before you make a final offer. The contract rate is the equivalent salary for the position, but you need more data to make sure it’s a fit. If they are clearly not a fit, give them a generous severance and assistance in finding a job elsewhere.

If they are a fit, issue a final offer (using the same salary as before) with the equity grant (how can you judge future multiplier potential after one interview?).

I understand not every candidate can and will agree to a one-month contract. In that case, make them an offer, slot them in to your ladder where you expect, but withhold the equity component until the end of the month, unless it’s a standard amount (the later the stage of the company, the less variable the equity calculations).

Update:

Evan Hensleigh (@futuraprime) notes:

“Good advice! But I’m not sure about the contract period. You might be turning away good people who need, e.g., health insurance.”

Fair point. In that case, I’d be more than happy to offer a bump in the contract rate to include 1 month of their previous employer’s COBRA. Ultimately, most US states are “at will” employment – you can be fired at any time for any reason. A few benefits aside (which can be resolved) being fired after one month is functionally equivalent (provided you treat the employee as a contractor under the rules of your state).