“Would Mahatma Gandhi have made a good software engineer?”
I mean, he’s Gandhi — can you even say no to that? His leadership skills would have helped guide his coworkers, but his technical knowledge would certainly have been lacking — he died in 1948, after all, a time far from Go, Python and other programming languages used commonly today.
This is a flawed question. But it’s also a brain bender known to be asked of candidates in the tech industry, supposedly measuring individuals’ intelligence, creativity, problem-solving skills and more.
Large companies like Amazon can ask and perhaps derive value from vague questions like that, according to Cyrus Kazemi, a software engineering manager at Porch, a home improvement marketplace that connects contractors and homeowners. But Kazemi said he likes to ask questions rooted solely in reality.
“The whole point is to try to emulate as close as we can what it would be like to work together.”
“At no point in the interview process would we want anything like a trick question or whatever. We want [interview questions] to try to accurately assess their skill,” Kazemi said. “The whole point is to try to emulate as close as we can what it would be like to work together.”
Interviewers from other Seattle firms like Tableau Software and Apex Learning agreed that their goal is to assess what it would be like to have that candidate as a coworker.
Zaheera Valani, director of engineering at Tableau, said she likes to ask candidates at the data visualization firm about their previous work, since she thinks it’s a good way to measure what they’re proud of or what they would have changed about their past. Gabe Geisendorfer, director of software development at Apex Learning, said he likes to know if a candidate has worked in an Agile work environment before, since being able to work quickly and iterate on products as they go is an important value at the education software firm.
We talked to interviewers at these three Seattle firms about how to structure a technical interview and ask the right questions.
It matters who is in the room. Include those with a variety of backgrounds, such as women, underrepresented minorities and engineers with a variety of skill levels. These interviewers can help companies avoid biased hiring.
Should you assign take homes to a potential hire? Opinions vary.
Whiteboarding can be an effective means to measure candidates’ communication skills and coding knowledge.
Know a red flag when you see one. Candidates who show up to the interview hung over, question the process or can’t think of a reason they’d want to work at your firm are not a good fit.
Paired programming can provide insight into how an engineer really works.
Brain-teaser questions aren’t a good measure of how smart someone is. Instead, try asking for specifics about someone’s previous work styles or job experience.
Standardized measurements keep interviews focused. Making sure everyone is on the same page before an interview, and standardizing the system rating candidates’ strengths, can help make sure interviewers are looking for the traits the company prioritizes most.
The best questions to ask
Valani said she likes to ask candidates to talk about a challenging project they’ve previously worked on, as well as what actions they took, what results they saw and what other solutions they could have implemented. She said candidates’ answers offer insight into what they have actually accomplished, and spotlights the roles they’ve held at other companies. Talking about past experiences also helps candidates feel comfortable, she said, because, in contrast to “brain teaser” type questions, they’re discussing an experience they actually lived through.
“What role did they play in that? How did they work with other people? What did they learn out of it?”
“It gives you a way to understand, what role did they play in that? How did they work with other people? What did they learn out of it?” Valani said.
When Geisendorfer was interviewing for his job at Apex Learning, he remembers being asked to write a program in a fictional coding language that would make two trains running on an infinite track collide. Albeit vague, Geisendorfer said he liked the prompt because he had never been asked it before, and there was still an element of it that related to his work experience.
“It was fun because I’d never been asked that question before, and the interviewer and I were working through it together,” Geisendorfer said. “It was a hybrid of a kind of coding problem and then that abstract type.”
That’s about as hypothetical as the questions at Apex get.
Geisendorfer said he likes to ask candidates how they solve technology problems and approach disagreements with their coworkers. Apex’s scrum master also spends about 45 minutes talking with candidates about their experience working in an Agile environment.
It’s mostly “how they interact and work through problems, their experiences with Agile methodologies, how they deal with feedback or resolve disagreements around technology or approaches, things like that,” Geisendorfer said. “There’s a whole piece to it that I think can be easy to overlook when you focus on just the technology.”
Whiteboarding measures how a candidate thinks through problems
Interviewers at both Apex Learning and Porch use a whiteboard to measure candidates’ technical skills. Both are looking at how candidates solve technical problems, and how they communicate their thought processes along the way.
At Apex, the further a candidate gets into solving a technical problem, the harder the problem gets. Geisendorfer said he generally asks candidates a systems design question that calls for them to iterate on their solution. For example: An Apex candidate could be asked how they would store data and then add a feature, like implementing spell check to a word processing system.
Geisendorfer wants to see how a candidate deals with difficult points in the problem and how they work with, and ask for help from the interviewer. How a candidate approaches these problems is more important than their result, he said.
“With the right approach you can gain the right support to get the results that are necessary.”
“Your approach, how you think about problems, what questions you ask, what information you need, how you work with someone else, I think it’s harder in some cases to teach,” Geisendorfer said. “With the right approach you can gain the right support to get the results that are necessary.”
Kazemi said Porch breaks their whiteboard portion into two parts: a systems design test, and a data structures and algorithms evaluation.
For the systems design portion, a candidate could be asked how they would design a chat application similar to Facebook’s, Kazemi said. The question is intended to allow for some creativity on the candidate’s part and gauge how they break down the problem at a high level.
For the data structures and algorithm design portion, Kazemi said a candidate would be asked a question that calls for a more narrow answer and has a few expected solutions, like solving a sudoku.
Both whiteboard questions are intended to highlight how a candidate interacts with the Porch interviewer.
“If a candidate just zoomed to the right solution and got it totally right but didn’t say a single word up until that point, then that is a red flag for sure,” Kazemi said. “It definitely is important that the candidate communicates their thought process, and makes it very clear why they’re doing what they’re doing.”
Paired programming measures how a candidate codes
Standing in front of a whiteboard can be stressful for some engineers. And in a profession that generally relies on highly focused, individual work, whiteboarding can feel very unfamiliar.
For these reasons, candidates at Porch are also handed a laptop, paired with an engineer and instructed to code for a portion of their interview. Kazemi said paired programming is the most realistic way to measure a candidate’s skill set, and that it also can relieve some of the public speaking pressure a candidate might feel at a whiteboard, since being at a computer with a single peer feels more natural.
“How you recognize when you’re stuck, where you recognize what the problem is, what the potential solutions are and how you elicit help and feedback and work that feedback into your solution — paired programming I think gives us the best assessment of all those things, and other actual development knowledge,” Kazemi said.
To use a take home, or not to use a take home?
When Valani was first interviewed at Tableau, she said part of her process included completing a take home assignment. Her experience represents an outlier at the digital analytics firm — a candidate is only given a take home if they show potential, but didn’t do well during the onsite technical screening, or if they’re interviewing for a leadership position, she said.
Take homes shouldn’t take more than a couple of hours to finish, Zalani said, and can range from spec documents that the candidate reviews and discusses with the interviewer, to designing a model application programming interface (API). Once the candidate is done with the assignment, the interviewer reviews their work with them, and asks about their decision-making process. Valani said this helps the interviewer understand what the candidate would be like as a coworker.
“It’s very collaborative and it really helps you get a feel for what it would be like to work with the other person in that environment.”
“It’s very collaborative and it really helps you get a feel for what it would be like to work with that other person in that environment,” Valani said.
Apex Learning, meanwhile, stopped leveraging take homes in their hiring process about three years ago — Geisendorfer said the firm previously had candidates look over a full-stack question that took no more than four hours to complete. They thought the question helped gauge candidates’ technical ability, but didn’t do a good job of measuring their experience working in an Agile environment. They also found that some candidates would drop out of the interviewing process after it was assigned, since they didn’t want to complete it.
“There was a significant upfront investment for the candidate,” Geisendorfer said. “Maybe some were not looking to engage at that level because of the market. There’s a lot of positions, a lot of opportunity. So we just felt, ‘OK, we’ll back off from that.’”
Apex now has candidates complete an online coding exercise during the technical phone screen, where they work on a problem generated by Collabedit, a web-based development tool.
When a candidate waves a red flag
In the nine years Geisendorfer has been interviewing candidates, he said that he’s noticed some red flags. For example, there were a few times when candidates arrived at their interviews obviously hung over. Other times, they’ve overshot their coding skills on their resume. Once, about six years ago, Geisendorfer had a candidate tell him, “I don’t know what the point is of all these questions.” At the time, Geisendorfer explained what Apex Learning was trying to measure with what he was asking and continued with the interview. But he also understood immediately that the candidate wasn’t a fit for the firm.
“If this is uncomfortable to you and for some reason you don’t feel like this is a place you’d want to work, then that’s part of the process for both sides.”
“This is what we’re doing and this is our process,” Geisendorfer said. “It’s not really up for debate if it’s not a fit. If this is uncomfortable to you and for some reason you don’t feel like this is a place you’d want to work, then that’s part of the process for both sides. We can end early.”
Kazemi said Porch has likewise had candidates who misrepresented their coding skills. He’s asked candidates to complete coding projects in languages listed as experience on their resume, and has had them reply that they haven’t used that language in years and have forgotten it.
“If someone says, ‘I have all this experience in React’ but cannot talk through different ways to structure components and fetch data, then I kind of have to wonder like, ‘Do they really have as much experience in that area as they say they do?’” Kazemi said.
He’s also had candidates show up and not be able to answer why they’d want to work for Porch. For Kazemi, that’s an immediate disqualifier.
“We definitely want to have people that are interested in the product that we’re building, [and] where we’re going,” he said. “If they don’t have any reason or justification for what they’re doing, then I can’t get that. I can’t even guarantee or have high confidence that it’d be a good fit on either side.”
How interviewers can correct biases
Geisendorfer likes to include a mix of junior and senior engineers in the interview, since he believes strengthening a mid-level engineer’s ability to give small group presentations and navigate uncomfortable social environments strengthens the whole team. Having two people in the room also helps to highlight and correct each other’s biases, he said.
“It’s being thoughtful about who’s actually in the loop.”
“Whether it’s presenting to a small group about a technical idea or other things, it’s a maybe less-than-comfortable social situation,” Geisendorfer said. “I think there’s some pieces of that that are nice to help support a person’s growth. So it’s both to have more folks that can give interviews, involve more people in the process, and then also to grow those engineers.”
Valani said Tableau is also careful about the interviewers it selects to review each candidate — they try to always include women, those from underrepresented minorities, and a mix of junior, senior and mid-level engineers on the interviewing team. All employees at Tableau must also be trained on diversity and inclusion in the workplace, and how to identify and correct unconscious biases.
“It’s being thoughtful about who’s actually in the loop,” Valani said.
Standard measurements keep interviews focused
Our sources recommend making sure the interview process is the same for every candidate. Before a candidate walks through Tableau’s door, Valani said the hiring committee holds a pre-brief to strategize about the interview. During that meeting, the hiring manager updates every member of the team about the candidate’s previous experience, what Tableau is looking for in the open role, what role each interviewer should play, important questions to ask and more. She said making sure all interviewers are on the same page makes the experience more successful for Tableau and the candidate.
“There’s a lot of preparation on our side that goes into making sure we understand the role and why we’re hiring, and [so] we understand the candidate that's coming in.”
“There’s a lot of preparation on our side that goes into making sure we understand the role and why we’re hiring, and [so] we understand the candidate that’s coming in,” Valani said. “I genuinely believe separation is in the preparation, because the candidate is making time to come in to meet with us, and then there’s also an investment on our side.”
After a candidate walks into the office and completes the interview, interviewers at Apex Learning meet for about 45 minutes to discuss how the candidate measured up, Geisendorfer said. He said they look at a standard evaluation matrix the firm has drawn up for helping individuals grow within and across the positions they offer. They use this matrix to evaluate an individual’s competencies — rating them on a scale one to five. Geisendorfer said standardizing the process makes sure everyone’s priorities are in the best interest of the firm.
“There’s a discussion, where folks are talking about what they saw, kind of representing their rating in a verbal way. It’s a chance for somebody to say, ‘I saw this, but I was unsure if it meant this,’” Geisendorfer said. “Just putting some discipline to the process and ensuring that we’re consistently looking for the most important things.”