Sign in
In Part 2 of our series on skills assessments , we’re looking at the issue of false negatives in tech and data talent recruitment and what roles skills assessments play. We cover why false negatives really do matter in tech recruiting, what drives this type of hiring error and what can be done to reduce a company’s chances of rejecting top applicants. (5 minute read)
Tech Giants Talk False Negatives
“Google has a well-known false negative rate, which means we sometimes turn away qualified people, because that’s considered better than sometimes hiring unqualified people. This is actually an industry-wide thing, but the dial gets turned differently at different companies. At Google the false-negative rate is pretty high.” – Steve Yegge, Senior Staff Engineer and Manager at Google
“There are some legendary-ish tales of me not hiring people because they used the wrong word in an interview… I’m sure we had lots of false negatives, but we have very few false positives.” – Max Levchin, cofounder of PayPal
“If you reject a good candidate, I mean, I guess in some existential sense an injustice has been done, but, hey, if they’re so smart, don’t worry, they’ll get lots of good job offers.” – Joel Spolsky, cofounder of StackOverflow
False Negatives – A Common Reality in Tech Hiring
These quotes from tech industry leaders are indicative of the prevailing strategy in tech and data talent recruitment – be really picky during your recruitment and make the interview process tough. Companies systematically aim to be selective, and in doing so implicitly accept that they might pass up great candidates – a result called a false negative.
It’s our experience that data science and analytics managers are quite concerned about false positives when recruiting members to their analytics teams. They seem less concerned with missing out on great candidates. This sentiment was echoed by one data science manager who recently told us, “I’d rather not hire than make a bad hire”.
Does this philosophy resemble your analytics team recruitment experience? In general, we are finding that companies view a skills assessment platform like QuantHub as a means to weed out less talented individuals and reduce the chances of a bad hire. Less often, they view skills assessments as a platform to give talented candidates a way to demonstrate what they can do.
Why False Negatives Matter
So why does this matter? Well, if technical and data science interviews are skewed toward avoiding false positives, this could be one reason talented data scientists and data engineers are so frustrated with the recruitment process (more on this later) and complain of having trouble getting a job in a market that has an acute skills shortage. Perhaps the heavy propensity to screen out candidates is contributing more to the skills shortage in the analytics field than we think. If your recruitment process is eliminating qualified people to a high degree, then consequently your talent pool may seem artificially limited.
It’s understandable that everyone worries about hiring the wrong person because a bad hire is particularly troubling for high intensity, project-oriented data science and engineering teams. And bad candidates come with big price tags! Nevertheless, a missed top hire in this industry can be a real loss to the company or team. This is especially true for smaller companies and startups.
And let’s not forget that it’s become difficult to retain data science talent because of the constant poaching from recruiters. You can bet that top people who are rejected after a tough phone screen by one company and who are subsequently hired by another reputable company are getting hit up by the former company’s recruiters a year or two later. You have to wonder how often companies are trying to hire away candidates they previously passed over.
Why False Negatives Are Common
The “B” Word
In his book Work Rules! Laszlo Bock, former VP of People Operations at Google describes interview bias:
“Based on the slightest interaction, we make a snap, unconscious judgment heavily influenced by our existing biases and beliefs. Without realizing it, we then shift from assessing a candidate to hunting for evidence that confirms our initial impression.”
The existence of recruitment and interview bias has been well documented by academics and industrial psychologists. Much has been written about how the first five minutes of an interview are what really matters because interviewers make initial judgement and then spend the rest of the interview working to confirm those assessments, good or bad. Several other types of bias can manifest during an interview as well.
There’s no doubt that solid data science candidates are being screened out for the plain and simple reason that the folks in HR or a future team member conducting an interview made a quick determination based on personal preferences.
Perhaps the clearest evidence of bias in data science recruitment are the statistics on minorities in the field. As the figure below and many studies show, women represent around 20% of the industry while Latino and African Americans combined make up less than 11% of data scientists. Hired.com’s 2019 Salary Survey also recently revealed that 41% of tech job interviews only interview men.
Aline Lerner, co-founder of Interviewing.io, a platform where candidates can practice technical interviewing, identified that companies who rely on over challenging technical interviews are are, “pumping resources into finding diverse candidates — who don’t understand the game-like nature of interviewing — and dumping them into a broken, nondeterministic machine.” Ouch!
The primary reason for bias is the nature of unstructured interviews, an issue that we addressed in our previous article. But even today’s sophisticated AI-driven recruitment tools are guilty of bias. Everybody knows the infamous story about Amazon’s AI resume screener filtering out resumes from women. It also failed to recommend strong candidates And in today’s age of applicant tracking systems, bias is introduced from sometimes arbitrary requirements, such as special degrees or years of experience, decided by someone in HR to be used as a screen.
With this kind of evidence it’s not a stretch to assume that companies are rejecting qualified candidates due to some kind of bias.
Unicorn Requirements (yawn)
Although it seems as if companies are becoming more savvy at creating data science job descriptions, a recent study by IDC indicates that companies still need to work on getting the data science job description right. This study showed that the top challenges data scientists face are that they must overcome skills gaps and learn to use multiple complex tools, while also spending a lot of time on data searching and data preparation activities. Companies are still expecting data scientists to do everything. By interviewing candidates with such unrealistic expectations, they will inadvertently eliminate people who have most, but not all the requisite and highly diverse skills.
What’s worse is that the highly qualified people who actually meet a lot of these extensive skills requirements know that a company with such an unrealistic job posting and subsequent over challenging skills screening is not a place for a talented and in-demand data professional. So they will seek employment elsewhere.
And so once again, top candidates are lost.
The HR Disconnect Lives On
We’ve been told by data team leaders and recruiters such as Burtch Works that hiring managers continue to struggle to get non-technical human resources personnel to understand, advertise and correctly screen for their specific data science experience and skills requirements. This disconnect could potentially lead to some top candidates being overlooked due to poor role descriptions and skills requirements and subsequent mismatched screening.
The fact is that HR personnel often can’t tell who is a good technical candidate and who isn’t because they don’t have those skills. They can’t necessarily tell the difference between a good bootcamp and a bad bootcamp. So the default answer for anyone without a degree or who has a qualification that HR doesn’t recognize is “no”. HR will instead screen for schools because that is what they understand, and in doing so eliminate a potentially significant number of solid applicants.
Further exacerbating this problem is that, not trusting Human Resources to properly screen hundreds of Indeed.com job applicants, hiring managers turn to the back door recruitment method, personally scouting candidates at meet ups and through industry networks. This further frustrates the dozens of qualified data science applicants being screened by HR who don’t get an interview. Word then spreads about the difficulty of getting an interview or call back from the company, and so the company’s ability to attract candidates may suffer.
Too Many Applicants
The rise of bootcamps and newer university analytics degree programs has resulted in a flurry of people rushing to label themselves a Data Scientist in order to earn the lucrative salary that this title brings. This has resulted in a rise in entry level data science applicants. Companies have thus taken the approach to develop complicated interview processes to weed out the least experienced and “fake” data scientists.
The issue is that some of these tactics are so unpleasant or unrealistic that they discourage great candidates. Companies give candidates tough white board challenges that have nothing to do with the particular data tools the company uses or send candidates home with a project that requires several days’ effort to complete. Other unreliable screens used to cut down on applicant numbers may include rejecting anyone who hasn’t participated in a data science competition, thoe who don’t have anything on GitHub, or who don’t have a certain academic pedigree – anything that reduces the number of applicants HR must sift through.
This kind of obstacle course drives away qualified candidates who know they’re in high demand, and so they look elsewhere. This brings us to the next reason the false negative rate is potentially high in data science.
The Technical Interview is Broken
A recent Reddit thread was titled “The data science interview process is terrible.” Hiring managers should have a read through these kinds of threads to understand what is going on. In this thread, a “real” Data Scientist with a Masters in Statistics and 3 years of experience was bemoaning that she couldn’t get past the take home tests, some of which took 20-30 hours to complete and which also provided valuable code for “free” to prospective employers. The applicant laments “I feel like my studies and experiences aren’t worth anything”. The 145 responses that ensue are indicative of the many problems with the current state of technical and data science interviews. For starters, this is a female Data Scientist. Remember that 20% statistic? Surely companies must be looking to create diversity in their team?
Many in the tech industry have proposed that the “broken” technical interviewing process is in part responsible for the current hiring crisis in data science and software engineering. Interviewing.io found that people who are overall pretty strong can mess up technical interviews as much as 22% of the time, probably even more often. When faced with a command such as “Alright, why don’t you head on up to the board and show me how you’d balance a binary search tree?” some top candidates who feel out of place at a whiteboard facing a high pressure or unrealistic challenge might give the wrong signal in that one instance. And so they may indeed be passed over.
Candidates also constantly complain about having greatly varied interview experiences within the same company, receiving what seem like “pet” interview questions by each interviewer that have little to do with the day to day work or anything else for that matter. Google, who infamously used case studies and brain teasers in its past tech interviews, eliminated these years ago after internal studies proved that they were ineffective at predicting candidate success.
For sure, traditional technical interviews are more prone to false negatives than false positives. This is because historically hiring one bad engineer was viewed as worse than failing to hire two good ones. But good data scientists and data engineers are so scarce these days, this attitude really should no longer apply.
Can Skills Assessments Help Avoid False Negatives Tendencies?
We laid out in our previous article how skills assessments might help companies prevent false positive hires, while also pointing out where they fall short. We would argue that skills assessments also have a role to play in helping companies to avoid eliminating top candidates due to biased recruitment, misguided screening tactics and so forth. Here’s how.
Skills Assessments Allow Companies to Cast a Wide Net
According to a Bullhorn study, 40 percent of firms don’t automate candidate selection, screening, or nurturing. And according to the Sourcing Institute the ideal number of results from a sourcing search is around 300. That would be a lot of applications to sift through manually for the 40% of companies who don’t automate their recruitment. This challenge introduces bias and unnecessary screens to narrow down the pack.
An online skill assessment test like QuantHub helps companies recruit at scale and attract and vet large volumes of applicants. With assessments in place companies know they have one relevant, standardized way to identify early on in the process who has the necessary skills among a large group of candidates. They can easily start narrowing down the interview list to top candidates early in the hiring process.
Skills Assessments Allow Non-Technical Personnel to Test Job Skills Upfront
Going back to the non-technical HR staff or recruiters’ unreliable screening methods, these employees don’t need to have technical knowledge to be able to send candidates an online technical skills assessment. They can simply email a link or invitation to each candidate and ask them to take the skills assessment. The platform then automatically sends scores to the candidates’ ATS file. An additional advantage of using skills assessments in in this way is that it frees up valuable engineer and data scientist’s time that would normally be used for skills screening.
Skills Assessments Eliminate Human Bias
Without human bias, there is little margin for error in determining a candidate’s knowledge or skills in tested areas. Even if you don’t believe assessments to be entirely unbiased, at a minimum, skills assessments add more rigor to hiring decisions and provide a second set of eyes that is much less prone to bias than human eyes. At a larger scale, skills assessments can help standardize the interview process and eliminate bias across many functions and levels of a company.
Skills Assessments Provide Accurate Skills-Based Screening
Just as a résumé may not always reflect the true weaknesses of the candidate, it may also not highlight the strengths. Resumes are often taken at face value and used to screen out candidates using irrelevant keyword tactics. Skills assessments provide an alternative method to resumes for determining skills, strengths and weak areas. The screening becomes more skills-based rather than keyword-based.
Skills tests can also enable a candidate to showcase their exemplary skills in certain areas. With QuantHub, candidates are tested in a variety of areas from statistics to neural networks and the testing platform highlights strong and weak skills areas. Thus, a quantitative pre-hire assessment can provide candidates with an opportunity to shine in a way that may not come across in an interview or resume.
Skills Assessments Retain More Qualified Applicants in the Funnel
Kevin Parker, CEO of HireVue Parker said
“Speed and agility matter more than ever in landing top talent. The data shows that the best talent is off the market within 10 to 15 days for roles in many industries.” This is certainly the case for data science and analytics. So it’s critical to keep qualified candidates in the recruitment funnel.
Remember that Reddit user‘s complaint that she was often asked to do 20 hours of work on take home tests? Linda Burtch, CEO of Burtch Works recruitment firm recently told us that long tests for candidates who are working full time is bordering on excessive. She has seen quality data science candidates drop out at the point of being asked to take so much time to prove their capabilities.
Online skills assessments provide a less laborious, more efficient, but nevertheless equally solid alternative to take home tests and extensive white boarding sessions. A candidate can take it from their desktop at their own pace and on their own terms.
Minimizing False Negatives
The key in data science recruitment is to find a process that minimizes false positives and avoids costly bad hires, while not causing an extreme of false negatives that creates a skills shortage for a company, or an industry at large for that matter. Here are a few suggestions for how to find that balance.
- Develop realistic screens and interviews that fit the job market – Give technical assessments and questions related to the actual job and which respect candidates’ time. Consider eliminating years of experience in the equation and replace that with demonstrable proof of solving business problems. Know exactly who you’re looking for and what the market looks like for people with those skill sets. Have more than one person in an interview and use structured interviews.
- Focus on giving candidates a reason to join your company – Rather than trying to proactively screen out people, make clear why the work they will do doing matters. This is very important to top talent. A great way to engage candidates is to show them the type of projects they will be working on. Instead of using interviews to set up obstacle courses to prove that a candidate is not worthy, use them to enable candidates experience the great people they will get to work with. Then see how they respond.
- Hiring managers do your own outreach – But submit your “pet candidates” to the same structured interviews and testing that other candidates would be subject to. Likewise, don’t let favorite candidates get favorable treatment versus other candidates who may be more qualified. If they do poorly on an online skills assessment, don’t just ignore this fact.
- Get job role savvy – Split that one data science job into separate roles, and/or determine which skills are deal breakers versus can-learns. Qualified applicants will see that you have realistic expectations, so they will be more likely to apply or stay in the funnel.
Many aspects of data science recruitment and hiring are hard to control, but the rate of false negatives is one that could arguably be controlled better. It’s a question of priorities and how you manage the recruitment and hiring process for data scientists. Given the evidence that the current interview process is geared towards avoiding false positives, companies may be missing out on potentially great hires (and diversity opportunities) more than they think.
False positives may continue to be the focus of most technical and analytics hiring managers, but they should remember that many questionable new hires can be redeemed with time, no matter how under-equipped they may seem at first. But great hires that you haven’t made are gone forever, or may only come back to you later in the form of a very expensive recruit. Good data science talent is hard to find. Don’t eliminate more of them than you need to.