Sign in
In this two-part series, we’ll look at each side of the false positive and false negative coin, starting with false positives. We’ve observed that once a hiring manager gets burned with a bad data science hire, they begin to distrust the pre-employment skills testing and the overall interview process. Some even admit to becoming very very tough on candidates, rejecting most of them in order to protect their data science team.
One such manager is Rod Whisnant, Senior Manager of Data Science and Machine Learning at Cubic. After experiencing a bad data scientist hire early on in his career, Rod told us that his recruitment philosophy evolved to “no hire is better than a bad hire.” What his reaction tells us is that bad hiring experiences in data science could be contributing to the large number of data science candidates who wonder why they can’t find a job in this “hot” market.
So in this article, we look at the issue of bad data science hires, or false positives, and what role technical pre-employment skills tests may or may not play in the unfortunate result. In a subsequent article, we’ll look at whether pre-employment assessments play a role in false negatives in the data talent hiring process.
What’s a False Positive?
Since we are talking data science, for you non-technical readers, here’s a quick reminder of what a false positive is in statistics:
A Type I Error, or False Positive, is a result that indicates that a given condition is present when it actually is not present. (source: Datasciencecentral.com)
In other words…
In terms of data talent hiring, a false positive would mean that the recruitment process and/or pre-employment skills testing identified someone as being qualified for a particular data science or engineering role when in fact they are not.
The Big Deal About a Bad Data Science Hire
A bad hire is not good in any industry. But it’s particularly bad in data science due to the high costs involved. Hiring an ill-matched data scientist or data engineer can be a real setback for any organization.
How much of a setback it is can be summed up in three data points (since we’re talking Data Science): $100,000+, $15,000 and 30%.
These three numbers represent the cost of a bad data science hire.
- $100,000+ is the expense of recruiting, hiring, eliminating, and replacing a bad data science hire.
- $15,000 is the average internal tech screening/interview costs for a single data science hire.
- 30% is the average churn rate of data scientists.
For smaller companies and data science teams, such numbers can have dramatically negative effects. And for data science consulting and other professional services firms, such high churn rates can wreak havoc on client projects and relationships. Moreover, the people doing the internal tech screening/interviewing are likely some of the most skilled and highly billable team members.
It’s understandable why companies fear hiring the wrong data scientist and putting too much trust in any one interview technique or skills test. So, let’s look at the pros and cons of standardized skills assessments to help determine whether or not they contribute to the risk of a false positive data science hire.
How Pre-Employment Tests Can Help Avoid False Positives
1. Pre-employment tests are objective
Organizational psychologists have shown[1] that unstructured interviews and verbally administered skills tests are inherently biased and poor predictors of performance. Whether it’s a recruiter or a future teammate doing the interview or testing, studies have shown that each will have their own subjective criteria against which they will judge a candidate’s responses.
2. Pre-employment tests are fair
Other methods of candidate assessment can be unfair if interviewers ask different questions to different people and in different ways. Likewise, unstructured skills “checks” may vary from one candidate to another. It becomes difficult to rate candidate responses in any comparative way.
Peter Cappelli, professor of management at Wharton, suggests in his HBR article “Your Approach to Hiring is All Wrong” that “The best interview strategy is to ask all applicants the same set of predetermined questions. That way answers can be fairly compared.” He further suggests that testing applicants at skills required for the job is probably the best approach to doing this.
The reason is that skills tests are standardized and administered in the same way for all candidates. In the case of skills assessments like QuantHub, the tests are further constructed in relation to the job criteria. Everyone has the same opportunity to succeed or fail in demonstrating their skills fit for a particular role.
3. Pre-employment tests are strictly job-related
Following on the previous point, skills tests are designed to focus on skills required for the job. In interviews, often there may be no pre-determined questions to ask. The interviewer may prefer to have a conversation about any number of topics. Such casual conversations tend to not guarantee job success even if they go well.
To this point, Dr. Tomas Chamorro Premuzic, Industrial Psychologist and Chief Talent Officer for Manpower Group concludes in his book Why Do So Many Incompetent Men Become Leaders?: (And How to Fix It):
“Most companies focus on the wrong traits, hiring on confidence rather than competence, charisma rather than humility, and narcissistic tendencies rather than integrity, which explains the surplus of incompetent and male leaders.”
It also explains the existence of poor data science hires.
4. Pre-employment tests are quantifiable
Sometimes experienced managers prefer to go on gut. After all, they have lots of experience so why not trust it? The problem with using your gut is that gut decisions are based on feel rather than facts. A clever candidate may say what he thinks an interviewer wants to hear. He may talk a lot about a skill set in which he has a good deal of competence, while masking or underplaying the fact that he has weaker skill areas.
Rod Whisnant tells us that his bad hire had a Master’s in Statistics and was supposedly working on his PhD. At the time of the interview, Rod had less experience and knowledge than the candidate he was assessing. This was one reason he says that he felt compelled to hire the candidate, even though there were red flags. Rob describes how the newly hired candidate behaved afterward, “He went walking around the company like he was the most important person…but he never built anything concrete. I had to put him on PiP (performance improvement plan) three times and eventually he quit.”
Rod’s experience is not uncommon. A study by Ere Recruiting found that 9 out of 10 hiring managers with failed hires reported that they had experienced distracting feelings during interviews, such as feeling overwhelmed at having to fit interviews into an already busy schedule or fearfulness over losing a good candidate to a competitor (a familiar feeling when hiring experience data talent). They also found the most frequently mentioned emotion managers experienced was candidate likability. The hiring managers liked the individuals, which allowed them to overlook negative traits. This liking turned out to be the “false positive” that led to the bad hire.
Technical skills assessments with their directly measurable and comparable scores avoid these traps by proving specific and concrete reasons for hiring or rejecting candidates. Therefore, they can replace or supplement intuition and feelings.
How Pre-employment tests Can Contribute to False Positives
1. Pre-employment tests don’t provide a full candidate picture
Assessments evaluate a specific set of qualifications and skills such as Python programming or math. However, they can’t indicate, for example, a data scientist’s willingness or ability to improve their skills. Likewise, a candidate may demonstrate great problem-solving skills on a skills test challenge, but not actually be all that curious of a person – a critical problem-solving quality needed for most data science positions.
Now, with more hiring experience under his belt, Rod Whisnant looks for signs of motivation in candidates early on in his screening. For instance, if a candidate has been job hunting for three months and in the meantime hasn’t been working on any side projects, skill building or challenges, Rod is unlikely to give their skills the benefit of the doubt.
Therefore, skills assessment results on their own won’t be able to tell you if a candidate is the right person for the job.
2. Pre-employment tests can invite cheating or lying
It’s hard to envision that someone applying for a technical job would dare cheat on a data science or technical skills test, but the fact is that not all cheating can be monitored on many types of job skills tests. For more senior data scientist positions, this would be harder to do of course and less tempting given the level of experience. However, for increasingly competitive junior or entry level technical positions, the temptation is there.
Rod Whisnant’s anecdotal experience shows that the propensity to fudge data science skills is there. “Sometimes (in a phone interview) I’ll ask a question and get silence for a while. Then suddenly the candidate comes back with an answer and I’ll think, hmmm, sounds an awful lot like Wikipedia.”(laughs)
3. Pre-employment tests don’t necessarily create diversity of skills
A much-needed change in data science recruitment is to recruit for a diversity of skills. If you are measuring and comparing all candidates based on the same skill set test, this doesn’t necessarily recognize that individuals may have other valuable skills that do not show up on that assessment. Rod’s team of seven machine learning engineers at Cubic consists of a geologist/physicist, a signal processing engineer, a PhD in Applied Mathematics and a graduate in Computer Science fresh out of college. “You’re never gonna find the perfect fit” he acknowledges, adding that he looks for a diversity of skills and accepts that diversity is necessary to complete his team.
So how do you avoid false positives and costly bad data talent hires?
There’s a lot of advice out there about how to avoid hiring the wrong data scientist for the job and how to spot a fake data scientist. In Rod Whisnant’s case, he says that there were three contributing factors to his poor hire:
1) Pressure to make a hire quickly before the annual budget ran out
2) Acquiescing to the candidate’s pressure to be offered a full time position, despite Rod wanting to try him out on a contract basis first.
3) Lack of experience and uncertainty on his part as the interviewer, which caused him to ignore red flags.
Given such difficulties that many hiring managers no doubt face, we’ve put together a few suggestions for avoiding a bad data science hire:
- Revamp your interview process – Recognize where it is biased and unfair or unorganized. Do you have standard questions that are asked of every candidate? Or, could subjective criteria such as the fact that the candidate graduated from the same school as the interviewer be clouding judgment? Does each interviewer in the process know their role in the process and what they should focus on in the interview, or are they each making that determination for themselves?
- Test for broad skills as a screen to reduce your applicant pool – If someone says they are a “data scientist” they must be, right? Wrong. Check for basic data science skills, like math, and separate the wheat from the chaff. Or give every candidate the same standard data science challenge. You can use a standardized skills assessment like QuantHub test to achieve this.
- Test for specific job role skills for every applicant in the same way – Don’t ignore the results! Don’t give a skills assessment and then find reasons to still include someone who did poorly on the job skills test, i.e. because they are a “cultural” fit or graduated from a certain degree program. This defeats the purpose of the test.
- Track your hiring process and data science hires over time – Track who stays and who leaves, when and for how long. Figure out why your data scientists are leaving or why they were a bad hire. Is it the skills test results, or something else? You won’t know unless you track this information.
To this effect, Rod Whisnant has been evolving his recruitment and screening process over the years. He’s happy to report that two and a half years into his career with Cubic, his data scientist and machine learning engineer recruitment process has resulted in no false positives and that his team is “heads down building products and solutions.”
So if you use data skills assessments to recruit your data scientists and data engineers will you continue to hire a bad data scientist here and there? Perhaps. We’re not suggesting that skills assessments are the end all be all of hiring data talent. But there’s plenty of evidence suggesting that at least trying some pre-employment skills testing could make a difference in helping to avoid a bad data science hire.
What’s your experience with and thoughts on false positives in data science hiring? Are you doing standardized skills assessments? Have you seen an improvement in your recruitment and hiring results? Let us know!
(We’d like to thank Rod Whisnant for his willingness to share his experiences and insights for this article. For more information about Rod, visit his website: https://www.rodwhisnant.com/)
[1]Some examples are: https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1744-6570.2010.01172.x and https://hbr.org/2019/05/should-companies-use-ai-to-assess-job-candidates and https://hbr.org/2019/06/will-ai-reduce-gender-bias-in-hiring