Sign in
Since December 2013, data science postings have rocketed 256%—more than tripling. – Indeed Hiring Lab, January 2019.
Demand for data scientists is booming and will only increase. The supply of skilled applicants, however, is growing at a slower pace. – Search Business Analytics, January 2019
Data Science has taken off like discounted Nutella in a European supermarket – Clint Chegin, Product Scientist at Indeed
We’ve all seen stats and headlines like these over the last couple of years. They’re usually accompanied by some incredibly high salary indicators (median salary of $95,000!) and descriptive such as “sexiest job of the 21st century” and “hottest jobs”. As in the above-cited Indeed article, the discussion that follows these headlines revolves around the chronic shortage of data scientists.
We know – Data Scientist shortage. Yawn.
At QuantHub we’re all too well aware of the #quantcrunch. We’ve made it our company’s mission to help companies solve their Data Scientist shortage and build up great advanced analytics teams.
Not too long ago, we penned a piece about the merits of having data science masters degrees. At the time, most degrees offered in data science and advanced analytics fields were master’s level. But the landscape has been changing dramatically since then. In less than a year, everyone from Auburn University right here in QuantHub’s backyard, to New York University is offering undergraduate degrees in advanced analytics and data science-related fields.
So, it begs the question: Is there STILL a shortage of Data Scientists?
We wondered this ourselves. After all, our customers come to us because often they are struggling to find the right candidates for their data science needs. So, we did a little research into the issue to try and determine if things are still as bad as these headlines indicate.
There is still a shortage of Data Scientists because…
“Indeed”, Data Scientist job postings are still growing faster than data science job seekers
According to Indeed, Data Scientist job postings (as a share of total job postings) rose 31% in December 2018, compared with the same period the year before. Yet data science job searches as a share of total job searches rose 14% in 2018 over the same period. The gap between these two rates of growth constitutes a “shortage” according to Indeed. There are still not enough data science candidates to meet the growth in demand.
As the chart below indicates, there are roughly 1,000 Data Scientist job postings per million job postings, but only about 600 Data Scientist job searches per million. Regardless of exactly how many applicants per job posting might be received, in this scenario the bargaining power still rests with the applicant. Therefore, this situation might still create a shortage for many companies.
Why? Well for starters, many medium and small companies may not be able to afford the high salaries to woo the best candidates. At best, it will take longer than usual to fill a data science job.
LinkedIn says it’s true too
In its August 2018 Workforce Report, LinkedIn reiterated that “demand for Data Scientists is off the charts”. One little known fact is that as of 2015, there was (gasp) a national surplus of Data Scientists – can you believe it?!? Believe it or not, employers in big cities like Atlanta had their choice of Data Scientist candidates in 2015. But after 2014, large companies in all industries began ramping up their big data analytics and data science.
Now, says LinkedIn, the landscape has changed dramatically and there is a shortage of data science talent in nearly every major US city.
The exact shortage? By LinkedIn’s calculations, as of July 2018 the shortage of Data Scientists was 151,717.
Here’s the chart depicting the change in major cities from 2015-2018. It’s hard to believe that in Austin, Texas in 2015 there was a shortage of just 26 Data Scientists, whereas in 2018 it was 5,000. That’s some serious job growth!
Source: LinkedIn
$50 bootcamps are not enough
As in any marketplace where demand exceeds supply, innovators enter the market with creative solutions. In the case of data science, the market has been flooded with all manner of bootcamps, certification courses and e-learning options. But some ask can you really skill up in data science with a $50 Udemy course?
Many would argue that being a good Data Scientist requires a significant amount of education and training on the job. Experienced Data Scientists and employers alike say that completing a few bootcamp projects is not enough to be able to do the many tasks expected of a Data Scientist and take on the role of driving potentially very expensive business decisions.
One telling letter from a self-trained aspiring Data Scientist to a Senior Data Scientist mentor highlights this issue:
I think my lack of academic pedigree is really what is killing me. It is not really skills (though they really need a lot of work and I am doing that). I am not even getting the interviews to show my skills so that’s why I say that.
I had an in person with [BigCo] and it was my first time ever doing in person coding or a whiteboard so that didn’t go well.
I had a take home that was a survival analysis from [big startup] and I had never studied that so that didn’t go well
I had a take home from [BigCo] that got me an in person and they passed because of my education (which is stupid since they had my resume already).
Despite an influx of junior level candidates, high pay data science skills are still in shortage
The highest-paid Data Scientists have highly specialized skills that set them apart from others in their field. These roles are in high demand but cannot be filled by undergraduates with no experience. For instance, Data Scientists in Finance fields like Insurance, who possess the range of both technical, soft and industry-specific skills to interact with the analytics teams as well as the C-suite, are still in great shortage.
In our article on master’s degrees, we cited Burtch Works’ study which asserted that employers are seeking Data Scientists with more specialized experience such as NLP and Image processing. Last we checked there were few undergraduate degrees specialized in these fields.
Finally to drive this point home, according to KPMG’s 2018 CIO Survey, big data and analytics was the number one place for skills shortage for the fourth year running. 46% of CIOs said they suffered from a big data and analytics skills shortage vs. 36% in 2015. This was seconded by a shortage in AI skills for 38% of CIOs. So, for half the companies out there it would appear that the shortage of true data science skills is real.
Companies still seek unicorns and mismatch skill sets
According to an IADSS study of 2018 LinkedIn job postings from 200 Fortune 2000 companies there is a clear gap between the knowledge/skills employers are seeking in Data Scientists and what true data science and analytics professionals put on their resumes. The real gap, they say, stems from how organizations define data science roles versus how individuals position themselves in the data science job market, and the variance in how analytics skills and knowledge can be measured.
As long as companies continue to focus on very specific skill sets such as Hadoop or R, and omit important skills such as data visualization, there will continue to be a perceived shortage for individual companies who don’t get their data science role description right and who continue to expect individuals to be adept at many very specific and sometimes unrelated skills.
The predicted need for AI skills
According to our friends at LinkedIn, AI skills are among the fastest-growing skills in demand, with a 190% increase from 2015 to 2017. And where are they most in demand? You guessed it – San Francisco, Austin, Denver, New York.
Going back to our previous point, machine learning/AI skills are highly paid for skill sets that in many cases cannot be fulfilled by undergraduates on day one. With this kind of growth trajectory expected to continue, it’s not a far stretch to expect a skills gap.
The same LinkedIn study also notes that among skills with the biggest gaps, development languages and operating systems like Java and Linux, which are required of some data science jobs is #3 in terms of skills shortage. Data science skills themselves ranked #9 in terms of a shortage.
And once again, Indeed agrees with LinkedIn, as is depicted in their graph below which shows the gap between jobs requiring AI-related skills and people with those skills seeking jobs.
AI-related jobs include machine learning engineer, predictive modeler, cmt analytics manager, data scientist, computer vision engineer, computational linguist, and information strategy manager. Source: Indeed.com
The Data Scientist gap is narrowing because…
Recently, there’s been quite a bit of speculation that the Data Scientist shortage is being eliminated. Let’s look at the arguments in favor of this conclusion.
There’s an influx of data science candidates
As mentioned before, the number of data science undergraduate programs, certifications, bootcamps and online courses has exploded. There are now hundreds of data science undergraduate programs, certifications, bootcamps, and online courses exponentially improving data literacy across the globe.
Looking at the hundreds of thousands of aspiring Data Scientists are fueling this new industry. Some data science managers claim to be seeing a 5-fold increase in the number of people applying for junior data science roles. College students majoring in computer science are also on the rise and growing by as much as 24%. All of this points to a potential flooding of Data Scientists into the job market.
Data Scientist salaries are flattening out
In TechRepublic’s recent article titled, “Is it still worth becoming a data scientist?” (FWIW, um, yes it is), Glassdoor is quoted as having determined
“As data science has risen in prominence, enrollments in data science programs and bootcamps have exploded. This huge increase in workers for limited entry-level jobs is holding down wages,”.
They say that salaries for Data Scientists shrank 1.2% year on year as of March 2019 because “it’s clear that there are now more candidates for data scientist roles than there are jobs available.” Yikes, really?
You can go recruit in cities that have surpluses
According to LinkedIn, while most major cities have chronic shortages of Data Scientists there are some metropolitan areas that do have surpluses. These include: Cleveland, Akron, Birmingham (home of QuantHub!), St. Louis and several others.
So what do the numbers say about the shortage of Data Scientists?
Just for kicks, we surveyed the internet to find as many “calculations” of the Data Scientist shortage as we could over the years. Unsurprisingly, these numbers indicate that the shortage of data science skills is not only real, but it has probably grown. It remains to be seen whether by 2020 or beyond higher education, employee training and other measures can begin to fill the gap. We’ll certainly keep you posted.
What this means for Data Scientists
Despite Tech Republic’s article, the answer to their question is “yes, it still pays to become a Data Scientist”. With a median salary of $95,000, a 1% decrease does little to move the Data Scientist needle.
That being said, you’re going to have to really stand out if you are applying for any junior data science roles. Ways to do this would include specializing or going into a related field like finance first. Many Data Science managers who are doling out career advice often recommend getting into Data Science “through the backdoor”. That is to say, by going into a related field such as marketing analytics, economics or just pure computer science and then working your way into the field rather than competing outright for a junior position.
Finally, for those who are self-training, it is important to keep your eye on the prize and recognize that practical experience, not just a bootcamp degree is in many cases what you need to get an interview. So participate in one of the many types of data challenges/competitions online. Do multiple internships. Freelance. Get a job in a related field that will allow you to build up a portfolio. Be realistic about your options.
What this means for companies
While there might not be an applicant shortage at the junior level, there is still a significant data science skills shortage and competition for top talent that does not appear to be letting up for the foreseeable future. And with the need for AI and machine learning skills on the rise, companies will have to get creative to fill the gap.
Some steps you can take include:
- Widen your recruiting pool – look inside the organization for employees who have the capacity to skill up. Consider a candidate with prior work experience but who has “only” a boot camp degree. Recruit from related fields such as finance and healthcare informatics.
- Boost retention rates – If you have Data Scientists it is critical to keep them. Strategies for doing this include facilitating team learning, investing in career and skills development, providing a solid pipeline of challenging projects and ensuring that Data Scientists are recognized by top management for their contribution to corporate results.
- Democratize and cross pollinate data skills – Encourage members of your data science team to pick up skills from other members. Encourage them to be responsible for educating business people on data science and transferring skills when possible to related departments such as data engineering or finance. Over time, this creates organizational flexibility. These tactics can also create a business that is more resistant to attrition.
- Invest in training and development of junior Data Scientists – In our article “Building Data Science Capabilities Means Playing the Long Game” we advocate that companies take a long term view towards developing data science and analytical capabilities. In particular, it is important to recognize and hire people who can learn and grow by partnering with university programs, providing internships, mentorships, etc.
Bottom line, the skills shortage is still real
Whichever side of the #quantcrunch argument you’re on, one thing is for sure: there is a definite and still growing need for quality data science and analytical talent for the foreseeable future. Yes, the shortage is hyped, but the shortage of experienced Data Scientists is real too.
Do you agree or disagree there is a shortage of data sciences skills and talent in 2019? Let us know your experience!