Will Demand for Data Engineers Fuel a Talent Shortage in 2020?

Demand for Data Engineers has outstripped supply since around 2016. The last few years have consequently seen a chronic shortage of data engineering talent. Many say that the Data Engineer shortage is even worse than the shortage of Data Scientists.

We’ve certainly seen a rise in skills testing for Data Engineers in recent months. This made us wonder if the demand for Data Engineers will continue to fuel a shortage of data engineering talent in 2020. Will data literacy skills still be needed after x years?

In this article, we take a look at the current indicators of demand for Data Engineers that help determine whether or not there is still a shortage of data engineering talent.

First a quick word about what Data Engineers do and know that creates such high demand for them.

Data Engineers make it possible for businesses to engage in all the fancy advanced analytics and insight generation that data science produces. They do this by creating organization and industry-wide access to clean, reliable data at scale with sound data infrastructure and architecture. No easy task!

 

Being a Data Engineer requires an abundance of skills in many cases. If you want to know more about what skills Data Engineers must have, check out our article “Data Engineers Must Be Adept in Several Skills-14 to be Exact”.

Demand for Data Engineers

So what are the indicators of the demand for Data Engineers? Let’s start with something less obvious, but perhaps even more important than job listings for Data Engineers – Google!

Google’s Data Engineer Certification

Have you ever noticed that Google is one of just a few companies that has a Data Engineer certification? Yet, this data goliath doesn’t have a data scientist certification. Why is this? It seems pretty simple to reason that Google recognizes a dire need for Data Engineering skills and that it finds there is a shortage of these. So it decided to create its own Data Engineers.

Here’s a very recent quote from Google Cloud’s Training department:

“With the market for artificial intelligence and machine learning-powered solutions projected to grow to $1.2B by 2023, it’s important to consider business needs now and in the future. We’ve heard from our customers and have witnessed internally that the data engineering role has evolved and now requires a larger set of skills. In the past, data engineers worked with distributed systems and Java programming to use Hadoop MapReduce in the data center but now, they need to leverage AI, machine learning, and business intelligence skills to efficiently manage and analyze data. To address the new skills data engineers now need, we updated our Data Engineering on Google Cloud learning path.”

That’s a pretty strong view that Data Engineers are important!

Other companies have followed suit with creating Data Engineer certifications – Cloudera, IBM, SAS and the Data Science Council of America. All in all, the rise in Data Engineer certification programs by these tech giants is a good indicator of strong demand.

Data Analytics Demand

Here’s a quote from Casey McNamara, a manager of Data Scientists and Data Engineers in a large management consulting firm:

“I had a team filled with data scientists and started shifting to more data engineers. The data engineering team works far better and solves real world problems far faster.

Data science is a real thing – but the world is moving to a functional data science world where practitioners can do their own analytics. You need data engineers, more than data scientists, to enable the data pipelines and integrated data structures.

Smart organizations have smart people who should know their own data. The reason data scientists exist is because most organizations aren’t data savvy yet. They will be.”

Companies are realizing as they try to quickly implement data science and AI initiatives, and most urgently, try to enable self-service data science and analytics across their entire organization, that they need more Data Engineers to make this all possible.

As companies “democratize data science “and create “data citizens” this will only fuel the demand for data and create even more data. And we know what that means – more Data Engineers will be needed!

Data Engineer Job Growth

According to one Data Engineer at the cloud data platform Snowflake, Carlin Eng, companies hiring data engineers set aggressive hiring goals in 2019 to keep pace with big data needs. He says of executives he talked to,

“Most were looking to double their engineering headcount by the end of the year, and more than double the size of their data engineering teams. More often than not, when I asked engineering leaders about their biggest challenges, hiring was number 1 on the list.”

Doubling headcount? That’s a lot of hiring!

A sure signal of this kind of aggressive hiring goal for Data Engineers would be job listings growth. We took a look at recent Data Engineer job growth. Here’s what we found.

DICE’s recent 2020 Tech Job Report reported Data Engineer as the fastest-growing job role in 2019, growing by 50% in 2019. The report also found it takes an average of 46 days to fill data engineering roles and predicted that the time to hire Data Engineers may increase in 2020 “as more companies compete to find the talent they need to handle their sprawling data infrastructure.”

DICE also noted that Amazon, Accenture and Capital One – all companies with deep pockets to pay high salaries – are hiring Data Engineers at high rates.

 

We were curious to know if other salary studies had found similar growth rates in Data Engineer jobs. Below is a summary of what we found.

Source	Timeframe	Y on Y Job growth
Hired State of Software Engineers Report	2019	45% (38% Y on Y in 2017-2018)
Dice Hottest Tech Jobs 2020	2019	50%
LinkedIn’s Emerging Jobs Report 2020	2019	33%
Burning Glass Nova Platform	April 2018- April 2019 June 2018 – June 2019	101% 88%

Clearly, if the Data engineer job outlook looks anything like the last 1-2 years, demand shows no signs of slowing in 2020!

Growth in Big Data Engineering Services

One final indicator of demand for Data Engineers is the growth in big data engineering services provided by consulting firms like Accenture and other tech companies like Cognizant. The global big data and data engineering services market is certainly experiencing high demand. Growth estimates from 2017-2025 range from 18% to a whopping 31% p.a.

Companies are clearly investing in data transformational projects with no end in sight. This signals tremendous future demand for Data Engineer skills.

The Shortage of Data Engineers

Now let’s look at the supply of Data Engineers. Has anything changed regarding the shortage? Here are some factors that contribute to the shortage.

Specialized Training

Data Scientists can reskill themselves from other areas of specialization such as economics, biostatistics, physics, and math. Data Engineers don’t have this luxury of being able to reskill on the job from say, a background in Finance.

Extensive skill sets are needed to become a qualified Data Engineer. It’s not a field in which you can “fake it until you make it.” You need to have many skill sets first. A formal certification or intense self-study program is typically required to move into this field.

Moreover, Data Engineering skills are constantly evolving, and this, in a big way. LinkedIn’s 2020 Emerging Jobs report listed several skills “unique to the job” of a Data Engineer. These included Apache Spark, Hadoop, ETL. New to their list this year was Amazon Web Services. In addition, Google just updated its Data Engineer certification path a few weeks ago stating,

“We’ve heard from our customers and have witnessed internally that the data engineering role has evolved and now requires a larger set of skills.”

Clearly the broad yet highly specialized and evolving skills requirements make finding qualified Data Engineers very difficult.

A Chronic Shortage of Software Engineers

According to a few different estimates, about half of Data Engineers have a software engineering background. Unless you’ve been under a rock, you’ll know that for years there has been a shortage of software developer talent. The unemployment rate for this role hovers around 1.5%.

Typically there are about 500,000 job openings for “software engineers” at any point in time (some predict this will rise to a million this year). Given that universities are only graduating about 50,000 computer science students a year, it would take 10 years to fill all the available software engineer jobs, let alone hire these engineers into Data Engineer roles.

If software engineer jobs keep growing at the 26% p.a. rate that the US Bureau of Labor and Statistics projects, the situation will likely get worse.

 

Add to this huge demand for Software Engineers the need for an additional 120,000 Data Engineers indicated per recent Glassdoor job listings, and another 135,000 Data Engineer job listings currently on Indeed.com and you’ve got a recipe for a fairly serious talent shortage!

Chronic Tech Talent Shortage

If you peel back the onion that is the Software Engineer come Data Engineer talent shortage, you’ll find that it all starts with the general shortage of tech talent. A 2018 survey by Indeed.com found that 90% of business owners are struggling to find and hire IT professionals. Hiring software engineers is the second most common problem they face.

Digital transformations, cybersecurity, cloud migration, app development – you name it. All are competing for tech talent and intensifying the gap between tech talent supply and demand. This dynamic is unlikely to change anytime soon and will continue to drive a shortage in the Data Engineer talent pipeline.

Data Engineer Salaries are Very High

A plethora of salary studies, some of which are summarized in the table below, have shown that Data Engineers are among the highest-paid talent and that salaries continue to grow. In fact, according to Hired’s 2019 State of Software Engineers report, average salaries for data engineering roles increased 7% in New York and 6% in the Bay area, vs. an average 3.5% for the rest of Americans. This is further exacerbating the Data Engineer shortage.

It should also be noted that some salary reports indicated quite low salaries at the bottom of the salary range. We’ll touch on why this is a little later.

Salary Study	Data Engineer Salary Indicators
Glassdoor 2020 and 2018	$102,472 median $137,776 average $110,000-$155,000 range
Robert Half 2020 Salary Guide	$120,000-$222,000
Payscale.com 2020	$91,694 average $65,000-$145,000 range
Dice Salary Report 2020	$113, 249
Emsi Tech of the Future: 10 Emerging Jobs for 2020	$76,500 – $221,500

This is not just a USA phenomenon. A recent Burning Glass study in the UK found that out of 143 skills clusters “Data Engineering” skills had the highest annual median salary and growth in demand.

Data Engineer Applicant Pool

Ori Rafael, CEO of Upsolver, a self service ETL platform, did an analysis based on Linkedin Data Engineer job listings in February 2020. He found the following:

 

This snapshot suggests that relative to other technical and non-technical job roles, Data Engineer candidates are short in supply. If you were a company posting a Data Engineering job on LinkedIn right now, you would be hard-pressed to get a large applicant pool – unless perhaps you were willing to pay top of the salary range.

What Can Be Done?

In the end, by our estimation, the shortage of candidates and tech talent, along with a high projected growth rate in Data Engineering jobs and really high salaries will continue to fuel a Data Engineer shortage.

So how can you find Data Engineers? Here’s a few suggestions based on our experience and research.

It’s Simple Math – Pay Data Engineers more!

We’re not getting any kickbacks from recruiters or Data Engineers by suggesting this. It’s just reality. When we recently looked on Glassdoor.com some Data Engineer jobs were offering pay as low as $35/hour! Other jobs in high-cost cities such as New York were offering just $73,000 a year where the norm for any tech job is well over $100,000.

The most oft-cited reason behind the difficulty hiring Data Engineers is that compensation offered by many businesses to Data Engineer or Software Engineer job candidates is simply below market value.

Numerous articles and anecdotes on forums written by Data Engineers moan about the unrealistic salaries that companies are offering to Data Engineers. This is due to the unrecognized value of what Data Engineers do. Data Engineers are often viewed as doing grunt work. The job is simply not as “sexy” or hyped up as that of Data Scientists or AI engineers. So companies and HR managers who don’t understand the role of a Data Engineer miss the salary mark and then wonder why they can’t hire someone away from their current role.

Compared to some of the salary averages listed in the salary studies above, $35/hour is way off the mark. Moreover, some senior Data Engineers are earning well over $200,000 a year. You are simply not going to get many short in supply Data Engineers to leave their current job with a $70,000 per year offer.

If your company is serious about solving its Data Engineer shortage, whether real or perceived, then you have to offer more compensation. If higher compensation is not possible, find other ways to make your offer more competitive. Data Engineers are highly driven by job content and cool projects. Try to fulfill this kind of motivation. Bonuses, flexible work, important job titles, recognition, fast track career development, etc. are other ideas as well.

Understand How Data Engineers Deliver Value to the Business

One reason that companies try to underpay Data Engineers is that they don’t understand the value of what they bring to an organization. The kingpins data-driven companies like Netflix and Facebook (the largest employer of data engineers) are said to pay some Data Engineers north of $500k. What does this tell you about how much they value data engineering?

Understand that data engineering work is first and foremost engineering work (another trade short on talent). If your company wants to get on the big data superhighway, Data Engineers will be the people building your roads and bridges to data value.

Remember these roads and bridges allow your Data Scientists, Machine Learning Engineers and AI Specialists to do their strategic magic.

Despite this fact, companies think of Data Engineers more like plumbers than civil engineers. They try to pay them as such (no offense to plumbers!) and communicate as such to candidates. So if you are not attributing high value to this function, then you are not attributing high value to big data in general. Candidates will understand this and drop out of your hiring process.

Stop Requiring Degrees and “A-lister” Experience

Mehul Patel, the CEO of Hired said,

“It’s critical for hiring managers to focus on assessing a candidate’s skills rather than over indexing on education. While 50% of software engineers have a computer science degree, another 32% either taught themselves to code or learned through a coding bootcamp — and they may very well have the same set of programming skills.”

It’s a widely known complaint amongst programmers who are self-taught that they have the chops to do the job but are simply overlooked and deemed “unqualified” due to lack of pedigree. If you are requiring a pedigree for your Data Engineering roles, understand that by default you are eliminating potentially 1/3 of the possible candidates.

Because of these kinds of restrictive job qualification requirements, many Software Engineers and Data Engineers feel that there actually isn’t a big shortage of their talent (we still think there is). Some say they’re simply having trouble finding jobs because of outrageous job requirements.

The talent shortage, they argue, is being created by poor hiring tactics such as requiring engineers to pass difficult whiteboard challenges, as if the company has an “A lister” need when in actuality they are hiring for a very basic data engineering role.

Many Data Engineers also feel that the technical hiring process is optimized for the HR department. They say that HR’s lack of understanding about what indicates a good match for a data engineering often ends up with candidates feeling like companies are looking for a “unicorn”.

So, consider relaxing your requirements. Instead, have a plan in place to ensure that less-than-qualified hires can be upskilled quickly. Think mentoring, in-house training and skills assessments, etc. Have a chat with HR and make a plan to change your screening and interview process.

And, if you are one of those companies who are unwilling to pay the higher salaries, then recognize that your unwillingness to pay higher rates perhaps signals that you don’t actually need top tier talent. Maybe you don’t actually need a Facebook level of Data Engineer. If that’s the case, make your job requirements reflect this.

Conclusion

It would appear on many accounts that the demand for Data Engineers shows no signs of slowing. If your company is struggling to hire data engineering talent, recognize that there is a clear shortage of this talent – and it may get worse this year. Recognize too that this shortage might not be the real root of your hiring problem. Ask yourself if you really understand the value of what Data Engineers bring to your organization and its data science and AI plans. Understand that data engineering work is hard, complicated, and can be incredibly frustrating. Appreciate what this kind of talent brings to the table and use it to inform how you go about adding data engineering talent to the team.

If you’re looking to hire Data Engineers, contact us to learn about how to assess Data Engineering skills: sales@quanthub.com.