The State of Data Engineering 2020

For years the humble hard working Data Engineer has been busy working in the shadows building data highways and bridges to clear the way for the revered Data Scientist to do their magic creating big value out of big data.  It’s possible that in 2020 and beyond, Data Engineers will finally get their day in the sun.

Back in 2016 a famous Stitchdata report found there were only 6,500 Data Engineers in the global workforce, yet there were over 6,600 open job listings for Data Engineers in San Francisco alone.

A shortage of Data Engineers was evident even back then.

As the matrix on the right shows, IBM predicted back in its 2017 Quantcrunch Report that Data Engineers would be a “disruptor” role.

This meant that data engineering roles would entail: high costs to hire, a strong need for new training programs, and high risk to productivity.

 

Over the past year as organizations around the world have seen their big data volumes explode and accelerate post-digitization a stark reality hitting them:  Data engineering is critical to the success of their data science initiatives.  Worse, Data Engineers are in short supply.

This growing trend towards ramping up the hiring Data Engineers has become evident to Quanthub in recent months as we’ve seen more and more requests for data engineering skills testing.

Looking to hire Data Engineers? Try a free trial of our Data Engineer Skills Assessment.

We asked ourselves, “What exactly does 2020 and the future hold for the data engineering profession?”  Are Data Engineers more valuable than Data Scientists now?  What is being done to stem the shortage?

While dozens of data science degree programs have cropped up across the nation, our research could only find ONE undergraduate degree program titled “Big Data Engineering“.  Sure, there are 40,000 computer science degree graduates every year, but these are quickly scooped up by industries in desperate need of software engineers and web developers.

Some of these computer scientists will in fact develop into Data Engineers through any of 13 or so certification programs like AWS’s Big Data Engineer specialty certificate.  But is it enough to meet the needs of the companies like GE which downloads 30,000 hours of operating data EACH DAY for their gas turbines alone into a 40 terabyte database?

With these kinds of data volumes accruing, Statista predicts that by 2025 the amount of data in the world will to rise to 3X all data output to date at 175 zettabytes (1021 bytes). 

Data volumes will further be driven by the Internet of Things, 5G, machine learning, maturing and new digitization business models and the move to storing data in the cloud.

To sum up, all trends point to the certainty that data engineering as a profession is facing a pivotal year in 2020.   So we put together a simple infographic highlighting the most recent data collected on the field of data engineering.

Take a look and let us know if you’re looking to hire more Data Engineers, QuantHub can help you find them!  Just reach out to sales@quanthub.com or sign up for our free trial of data engineering tests!