{"id":9639,"date":"2020-09-10T09:31:14","date_gmt":"2020-09-10T09:31:14","guid":{"rendered":"https:\/\/www.experfy.com\/blog\/?p=9639"},"modified":"2023-11-08T10:31:12","modified_gmt":"2023-11-08T10:31:12","slug":"data-engineering-in-2020","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/bigdata-cloud\/data-engineering-in-2020\/","title":{"rendered":"Data engineering in 2020"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

It is incredible how fast data processing tools and technologies are evolving. And with it, the nature of the data engineering discipline is changing as well. Tools I am using today are very different from what I used ten or even five years ago, however, many lessons learned are still relevant today.<\/mark><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

I have started to work in the data space long before\u00a0data engineering became a thing<\/a>\u00a0and\u00a0data scientist became the sexiest job of the 21st century<\/a>. I \u2018officially\u2019 became a big data engineer six years ago, and I know firsthand the challenges developers with a background in \u201ctraditional\u201d data development have going through this journey. Of course, this transition is not easy for software engineers either, it is just different.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Even though technologies keep changing \u2014 and that\u2019s the reality for anyone working in the tech industry \u2014 some of the skills I had to learn are still relevant, but often overlooked by data developers who are just starting to make the transition to data engineering. These usually are the skills that software developers often take for granted.<\/p>\n\n\n\n

In this post, I will talk about the evolution of data engineering and what skills \u201ctraditional\u201d data developers might need to learn today (Hint: it is not Hadoop).<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

The birth of the data engineer.<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Data teams before the Big Data craze were composed of business intelligence and ETL developers. Typical BI \/ ETL developer activities involved moving data sets from location A to location B (ETL) and building the web-hosted dashboards with that data (BI). Specialised technologies existed for each of those activities, with the knowledge concentrated within the IT department. However, apart from that, BI and ETL development had very little to do with software engineering, the discipline which was maturing heavily at the beginning of the century.<\/p>\n\n\n\n

As the data volumes grew and interest in data analytics increased, in the past ten years, new technologies were invented. Some of them died, and others became widely adopted, that in turn changed demands in skills and teams\u2019 structures. As modern BI tools allowed analysts and business people to create dashboards with minimal support from IT teams, data engineering became a new discipline, applying software engineering principles to ETL development using a new set of tools.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

The challenges.<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Creating a data pipeline may sound easy, but at big data scale, this meant bringing together a dozen different technologies (or more!). A data engineer had to understand a myriad of technologies in-depth, pick the right tool for the job and write code in Scala, Java or Python to create resilient and scalable solutions. A data engineer had to know their data to be able to create jobs which benefit from the power of distributed processing. A data engineer had to understand the infrastructure to be able to identify reasons for failed jobs.<\/p>\n\n\n\n

Conceptually, many of those data pipelines were typical ETL jobs \u2014 collecting data sets from a number of data sources, putting them in a centralised data store ready for analytics and transforming them for business intelligence or machine learning. However, \u201ctraditional\u201d ETL developers didn\u2019t have the necessary skills to perform these tasks in the Big Data world.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

Is it still the case today?<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\n

I have reviewed many articles describing what skills data engineers should have. Most of them advise learning technologies like Hadoop, Spark, Kafka, Hive, HBase, Cassandra, MongoDB, Oozie, Flink, Zookeeper, and the list goes on.<\/p>\n\n\n\n

While I agree that it won\u2019t hurt to know these technologies, I find that in many cases today, in 2020, it is enough to \u201cknow about them\u201d \u2014 what particular use cases they are designed to solve, where they should or shouldn\u2019t be used and what are the alternatives. Rapidly evolving cloud technology has given rise to a huge range of cloud-native applications and services in recent years. In the same way as modern BI tools made data analysis more accessible to the wider business several years ago, modern cloud-native data stack simplifies data ingestion and transformation tasks.<\/p>\n\n\n\n

\n

I do not think that technologies like Apache Spark will become any less popular in the next few years as they are great for complex data transformations.<\/em><\/p>\n

Still, the high rate of adoption of cloud data warehouses such as Snowflake and Google BigQuery indicates that there are certain advantages they provide. One of them is that Spark requires highly specialised skills, whereas ETL solutions on top of cloud data platforms are heavily reliant on SQL skills even for big data \u2014 such roles are much easier to fill.<\/em><\/p>\n<\/blockquote>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

What skills do data developers need to have?<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

BI \/ ETL developers usually possess a strong understanding of database fundamentals, data modelling and SQL. These skills are still valuable today and mostly transferable to a modern data stack \u2014 which is leaner and easier to learn than the Hadoop ecosystem.<\/p>\n\n\n\n

Below are three areas I often observe \u201ctraditional\u201d data developers having gaps in their knowledge because, for a long time, they didn\u2019t have tools and approaches software engineers did. Understanding and fixing those gaps will not take a lot of time, but might make a transition to a new set of tools much smoother.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

1. Version control (Git) and understanding of CI\/CD pipeline<\/strong><\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

SQL code is a code, and as such, software engineering principles should be applied.<\/p>\n\n\n\n