Panel discussion: Fixing the data eng lifecycle from Coalesce 2023

"It's so exciting to see that our field is taken that much more seriously, and people are starting to see us go from cost centers to revenue drivers."

- Sung Won Chung, Solutions Engineer at Datafold

Louise de Leyritz from The Data Couch podcast hosts a conversation on the life cycle of data engineering, the evolution of roles within data teams, and the emerging technologies transforming the field. The participants discuss various challenges in data engineering and the potential solutions to these issues.

A focus on human skills and collaboration can lead to better practices in data engineering

The participants agree on the importance of human interaction and collaboration in the data engineering field. They suggest that the democratization of data engineering has made it easier for people to enter the field, but it has also increased the responsibility of data engineers. This means that they must not only have technical skills but also strong communication and cooperation skills.

"With democratization, you need a lot more responsibility to go with that. We're still working on that part," says Matt Housley, CTO of Halfpipe Systems. He emphasizes the importance of training in best practices and moving toward collaboration to improve the job of data engineering.

The group also discusses the need for more training and the establishment of best practices in the industry. They highlight the importance of communication between data engineers and other team members. "If your colleagues have more training, then your job as a data engineer is going to be better. And if you have more training, your life is also going to be way better," Matt adds.

Rapid advancements in technology are changing the data engineering landscape

The participants discuss the impact of new technologies on the data engineering life cycle. They note that while advancements have made data engineering tasks easier, these developments also require data engineers to continually update their skills and knowledge.

“...companies adopt new technologies at different pace[s], and they also, you know, have different interpretations of what a data engineer should be doing,” says Mehdi Ouazza, Developer Advocate at MotherDuck.

Mehdi also discusses the impact of emerging technologies like Webgpu and Awesome, which allow for more efficient and powerful data processing. He emphasizes the need for data engineers to stay updated with these emerging technologies.

The roles of data engineers are evolving, requiring new skills and approaches

The group discusses the evolving roles of data engineers and how this is changing the data engineering life cycle. They highlight the shift toward more software engineering principles in data engineering practices and how this is influencing the roles and responsibilities of data engineers.

"There is an issue with the role definition in data. So it's a big mess at the moment," says Mehdi. He suggests that as tasks traditionally performed by data engineers become more automated or simplified, they need to focus on higher-level tasks, such as understanding the business side of things and contributing to the overall strategy.

Sung Won Chung, Solutions Engineer at Datafold, also notes that the changing roles of data engineers are leading to higher salaries in the field. "I've seen data engineering job postings, specifically for companies that sell proprietary data…go all the way to FAANG levels of 200-300k," he says. He adds that this shift makes data engineering more attractive as a career choice.

The participants' key insights

• There's a need for more training on best practices in the data engineering field
• The technical barrier to entry in data engineering is getting lower, making the field more accessible
• As data engineering evolves, there's a shift towards incorporating more software engineering principles
• The democratization of data engineering has led to a need for more responsibility in managing data
• The explosion of the data stack and the complexity of integrating various tools are challenges in the data engineering life cycle