Enterprise MDS deployment at scale: dbt and DevOps for efficient data process automation from Coalesce 2023

"Behind any good modern data stack is a solid DevOps design."

Team members from Datatonic discuss the benefits of using dbt and DevOps in data architecture. They demonstrate how this combination can be used together to enhance data reliability and quality, and they touch on the role of generative AI in data analytics.

DevOps and the modern data stack are key components of reliable data analytics

The Datatonic team highlights the relevance of DevOps in building a modern data stack. They point out that a strong DevOps design is fundamental to creating a modern data stack at scale.

Ash Sultan, Lead Data Architect, emphasizes, "Behind any good modern data stack is a solid DevOps design." He notes that although the concept is not new in software engineering, it has often been overlooked in the BI world.

Ash also explains the importance of DevOps in ensuring data reliability and quality, stating, "We like to get those things quicker because we want to make business decisions faster. But in order to do that, you need the data to be reliable. You need the data to be trustworthy."

dbt is a game-changer in the realm of DevOps and data analytics

Datatonic’s team provides insights into how dbt is transforming the field of DevOps and data analytics. They explain that dbt has brought features that ensure autonomy, security, and reliability, making it a game-changer in the field.

Ash explains, "dbt has been a real big one. That's why we're all here today. It’s really touched us all, in one way or the other, when we're working in the cloud world." He goes on to explain the power of dbt in giving analysts more power: “Suddenly you have the capability to work like developers and reap all the benefits."

Generative AI has the potential to transform DevOps and dbt

"Generative AI will lower the barrier to entry for information across companies, giving swift access to insights."

Datatonic’s team also discusses the potential of generative AI in transforming the fields of DevOps and dbt. They suggest that generative AI could provide instant insights into data using plain English language.

Stan Hill, Senior Cloud Architect, explains, "In the very near future, businesses will be able to get almost instant insights into their data using plain English language. Simply ask a question…‘Give me sales for customer X, for product Y, on a daily basis,’ and the model will give you the answers straight away…”

Stan also suggests that generative AI could be used with dbt and DevOps to make observing data within dbt projects and data pipelines more accessible. He explains, "Every time dbt Cloud runs, it generates and stores information about the project... with the API you can query this metadata to gain a better understanding of your DAG and your models...but it requires a level of technical expertise to get going...This is how we see generative AI removing the barrier for observing your data within your dbt projects and your data pipelines."

Stan and Ash’s key insights

• dbt and DevOps can significantly enhance data reliability and quality
• dbt is seen as a game changer in data analytics as it allows analysts to work like developers
• Automation of integration testing can prevent issues from reaching production
• Generative AI can lower the barrier to entry for information across companies, providing swift access to insights