The next era of analytics: How AI is changing the game

AI is having a tremendous impact on every company. Make no mistake—the fundamentals of producing good, high-quality data remain unchanged. At the same time, AI is changing how everyone—engineers, analytics, decision-makers, and end users—works with, understands, and uses data.

In this article, I’ll talk a lot about the Analytics Development Lifecycle (ADLC), which we here at dbt Labs view as an important way of thinking about analytics. I’ll speak a little about how dbt has evolved into a data control plane that supports every phase of the ADLC. Finally, I’ll look at what AI means for data and analytics and how I think businesses can adapt—and how dbt Labs is adapting to meet the new challenges and opportunities that generative AI presents.

The importance of data quality in analytics

Our founder and CEO, Tristan, has talked about how dbt Labs views the Analytics Development Lifecycle (ADLC) as the best path to building a mature analytics practice within an organization of any size. Our mission here at dbt Labs has always been to empower data practitioners to provide great data and information that companies can utilize to alter their direction. Using the ADLC, teams and companies can do just that by talking to each other using the same language.

Let’s see why that’s important. Every year at dbt Labs, we run the State of Analytics Engineering survey to understand the pains, gains, and areas of investment for global data teams . Here are last year’s results.

When we founded dbt Labs in 2016, building data transformations and discovering already available data products were huge problems. We’re proud to see that that’s gone down. Constraints on computing resources also went down thanks to products such as Snowflake, Databricks, BigQuery, Athena, and Redshift.

So let’s look at the big problems now.

• One is ambiguous data ownership. People can get at the data, but they don't know who owns it—and presumably where it came from and how high-quality it is.
• Data literacy is a big problem—it’s challenging to be an effective analytics engineer or analyst when your stakeholders don’t understand what the data means or how it can affect business decisions.
• Sadly, the largest problem is data quality. When you think about that for a moment, that’s a big problem. If we can’t trust the data, then how useful can our work be? How confident should we and our stakeholders be making decisions on that data?

I’ve spent most of my career building and operating OLTP databases, and data quality in those systems is largely around accuracy, i.e. “Is the bank balance right?”. That’s one kind of data quality. But when you're running analytics, there are lots of different dimensions of quality. To list a few: completeness, consistency, timeliness, traceability, lineage, and uniqueness.

You can't, or at least shouldn’t, present a dashboard to your CEO at 9am on Monday that doesn’t have clean data on every one of these dimensions—decisions will be made based on that data. Business is moving faster than ever before, is more complex than ever before, and those poor decisions have real consequences. This means that data quality has risen to a whole new level of importance. And that’s where the ADLC and dbt both come into play.

The initial problem that dbt solved

I used to do analytics for a very small company. We had about 350,000 users, and we had eight MySQL databases on which I ran all the analytics.

I had one file in Vim that was 38,000 lines long. Every day, I cut and pasted parts of that SQL script into the various MySQL databases to do our analytics.

You can guess how well that worked.

To fix this, dbt Labs brought the concept of software development to analytics. Using dbt, you can run all of your analytics code through a process modeled after the Software Development Lifecycle (SDLC).

dbt enables creating data transformations in the form of vendor-agnostic dbt models. You can then test your transformations instead of shipping them to production prematurely and inadvertently trashing someone’s dashboard. Once tested, you can deploy your changes using Continuous Integration (CI) and observe them.

You can also use dbt to find and discover data and unearth facts. Of course, once you discover facts, you're never actually done because your questions lead to more questions. And so you analyze those facts, and then you plan and develop more analytics code to provide new answers. And that's why the ADLC is a figure-eight.

The evolution of dbt and the ADLC

dbt has grown a lot over the years. At first, we ran SQL and Python data transformations, and that was it.

But that was big! It changed analytics from “Mark owns the SQL script” to “everyone in the company who deals with analytics can see, share, test, and check in changes to the SQL script.” You could also roll back to the last known good state if you had an “oh crap, I broke the dashboard the CEO uses to keep the company running” moment.

Next, we added orchestration, so that you could automatically run jobs and see if things were taking too long to run. Then, to make a very long story currently involving around 580 people short, we added all these other capabilities.

The Semantic Layer, for example, means you can have one definition of what a term (such as “fiscal year”) represents, rather than having every team define its own, divergent version. We have a visual editor for writing SQL and YAML code.

We have a number of new features to help manage data and speed deployment of new data-driven projects:

• Most recently, and most exciting, we have dbt Copilot, which can fetch the data from your data warehouse and build a dbt model around it.
• We added dbt Mesh, so that teams in large companies could own their own data and make data contracts with other parts of the company.
• We’re also focused on optimizing costs, providing support for the Iceberg open table format, and launching a new Advanced CI feature where you can learn more about what’s going on with your data.

dbt grows into a data control plane

The product that dbt Labs owns today has expanded beyond transformation. No, we haven’t expanded into ingestion. There are lots of tools that do that well. And we've not expanded into AI and BI analytics; instead, we partner with AI and BI analytics tools that do what they do well.

What we have done via dbt Cloud is expand into a complete orchestration and observability system. This way, when your data warehouses are burning all their compute overnight, you know what they're doing and whether it worked.

We call this the data control plane: an abstraction layer that sits across your data stack, unifying capabilities for orchestration, observability, cataloging, semantics, and more. Features such as the Semantic Layer and our data catalog round out our support of the ADLC and create a single place to observe and orchestrate data across your company.

We did something else pretty wild: we acquired SDF. SDF gives us complete SQL understanding, unlike dbt’s previous simple text parsing. With SDF, we'll catch all syntax errors before sending them to the warehouse and help you fix them in the editor. We'll also quickly query your warehouse's data dictionary to identify non-existent tables or columns.

This fundamentally changes how developers work with dbt, providing:

• Much faster developer experience
• Reduced warehouse costs
• Better DAG lineage tracing, which is especially important for GDPR compliance

This integration work is still ongoing, but will provide faster development cycles and additional cost savings for data teams.

How businesses can adapt to the AI era

That raises the question of what’s next for analytics. Particularly, it raises the question of how AI impacts analytics and the work of data engineers.

If we think about previous disruptive shifts—e.g., the shift from on-premise to cloud, or the shift from databases to data warehouses—the trends were much slower. They took five, 10, or 15 years and were less disruptive.

By contrast, AI is an immediately disruptive shift. Take agents, for example. Did anyone actually think that computers were going to formulate English questions to each other to communicate? That was silly. It wasn't like one computer was going to say, “Hey Joe, can you tell me about revenue last week?”

And yet, here we are. With agents and new innovations like Model Context Protocol (MCP), we’re on the verge of seeing an absolute explosion—data talking to data, computers talking to computers, and analysts being able to ask different questions.

So AI is different. At the same time, part of our job as a technology company is not to buy into hype. When it comes to AI, that means being realistic and setting expectations.

When I was at Grab, I had 1,700 people reporting to me in engineering. 150 of them were AI engineers. And they were developing completely traditional models—not using GenAI Large Language Models (LLMs).

So when it comes to adapting your business to AI, I have three recommendations:

• Lean into every traditional model. They’re incredibly valuable. You don't need to use LLMs. You don't need to have a huge bill with your favorite LLM company. Build models, fine-tune models, work on models.
• Start simple. Most companies are having trouble getting started with AI. Start simple - summarize support tickets, use Notion summaries. Start getting familiar.To be clear, I don’t think there’s an AI chatbot I’ve used yet on a company website that didn’t require human intervention. We’re still in the early days here. 
• Data quality is key. AI doesn’t know its data better than any expert or random social media user does. The quality of data—particularly, the quality of its source and whether we can trust that source—will be key.

I think we’ll see AI systems start to do two things. One, they’ll be able to trace data. Two, and more importantly, they’ll start to grade data, tell us how well-governed it is, and where it came from. And then that happens, it’s going to be game-changing.

AI and the developer experience

Another angle to consider isn’t just how AI is disrupting business, but how it’s disrupting data engineering. Tristan recently wrote a piece on this in which he talked about AI’s disruptive role in how we develop and manage analytics code.

I manage a group of software engineers. For them, AI is changing how they code, document, and even debug. It’s also changing how my product team and designers are doing research. They use a deep research site or ChatPRD and get a great PRD done in minutes, not days.

In data, too, tools like dbt Copilot are changing the way data engineers create models as well as how business users interact with data. I think you’ll see a lot of fast development here, particularly with agentic AI.

There’s all this hype in the world about everyone losing their jobs to AI. My son, who’s in the industry, sees it differently. He said that AI is going to let his team do the jobs the rest of the business always wanted to do at the speed, quality, and intensity that they always wanted.

Back in the 1960s, computers could do X amount of work and people wanted us to do Y amount. It was the same in the 1980s. It’s the same today. We’re still, even with AI, not even close to computers being able to do the things that people want them to do. As long as that band gap exists of human ingenuity, of human cleverness, of human directed creativity, I don't think anyone has to worry about their jobs.

How dbt is changing to serve the next era of analytics and AI

When we launched dbt, it was for a very specific persona: the data nerd. The data nerd loves to sit and write some SQL statements, chain them all together, and feel proud.

And that’s great. Nothing wrong with that. dbt and dbt Cloud remain powerful tools for data engineers and analytics engineers.

But over time, more and more analysts started using dbt to drive business insights. So one of the things that you're going to see us doing more is really catering to a set of workflows that help analysts have a lower-friction workflow.

For example, today with dbt Copilot, you can ask it to build you a model based on your data and a natural language description of the output you need. Alternatively, you can take an existing model developed by someone else that’s extremely complex and say, “Tell me what this does.”

This is a critical shift, as there are anywhere from five to 15 times more analysts at a company than there are data engineers and analytics engineers. Giving them more power to transform and manage data means greater data democratization and faster data project velocity.

There’s this third persona we’re targeting, and that’s the data executive. They need their data to be accurate and to be assured that data fed to AI systems is governed properly. And they also care about tracking costs.

In the past, the people pushing for dbt and the ADLC at companies were data and analytics engineers. Today, we find analysts and data executives joining the call. As dbt continues to evolve, you’ll see more of this evolution: using the power of AI and other technologies to get high-quality data quickly into the hands of everyone who participates in the data lifecycle.