Fivetran + dbt Labs: The future of dbt Core v2.0

Last year, we made two announcements that injected a bunch of uncertainty into the dbt community: the dbt Fusion engine and coming together with with Fivetran.

Change is scary, and the questions came fast. What was the future of dbt Core? One Reddit user went so far as to predict the "final nail in the coffin of OSS dbt," shorthand for open-source software.

What has‌ turned out to be true over the past year is that we have wanted to ship more code in the open, not less. And we anticipate that being true for many years to come.

The clearest proof is dbt Core v2.0: the same fast, capable foundation that powers the dbt platform, now under an Apache 2.0 license. Alongside it, we've shipped dbt State, a caching layer that cuts customers' dbt-driven compute by 30%+, and dbt Wizard, a coding agent purpose-built for dbt.

Why the data stack needs to be rebuilt for agents

Taylor Brown, Co-founder and COO, Fivetran + dbt Labs, and Tristan Handy, Co-founder and President, Fivetran + dbt Labs, have been working for a long time to bring the two companies together. The two joined a webinar recently to discuss the rationale, and share what comes next. For the full conversation, including the live Q&A, watch the webinar recording. And if you want to see the speed for yourself, create a free dbt account and invoke dbt Core v2.0.

Back in the 2010 to 2014 timeframe, everyone was trying to solve the problem of getting data out of siloed places and leveling up the overall analytics infrastructure to drive business value, largely for reporting. That's where Fivetran plus dbt originally helped coin and build the modern data stack.

Over 100,000 teams adopted dbt as a standard for transformation. Over 8,000 customers adopted Fivetran for data replication.

The age of AI has changed the game. As Taylor puts it, the outcome that data stacks drive is no longer just for humans and reporting. AI and agents are now large consumers of data, and they're driving business operations and revenue applications. That shift exposes three structural breaks in the stack at agent scale:

• Trust breaks. If you don't have complete data, consistent metrics, and lineage, you don't have trust. And trust is now infrastructural, because agents are taking action on this data, not just humans who understand its quirks.
• Scale stalls. With ungoverned data, fragmented context, and locked-in formats, agents will amplify how bad the underlying data is and operationalize it in negative ways.
• Cost explodes. Repeated retrievals and unoptimized modeling leave you with many more agents running on very expensive compute.

When we hit the cloud era, the cloud was so much more useful than on-prem systems that we had to rip everything up.

Fivetran and dbt are a result of what happened there. We're doing that type of work again in the agentic era because we're going to see at least an order-of-magnitude increase in data consumption.

Our answer is an architecture we call Open Data Infrastructure: a data stack built for agents and humans that's flexible by design, fresh and trusted by default, and efficient at scale. An open data infrastructure is:

• Built on open standards, separates storage and compute.
• Loads into modern data lakes in open formats like Iceberg and Delta.
• Relies on strong data movement and transformation in the pipeline layer.
• Provides rich metadata and context in the management and governance layer.

Companies are already putting this together.

• Zendesk, the global customer experience platform, used Fivetran plus dbt to scale out analytics agents and AI across the entire enterprise in a fraction of the time it would normally take.
• Shutterstock built a trusted, more real-time analytics and AI platform for emerging AI workflows.
• At Inova Health, a leading nonprofit healthcare provider, Jon McManus and his team compressed a four-year data modernization roadmap into six months, with Fivetran and dbt as key parts of the new architecture. For healthcare, that pace is unheard of.

dbt Core v2.0: One engine for all of dbt

If you've been on this journey with us for any length of time, you've seen us trying to do two things at once. The first is to support a widely used piece of open source software infrastructure. The second is to make money so we can keep doing the maintenance.

About 18 months ago, dbt Labs acquired SDF Labs. Why? Simple: their engine was, in many ways, technically superior to long-term dbt Core. It's written in Rust and includes a bunch of other capabilities. We combined dbt and the SDF engine and shipped it as the dbt Fusion engine.

But Fusion and dbt Core lived side by side, on different technical foundations (Rust and Python) and different licenses (Elastic and Apache 2.0). Over the past year, as we did the work of making Fusion ready for general availability (GA), we realized we wanted to bring these two engines together.

That work culminated in the launch of dbt Core v2.0. The alpha shipped on June 1, and it's still early, but the Apache-licensed dbt project is now based on the Rust implementation that Fusion created.

The first thing you'll notice is that it's just really fast. The first time you invoke dbt Core v2.0, the responsiveness in your command-line interface (CLI) is dramatically improved. And there's a change we’re especially excited about in how we deal with metadata. dbt has published artifacts like manifest.json and run_results.json for a long time, but dbt now produces Parquet files as well, which form the foundation of a context layer. You can query them locally with DuckDB and immediately find out facts about your dbt project.

Fusion isn't going anywhere. Fusion is a superset of dbt Core v2.0. Both are based on the same core technology, and pure OSS dbt Core is now a strict upgrade over v1. Installing the Fusion binary gets you everything in Core plus more, and the heart of that "more" is SQL comprehension.

Fusion natively understands the SQL you write. That unlocks developer experience benefits such as easier refactoring, autocorrect, and column-level lineage in your docs. You can see the full comparison of Fusion and dbt Core v2.0 in our docs.

The same binary you can use free, without ever speaking to us, now also supports dbt login, which gives you access to proprietary features, some free and some paid. And if you're a dbt platform customer, upgrading to v2.0 or Fusion is straightforward: an auto-migrator tool with AI to fix any remaining issues.

All of this lands just past a milestone that's honestly shocking. We recently celebrated the 10-year anniversary of the first commit to dbt Core. There are now over 100,000 teams using it in production every single week, and over a billion downloads. Many of the ideas dbt started with, like testing your data code and keeping data in version-controlled repos, were controversial a decade ago. Now they're defaults.

dbt State: A caching layer that saves you money

dbt State is one of the most important features we've ever shipped.

dbt State is a caching layer for dbt directed acyclic graphs (DAGs). If nothing has changed at the column level, the table level, or the code level, and you're inside the freshness window you've defined, dbt just skips that model. It doesn't need to build it.

The funny thing about building a caching layer is that you don't know how much more efficient everything could be until you implement it. It turns out dbt State saves everybody a lot.

That works out to a conservative 30%+ of a customer's dbt-driven compute bill. For us at dbt Labs, it was bigger, more like 64% of the compute bill. We've been able to cut about $400,000 annually from our budget for our underlying data platform. We added headcount as a result.

The savings aren't just in production. Taylor shared that Fivetran had a large revenue model that took something like 40 minutes to run; once dbt State was turned on, it ran in about 40 seconds. The analyst team said they literally can't go back. They did see more queries as a result, which drives prices up slightly, but the experience was profoundly better.

In development, dbt State makes your workflow really tight because you don't have to sit and wait to rebuild your entire development environment. In production, it saves you a tremendous amount of money.

One thing to be clear about: dbt State is a paid feature, but it doesn't require the dbt platform. If you're using dbt Core plus Airflow today, you can still run dbt State, and we'd love it if you did. The lowest supported version is dbt v1.7 with a plugin; from v1.12 on, and certainly in v2.0, dbt State is an incorporated feature.

dbt Wizard: A coding agent purpose-built for dbt

There's one more feature we have that will supercharge your development times: dbt Wizard.

dbt Wizard is a coding harness, similar to Codex or Claude Code, but tuned for dbt tasks. It turns out vertical-specific harnesses have a ton of benefits relative to generic harnesses built for all software engineering tasks.

The difference is that we control the harness from the ground up. So we can tune it to perform better and be more token-efficient for the kinds of problems analytics engineers are solving all day, every day.

dbt Wizard uses whatever model you run internally. You plug in your API key and pull from the same token budget, but it will frequently do a better and more efficient job than a generic agent. Try it out, and let us know in the dbt Community Slack what your experience is.

A foundation for the next decade

A decade in, the ideas dbt introduced are now just how data work gets done. The next decade is about making that same trusted foundation work for agents as well as humans. dbt Core v2.0, dbt State, and dbt Wizard are the first big steps.

For the full conversation, including the live Q&A, watch the webinar recording. And if you want to see the speed for yourself, create a free dbt account and invoke dbt Core v2.0. That wow moment is the whole point.

Frequently asked questions

Will dbt Core continue to be open source?

Yes.

Are there plans to make dbt packages private?

No. As Taylor put it, we think it's important that they remain open source and that anyone can access them. We'd still like you to run them on Fivetran plus dbt, but you don't have to.

Are dbt Core v2.0 and the dbt Fusion engine the same thing?

They're built on the same engine, and Fusion is a superset of Core. If you're already running dbt Core v2.0, there is no upgrade to Fusion; it's just a question of whether you want to flip on additional features. Many of them are free, so it's an exclusively better experience to run with them turned on. The Fusion availability page breaks down exactly what's included where.

What advantages remain in Fusion if Core v2.0 uses the same engine?

The core capability Fusion has that dbt Core doesn't ship is SQL parsing. Fusion natively understands the SQL you write, which powers improved editor capabilities and column-level lineage in your documentation, and gives us a foundation for future functionality like personally identifiable information (PII) classifiers that flow through the DAG.

Is dbt Core v2.0 still a Python package, even though it's implemented in Rust?

Yes. If dbt had started in the Rust community, it never would have been distributed via PyPI in the first place. But dbt grew out of the Python community, and there are over 100,000 deployments that start with pip install dbt.

So, while much of the code is, in fact, Rust, we're continuing to make it available via PyPI. You can also install the binary directly.

Is dbt moving to usage-based pricing?

We do anticipate monetizing dbt State on usage. But dbt State is a brand new feature, and we often make independent decisions about how to monetize brand new things. There's no specific plan to change how the products and services that existed before dbt State are priced.

Over time, we think of dbt more and more as infrastructure and want to monetize more in line with consumption. But that's a long-term arc, and we'll take every step carefully in consultation with customers.

How do the dbt State costs actually work out?

We haven't released final pricing yet. The entire point is that every dollar you spend on dbt State saves you more than that.

As Taylor put it, we'll give you a dollar for 50 cents. With dbt State, you spend a lot less money on your compute provider, we charge you a little more, and your total cost of ownership goes down.

When we deploy dbt State with a customer, we'll frequently run a proof of value in production for a couple of weeks. The before/after delta makes it a straightforward conversation.

Over a hundred customers have adopted dbt State already. Everyone's environment is different, so it's worth testing to see how much you'd save.

What's changing in the semantic layer?

We released a new spec for the semantic layer in v2.0 and Fusion. Previously, you had to write a top-level semantic model block as a separate thing and map it to the model, which was clunky. Now, natively within the model YAML, you can declare a semantic model, mark dimensions in the same syntax you use to describe a column, and define metrics right in the YAML.

We're also investing in making it more expressive, because folks coming from LookML sometimes hit patterns it can't support natively. That’ll take a good six to 12 months for the semantic layer, so keep an eye out for upcoming changes.

If AI agents become the primary data consumers, do SQL files become obsolete?

Some folks have said that in the future, the spec is the code, and everything else is an intermediate compilation artifact. There are versions of the world where that's true. But for the coming several years, we don't anticipate it.

Intelligence is really expensive. Executing software is dramatically more resource-efficient than starting from natural language text every time. So we don't think SQL files are going anywhere, and neither are Python or Rust files. The ways we author them will change, and it's becoming easier to build the surrounding infrastructure of tests and documentation, but they aren't losing relevance.

The previous data stack was about building analytics for humans. The emerging stack is about building context for agents. ontext engineering is a real, broad role, and analytics engineers will play a huge role in it. It's a tailwind for the career of every analytics engineer.

Will Fivetran and the dbt platform become a single app?

Today we haven't spent a ton of time on a single app experience, partially because the integration between the two products is already very tight. We'd rather invest the energy in innovation like dbt State, dbt v2.0, and dbt Wizard.

Looking five years out, it's hard to imagine two separate auth systems for transformations and integrations, especially in a world where more of this gets built by agents. But our number one priority is integrating the teams and keeping up the velocity of shipping things that matter to customers.

For the full conversation, including the live Q&A, watch the webinar recording. And if you want to see the speed for yourself, create a free dbt account and invoke dbt Core v2.0.