Empowering the enterprise for the next era of AI and BI

dbt emerged on the scene nine years ago. It has since become the standard in data transformation, introducing best practices such as modularity, testing, and documentation to analytics workflows.

However, a lot’s changed in that decade. AI workflows are now a reality and are here to stay. Technologies like Apache Iceberg see us slowly inching closer to the commoditization of compute.

To stay ahead of the demands of modern data teams, dbt needs an update. It needs something that keeps developers in the flow, speeds up their development loops, and integrates natively with AI systems.

That’s why dbt Labs acquired SDF - and why we embarked on a five-month journey to rewrite dbt entirely from the ground up to incorporate it. The result is the dbt Fusion engine, a new state-of-the-art data transformation engine and SQL compiler that powers a smarter, faster developer experience and optimizes data engineering costs.

Let’s take a look at how the new dbt Fusion engine improves the developer experience, enables AI-native development, and provides enterprises more confidence in their data through real-time validation and impact analysis.

dbt’s innovations and limitations

For the past nine years, dbt Labs has released features that make it possible to institute a mature Analytics Development Lifecycle (ADLC) around data projects.

However, there’s a disconnect between dbt and the systems against which it operates that slows developers down. At its heart, dbt takes Jinja templates and resolves them down to raw SQL strings. It defers understanding of these strings to data storage systems like Snowflake or Databricks that are hosted in the cloud.

This means a developer doesn’t know if their dbt code will compile as expected or not as they’re writing it. They have to run it against their target system, which responds with a status on the code. That introduces network latency as well as compute cost. All of this leads to slow development loops, as data developers are dependent on warehouse validation for even basic syntax checking.

How Fusion changes the game

Fusion is a ground-up rewrite of dbt that incorporates the technology from SDF, a high-performance SQL compiler that can parse every major SQL dialect (Snowflake, Redshift, BigQuery, etc.) and validate it locally. With Fusion, the entirety of your data warehouse is fully defined and statically analyzed as code.

Fusion provides a host of benefits over dbt Core:

Blazing fast performance

SDF originally built its engine around dbt’s Jinja syntax. However, when we sought to integrate the two platforms, we ran into a major issue: dbt was written in Python, while SDF was originally written in Rust.

To address this, the (now larger) dbt Labs engineering team undertook a five-month migration of the dbt engine from Python to Rust. This spells huge performance gains: dbt Fusion brings 30x faster project parsing and compilation to the table.

The move to Rust also simplified dbt Fusion usage. Since Rust compiles down to binary executables, there’s no need to install Python, run pip, debug Python library dependency conflicts, etc. Fusion just works out of the box.

Deep SQL comprehension capabilities

Fusion takes dbt’s SQL models, resolves them to logical plans, and validates each query in the model according to its representation of the data warehouse’s SQL dialect. That means it can validate the correctness of SQL in the code editor, before it even hits the data warehouse.

It also means it can statically analyze that plan to produce column-level lineage and propagate metadata. That gives data developers deep insights into how a given code change affects their existing environment.

State awareness and optimization

Fusion understands the freshness of data every time it runs. That means it can skip reruns of a model if the upstream data hasn’t changed. That simplifies orchestration, as it means you can run your pipelines at whatever interval and rest assured that they won’t do any unnecessary work.

This optimization saves both time and money. Currently, we’re seeing early estimates of 10% cost savings. We expect to get this up to 20 to 30% with additional cost optimization capabilities we plan on launching soon.

dbt Fusion engine + IDE: a supercharged developer experience

Fusion’s power really shines through when paired with a first-class Integrated Developer Environment (IDE). You can leverage these advanced features using the dbt Studio IDE, Canvas, or our new VS Code extension. Let’s dive into how these features can supercharge data development teams’ work using the VS Code extension as an example.

(Note: You can see an interactive demo of these features in our corresponding YouTube video.)

Validating SQL and viewing data lineage locally

Assume you have a simple model that tracks baseball players and related data, such as teams and salaries, that’s stored in Microsoft SQL Server.

Let’s say you have a simple dbt model, stg_players.sql, which is part of your model’s staging layer. You can run your model right in VS Code using the new dbt extension and see the results on the Query Results pane. More powerfully, however, you can dig into the data lineage for a specific column - let’s look at FULL_NAME - and see where the data comes from and where it’s going.

Note that this is all happening locally. You could be offline, and this would run in the exact same fashion.

You can really see the power of the Fusion compiler by looking at the developer experience. Say you have a block of Common Table Expressions (CTEs) in your pitcher_salary_analysis.sql model for various intermediate data, such as salary.

Let’s say you wanted to add a median annual salary for the pitchers on teams. So you add a yearly_median_salary CTE…but forget to include the trailing comma at the end, making this invalid SQL.

As soon as you save the file, you get a syntax error. dbt Fusion detected this error locally without pinging your data warehouse. Instead of waiting to detect this after you run it against a development copy of our warehouse or kick off a lengthy CI/CD deployment process, you get instantaneous feedback as you type and work.

Finally, dbt Fusion in VS Code supports previewing a single CTE. In the past, if you wanted to do this, it required commenting out the rest of the file. Now, you can accomplish this by simply clicking the Preview CTE command in the editor and viewing the output.

Intellisense and Copilot feedback loops

The VS Code editor also brings powerful IntelliSense capabilities. If you start adding median() function to your code, for example, you can see the editor will pull up a list of functions available to us in Microsoft SQL Server.

Let’s say you try to complete this CTE by defining a field called median_salary as:

median(salary) as median_salary

Attempting to save this, however, gives you another and even more fundamental error detected by the compiler.

Fusion’s compiler detects that median() is an aggregation function in your SQL dialect.

Real-time impact analysis and refactoring

Fusion in VS Code gives you instant feedback as you code, flagging downstream errors as soon as you make a change, before it ever hits CI/CD. No more waiting to find out something broke.

Let’s say you alias the power_id column and break dependencies. Fusion detects the impact immediately: errors surface directly in your editor, so you can fix them before committing anything. Behind the scenes, dbt is compiling your project and analyzing downstream references as you type.

Fusion also expands macros inline, so when you use ref(), you see exactly what it resolves to. Change a macro definition? You’ll see any model that relies on it light up with errors.

And when you rename a model, Fusion prompts you to refactor. One click updates all references across your project, with the option to preview or revert.

This is the kind of developer experience that makes high-trust data work possible. When refactoring is safe and fast, you're empowered to improve your codebase instead of working around it.

Conclusion and what’s next

With Fusion, data developers can ship data-driven projects faster than ever. Its blazing-fast SQL compiler, IntelliSense, and support for embedded AI copilots mean developers can create, test, and publish models in a fraction of the time they could before. Plus, with local validation and state-aware orchestration, you can slash around 10% off of your data warehouse bill.

We’re continuing to evolve Fusion to add new and useful features for faster analytics engineering. Later this year, we’ll release new data governance features that leverage Fusion’s advanced data lineage capabilities to create advanced tagging and classification workflows, such as removing the classifier tag from aggregated data.

Fusion is live and available now to supercharge your team’s data development. To try the next generation of data engineering for yourself, sign up for a free dbt account now or book a demo with us today.