How to get ready for the new dbt engine

Big things are coming to dbt. On May 28, we’re releasing a major upgrade: an all-new, next-generation dbt engine, built in Rust using the technology from our recent acquisition of SDF Labs. It’s designed to bring unprecedented speed and cost-savings to your data workflows, while serving as the foundation for the next era of data work.

Most teams using dbt will be able to try the new engine on day one. But to ensure the smoothest possible upgrade path when it’s released, there are a couple of simple steps you can take right now.

Let’s walk through what’s coming, and how to get prepared.

What’s coming

The next-generation dbt engine will bring with it major improvements to how you develop, deploy, and manage your data products. Here’s what you can expect:

⚡ Lightning-fast development

The new engine will bring deep SQL comprehension to dbt for the first time. dbt won’t just pass along your SQL to a data warehouse, it will actually be able to understand it. This unlocks powerful capabilities like live error detection as you write code, smart autocomplete suggestions, compiled code previews, and more. All that, coupled with model parse times 30x faster than before, amounts to a best-in-class developer experience for data practitioners.

💸 Built-in cost efficiency

By emulating your data platform locally, the new engine will be able to validate code and catch errors as it’s written... and before any warehouse compute is used. Plus, with capabilities like state-aware orchestration, you’ll be able to automatically run only what's changed in your DAG, helping you avoid unnecessary runs and reduce warehouse spend.

🔍 Better visibility and trust

SQL comprehension brings even more rich metadata to dbt. Because the engine understands your SQL ahead of execution, it will be able to generate rich model- and column-level lineage out of the box, allowing for faster debugging, better visibility, and impact analysis. In the near future, this will also unlock features like built-in PII tracing and policy enforcement.

How to prepare your projects today

During development of the new engine, we've taken the opportunity to reconsider and resolve some long-standing quirks of dbt. These quirks made it harder to guarantee consistent behavior in dbt and can cause problems for developers of third-party tools that integrate with dbt, maintainers of dbt itself, and especially end-users.

To take advantage of the new engine, you will likely need to give your project(s) a quick spring clean. We’re building tools and guides to make this upgrade simple, but there are two key things you can do right now to be ready when the new engine arrives.

1. Upgrade to the latest version of dbt

dbt Core v1.10 contains new deprecation warnings for soon-to-be-unsupported behaviors. In 1.10 they are informative only—your project will still run as normal—but you'll need to resolve them to successfully use the new engine once it's available.

That means you'll want to upgrade to the new engine from a project on the Latest release track (for dbt Cloud users) or dbt Core v1.10 (for self-hosted users). This will ensure the simplest, most predictable experience: you can pre-validate that your project doesn't rely on deprecated behaviors.

If you're one of the vast majority of customers already using release tracks, sit tight! You will automatically see relevant deprecation warnings (along with all other vetted new dbt capabilities and fixes) ahead of May 28th.

📘 Learn how to switch to the Latest release track →

2. Resolve deprecation warnings

Many deprecations (such as moving arbitrary configs into the meta dictionary) can be resolved automatically. Users with a local filesystem (using the Cloud CLI or dbt Core) will be able to use this auto-fix script developed by dbt Labs and then review any other deprecations requiring manual attention.

The dbt Cloud IDE will surface the same list of configuration or syntax issues that need updating. It will soon provide an interface to the same auto-fix script as well.

Resolving these warnings helps ensure your project will be compatible with the new engine when it’s released.

📘 See how to fix deprecation warnings →

What happens on May 28?

In dbt Cloud, most projects will see an in-product prompt asking if they’d like to enable the new engine at the environment level.

It’s as easy as flipping a toggle. Once enabled, the environments you select will run on the new engine, and you will be able to start experimenting immediately.

Outside of dbt Cloud, users will use the upcoming dbt VS Code extension or install the new Rust binary locally. In either case, if your whole team isn't ready to adopt the new engine on day one, it will also be possible for individuals to start experimenting with it without interfering with production workloads.

Join us live on May 28 for the dbt Launch Showcase, where our product team will go into more depth on the new engine, and share everything else that’s new in dbt. We’ll be showing:

• The powerful new VS Code extension for dbt development powered by the new engine
• A reimagined visual editing experience for analyst- and governance-friendly modeling
• Cost management features to help you understand and optimize resource use

… and much more. You’ll get a front-row seat to the future of data development, and we’ll show you exactly how to take advantage of it.

We’ll catch you at the dbt Launch Showcase.