How dbt State cuts warehouse compute and speeds up every run

Every day we see about 159,000 dbt jobs run. Most of them run hourly. And most run hourly even when the data hasn't changed, so thousands of models get rebuilt to produce the same results.

For a long time, that made sense. Cloud compute kept getting cheaper per unit, and it was easy to just use more of it.

Then everyone started using more data: more dashboards, and thus more pipelines powering machine learning models and recommendation engines. Once agents arrived, anyone could write a complex SQL query, or even create a dbt model.

What started as a reasonable way to run your pipelines got expensive. Those cheap cloud costs started to skyrocket.

dbt State is our answer. It takes what was a stateless engine, dbt, and makes it stateful: every time you run dbt run or dbt build, it only builds what's‌ changed and skips everything that hasn't.

The tagline we keep coming back to is simple: build what's changed, skip what hasn't. Here’s how it works and how one of our customers uses dbt State to push model reuse rates as high as 25%.

Why we built dbt State

For all of our data people, this was always about more than compute. Saving 30% on the data warehouse bill is great, and we wouldn't turn it down. But the thing you feel every day is the time, the focus, and the trust it takes to keep a project running.

Think about the friction:

• Every time you develop a new model, you have to ask which selectors it belongs in. Managing 20 different selectors across 10,000 models becomes a problem in itself.
• Every time you run in development, you wait for 100 or 200 models to build, 10 or 20 minutes, just to figure out whether you built the right thing. That costs focus.
• As costs climb and the business doesn't see more value, because the data isn't refreshing any more often, people start asking whether it's worth it.

dbt State started at our last dbt Summit, where we shipped state-aware orchestration (SAO) into preview. The feedback was clear. Customers were saving 30%+ on their data warehouse bill.

They came back with two questions. First, why is this only in production, when I'd love to use it in staging, continuous integration (CI), or development to make my runs faster? Second, why is it available only in the dbt platform on the dbt Fusion engine when I want to run it anywhere?

We listened. We took the best parts of SAO and made them work everywhere. The result is dbt State.

What dbt State does

The basic idea stays the same as SAO: build what's changed, skip anything that hasn't. That unlocks three things.

• Optimize costs. Teams see 30% less warehouse compute on average by only building what's changed upstream and adhering to the freshness service-level agreements (SLAs) you set, so the business isn't paying for fresh data more often than it uses it.
• Run freely. When you kick off a dbt run or dbt build, you can be confident you won't build 100 models when only 10 have changed. You'll build 10.
• Speed up development. You can increase the speed of every run without resorting to custom workarounds like sampling or using different data across development and production. You turn dbt State on, and it just works.

Run it anywhere you run dbt

dbt State is available natively in the dbt platform and locally in dbt Core v2.0. If you're on v2.0, you can run dbt login and set up dbt State right away.

If you're on v1, from v1.7 through v1.12, it's available as a plugin: run pip install dbt-state and you can be up and running across development and production in less than five minutes. There's no platform upgrade required.

Anywhere really does mean anywhere. Run dbt in an orchestrator like Dagster or Airflow. Add it to your own scheduled setup in a Python environment on EC2. If you can run dbt, you can run dbt State.

That last point is the part that matters most. SAO and dbt State were built on the same premise, skipping builds when the upstream data or code hadn't changed.

What we added with dbt State was reach: no longer requiring Fusion, no longer limited to production, and a way to reuse data you've already built instead of paying to rebuild it. When the same logic and data already exist somewhere else, say in a staging or production environment, dbt State clones them in instead of rebuilding, and you won't even notice the difference.

How dbt State works

It helps to separate two things: the data plane, where you run dbt, and the control plane, where dbt State runs.

In the data plane, every dbt run executes against your warehouse, where it has access to your data and builds your models. The control plane is where dbt State decides what actually needs building. It keeps a list of your tables and a hash of the last data state and code state. There's no actual data there, just a hash of what the state was.

So on every run, dbt State asks a short series of questions for each model. Has it actually changed? If it has, does a version already exist somewhere else that we can clone in? Only when the answer to both is no do we build it.

For a demonstration of dbt State in action, check out this on-demand webinar.

Paying only for what you reuse

All of this starts with a 30-day trial for every user. Run pip install, try it in development, run it in production, and tell us what you find.

When you do start paying, we only want to charge you for the parts that produce value. Our metric is the daily active target table (DATT): a model or test that dbt State reused. We count unique reuses per day, so we don't double-charge you.

If your fact_orders and dim_customers models rebuild six times a day but the data didn't change, you're not paying for six reuses each. You're paying for two, one per unique table, no matter how many times it was reused that day.

How Fanatics Betting and Gaming put dbt State to work

One of the teams that turned dbt State on early was Fanatics Betting and Gaming, and they saw double-digit compute savings on the very first project they tried it on.

Alvin Chai, a senior analytics engineer at Fanatics, runs a team responsible for building and maintaining gold-layer data models across the betting and gaming business, everything from sportsbook trading analytics and product insights to regulatory and financial reporting. Their stack was Snowflake for the warehouse, dbt Core, and Airflow to orchestrate.

About 18 months ago, they moved to the dbt platform to remove a bottleneck: refreshes and backfills used to require a specific data engineer to kick them off. Today they run about 11 projects, one per analytics domain, 8 of them on Fusion, with a couple thousand models in total.

Democratizing scheduling gave every analytics team the freedom to schedule their own refreshes. The downside, as Alvin says, was overscheduling with little visibility into it:

"Maybe you have an analytics team that might set up their models to run every 30 minutes. But the source that model is built on might run maybe only every hour or so."

dbt State surfaced their model reuse rates and effectively put guardrails on overscheduling, refreshing models only when they actually need it instead of rebuilding the same output over and over. That made the whole thing more hands-off, so the team no longer had to pester other analytics domains about their costs getting too high.

The rollout is a useful lesson in how to get value out of dbt State. Fanatics started with lower-risk work. Their pilot was a governance project supporting responsible gaming and anti-money laundering use cases, chosen because it had few downstream dependencies through dbt Mesh and would be easy to roll back. They then worked down the hierarchy by risk, migrating everything except their most critical regulatory and financial reporting.

Turning dbt State on with no configuration gave them a model reuse rate of about 0.2%, essentially nothing. The effectiveness came after they configured source freshness and declared model-level SLAs, what we now call lag tolerance.

Their approach was to set a baseline model-level SLA for every model based on its current refresh frequency, add a more aggressive project-level SLA on top, and write a small script to translate their existing schedule tags into those SLAs. A model tagged "daily," for example, became a 24-hour refresh window.

That took the pilot from 0.2% to roughly 15% model reuse. Historically, overscheduled projects later hit as high as 25%. Others landed nearer 5%. Across all their jobs, the average sits around 8%.

The compute savings are real, but Alvin's take is that the biggest long-term win is operational simplicity:

"The question kind of turns from, 'What job should this model belong to?' to 'How fresh does this model actually need to be?'"

Instead of juggling separate jobs by cadence and managing tags for each one, Fanatics can move toward DAG-based jobs: point at a final consumption model, refresh everything upstream of it, and let the model-level SLAs handle the different freshness requirements underneath.

The payoff is where the team spends its time. Rather than burning mental energy figuring out how to optimize schedules, they can focus on building new models and delivering more insight and business value.

Try it yourself

A decade of dbt has been about bringing engineering discipline to analytics. dbt State extends that discipline to the compute itself: stop paying to rebuild what hasn't changed, wherever you run dbt.

You don't have to take our word for it. Create a free dbt account or run pip install dbt-state on your existing project, and watch your development runs get faster and your warehouse bill get smaller. To see dbt State in action, watch the on-demand webinar.