The productivity gains hiding in your data infrastructure

Demand for data is exploding thanks to AI. Some experts estimate that global spending on data centers capable of handling advanced AI workloads could hit $7T by 2030.

Data teams aren't necessarily getting more resources to deal with it, though. When we talked to companies, we found that only 36% of data teams reported increasing budgets. That means data teams have to shoulder more work with existing capacity.

AI itself helps, of course. Tools like data copilots reduce the burden of generating code that produces clean, high-quality data for AI agents.

But AI can only do so much. The hard problems still need humans to solve them. And currently, those humans are mired in maintenance, working assiduously to keep the data house of cards from falling down around them.

The good news is that data teams can free up significant capacity by taking a modular, reusable, automated approach to managing AI and analytics data workloads. This isn't just speculation. A recent IDC report quantifies exactly how much using a platform like dbt can save at every step of the data lifecycle.

The maintenance nightmare

No data team handles data from a single source. Everyone is constantly wrangling data from a variety of data storage platforms and formats.

The advent of AI has made this even more of a challenge. Data teams aren't just dealing with structured relational data and semi-structured data sources any longer. They're also mining PDFs, emails, and social media posts for insights.

All this means that most teams end up taking a scattershot approach to managing data pipelines. Most are written on the fly as quick and dirty one-offs meant to get the job done.

The result? A maintenance nightmare.

• Pipelines are brittle and prone to breaking. Respondents in our annual State of Analytics Engineering Report reported that they spend a significant amount of their time maintaining data sets, platforms, and infrastructure.
• Most work isn't reusable across data pipelines, forcing teams to rebuild what they need from scratch each time.
• Testing, deployment, and review are slow, manual processes, if they exist at all.
• New data contributors face a long ramp-up time learning how to navigate heterogeneous data systems. Many systems end up being too complex for non-technical contributors to use efficiently.

Each of these factors eats up precious time that data team members could instead be spending on more strategic work, such as streamlining data intake, optimizing storage, improving overall scalability, and automating key processes for faster delivery and more consistent quality.

dbt: Unlocking capacity without hiring

For years, dbt has served as the data control plane for companies worldwide. By taking a single, vendor-agnostic approach to modeling data, testing changes, and orchestrating data pipelines, dbt reduces complexity, boosts reusability, and improves quality across all analytics and AI data workloads.

The results, as summarized by the IDC Business Value report, are real and measurable. IDC interviewed eight enterprise companies that use dbt, with an average of 20,000 employees and $16.75B in annual revenue.

In headcount numbers, across various data functions, companies reported recouping the equivalent of 58.7 FTEs' worth of capacity. The businesses reported doing more with less headcount across all data functions:

• Analytics and data teams: +25 FTEs
• Developers: +11.2
• Business analysts: +17.6
• Data governance: +2.5
• Platform management: +2.4

The analytics/data team savings alone are the equivalent of $1.75M of additional business value delivered every year just by using dbt.

In each case, functionality provided out of the box by dbt enabled teams to achieve these gains:

A speed boost across the data stack

These savings come from reducing the cycles required to perform basic data tasks. At every step of the data lifecycle, dbt enables teams to deliver more in less time:

• Report delivery drops from 16.3 days to 8.4 days, a 49% acceleration
• Testing is 44% faster for new apps, and 46% faster for pipeline updates
• Development cycles are 41% and 37% faster for new features and updates, respectively
• Teams reported delivering new solutions to market 34% faster, and scaling 33% faster

"dbt has significantly improved developer collaboration and increased our development velocity," one customer told IDC. "It introduced a structured deployment process through its integration with Git."

Faster onboarding and reuse

Teams also reported significantly faster onboarding times. Onboarding dropped from an average of 3.3 weeks to 1.7 weeks, a 47% acceleration.

The driver? The availability of existing models. With data modeled consistently and available for self-service discovery via dbt Catalog, new hires have access on average to 101,970 reusable models from day one.

This means the productivity gains offered by dbt aren't a one-time bump. Each new model is an investment that compounds into the future as new hires plug existing data transformation code into their own pipelines.

"dbt platform is very easy to use for new employees," reported one customer. "It's all templates and SQL, so people are comfortable with it."

Reducing the data quality tax

Testing, automated deployment, and clear documentation have a measurable impact on data quality. Companies reported 33% fewer data quality issues due to dbt. They also reported 35% fewer late-data and 13% fewer incomplete-data instances.

This is important because data teams historically spend much of their time responding to and fixing data quality issues retroactively, after they make it to production. Studies in software engineering have shown that fixing bugs in production is significantly more time-consuming and expensive than ensuring those defects never ship in the first place. A bug that costs $100 to fix in the design stage of a project may become a $10,000 problem if it makes it to production.

"We use dbt to run tests such as checking for duplicates, missing values, and incorrect data," another customer said. "dbt helps identify issues and traces them back to their source, allowing us to fix problems at the system level and deliver more reliable data."

The capacity was never missing

None of these gains come from a single feature. They come from applying a single discipline, the Analytics Development Lifecycle, to every stage of data work:

• Build: Reusable models replace one-off scripts. Teams solve each problem once, not repeatedly.
• Test: Automated checks catch errors before they ship, not after they've reached a dashboard or an AI model and led to a costly business decision.
• Deploy: Continuous Integration and Continuous Delivery (CI/CD) move changes automatically through a rigorous change management process, not via manual pushes and after-the-fact firefighting.
• Operate and discover: Data lineage makes it easier for engineers to identify the root cause of data issues. Data cataloging and health reports enable self-service so that everyone can leverage what already exists instead of re-inventing the wheel.

This capacity wasn't missing. It was buried in maintenance and the errors caused by manual work. dbt frees up this capacity so that teams can ship more and ship faster using the resources they already have.

To learn more about how dbt saves companies time and money, download the full IDC report. To get started enhancing your data productivity, talk to us today about how dbt fits into your existing data stack.