Reducing ETL licensing costs with the dbt Fusion engine

As the data landscape changes, Extract, Transform, and Load (ETL) tools are struggling to scale. A key reason? Cost.

Managing ETL data pipelines often requires licensing expensive, turnkey software tools. The licensing costs for these tools alone can be steep, ranging between $20,000 and $100,000 for mid- to large-scale enterprises.

As data systems scale, this pricing can go from costly to prohibitive. For fixed-seat licensing systems, costs increase as companies bring on more employees to handle the growing volumes of data required for analytics and AI solutions. For usage-based systems, costs can spike dramatically as companies grow from terabyte to petabyte workloads.

The fact is that traditional ETL systems are struggling to keep pace in the age of AI. Let’s dive into how traditional ETL licensing works, why costs are rising, and how dbt and the new dbt Fusion engine can help you nip rising ETL costs in the bud.

Issues with ETL licensing costs

ETL is the legacy approach to data transformation, where data is transformed before it’s loaded into a data warehouse. It’s being supplanted by the Extract, Load, and Transform (ELT) pattern, where raw data is loaded into a data warehouse first and transformed multiple times, each time tailored to a specific use case.

These days, most traditional ETL data pipeline tools support either ETL or ELT approaches. In what follows, we’ll use “ETL” as a shorthand for both.

ETL tools use one of two pricing models:

• Fixed price: Vendors charge companies for user seats and/or for a fixed amount of storage and compute
• Pay-as-you-go: The cloud or “utility” model, in which companies only pay for the amount of computation and storage they actually use

Both of these models have their downsides for managing ETL costs at scale.

Fixed price is good for controlling costs and budget planning. However, “fixed price” can also mean fixed performance and fixed scale. Companies that hit their fixed-price limits may see their data processing grind to a halt, leading to service outages or processing bottlenecks.

Pay-as-you-go has become more popular in the age of the cloud, where people have grown accustomed to treating compute as a service akin to electricity. This is much more flexible, as you won’t run into usage limits and are assured of continuous performance.

The problem with pay-as-you-go pricing is that, as usage grows, costs can spiral out of control. Multiple dimensions of scaling—e.g., user base growth, service complexity, data volumes, etc.—can lead to fees growing at multiples of prior usage.

Most ETL tools don’t provide a built-in way to split the difference between these two models and manage costs effectively. And that’s a problem, because figuring out how to control costs is crucial to maintaining scalable and stable data systems.

How dbt helps with ETL costs

dbt is a data control plane that provides a complete framework for developing, testing, deploying, and monitoring ETL/ELT workloads. With dbt, data teams can treat develop reusable code they can use to manage their data transformation logic in a vendor-agnostic manner.

dbt provides numerous features that make developing and shipping ETL pipelines more efficient out of the box. It supports developing data transformation code for analytics and AI workloads in accordance with the analytics development lifecyle (ADLC), which treats analytics code as software. This means that all code can be thoroughly reviewed and tested beforehand to ensure it’s written in the most performant, cost-conscious manner possible.

The pain of JIT and round-tripping

One of dbt’s main advantages is that it provides a single and consistent approach for modeling data transformations across the industry’s leading data warehouses. It provides data producers and consumers with a common platform and syntax for working with data.

When it comes time to render these models, dbt Core, has always used a Just In Time (JIT) model. It connects to your data warehouse to run and verify your SQL or Python code. This is true, not only for production code, but for all code under active development.

This means that developing and testing code are also contributing to your increasing data warehousing costs. Every time a data engineer creates a new data transformation model, fixes a bug, tests a fix, etc., they’re consuming more data warehouse compute. If their SQL is incorrect or malformed in any way (using an invalid column name, spelling a keyword wrong, etc.), they’ll only discover this after running the code remotely.

Magnify this by dozens or hundreds of data and analytics engineers across a company, and you’re talking a significant impact on your bottom line.

How the dbt Fusion engine cut licensing costs

The new dbt Fusion engine solves exactly this problem. Fusion is a complete rewrite of dbt in Rust that natively understands SQL across multiple data warehousing engine dialects. This gives dbt a powerful new set of local development capabilities. With Fusion, data engineers can:

• Catch SQL errors immediately as they type in Visual Studio Code, Cursor, or dbt Studio
• Preview inline Common Table Expressions (CTEs)
• Trace model and column definitions across your dbt project

Besides accelerating data pipeline development times, Fusion helps companies save on costs in three ways:

• Ahead-of-Time (AOT) compilation
• State-aware orchestration
• Super-fast performance

Ahead-of-Time (AOT) compilation

The key is that Fusion can do all of this without ever connecting to your data warehouse. That’s because Fusion uses Ahead-of-Time (AOT) compilation. Instead of running SQL directly in your data warehouse (the JIT model), AOT comprehends and analyzes your project locally, statically analyzing every model’s logic plan before running anything in your data warehouse.

In other words, developers can now be assured that their models are syntactically correct before running them. That cuts down round-tripping to the data warehouse, which reduces compute costs.

State-aware orchestration

Fusion also grants dbt another superpower. It uses state-aware orchestration when running Continuous Integration (CI) jobs in a dbt environment, further reducing data warehouse costs. State-aware orchestration is now in preview for Fusion projects.

State-aware orchestration maintains a real-time shared state of every model in a project. This means it knows if, for example, Job 2 is using an upstream model that Job 1 just rebuilt a few seconds ago. In this case, it will use the results from Job 1’s run, saving you from an unnecessary rebuild.

State-aware orchestration implements a model-level queue, which avoids collisions and prevents unnecessary model rebuilding. It’s also simple to configure, as it works out of the box and is Fusion’s default for model builds. You can use advanced configuration to further tailor Fusion’s builds to meet your specific needs.

Built for speed

Finally, Fusion saves on cost and time through sheer performance. Whereas dbt Core was written in Python, the dbt Fusion engine is written in Rust. As a compiled binary written in a high-performance language, it parses even large projects up to 30x faster than dbt Core. This translates to less compute time for data pipeline jobs - as well as faster development times for data engineers.

Conclusion

ETL has been around for a while. However, while the data industry has changed significantly in the past two decades, many ETL tools have failed to keep up. Fixed-price tools are too rigid to meet the rising demands for data. On the other hand, most pay-as-you-go tools don’t provide any out-of-the-box tools to monitor and control costs.

The dbt team has always strived to save our customers money by providing them with tools to keep the overall price of data transformation as low as feasible. The dbt Fusion engine builds on this legacy. Features such as Ahead-of-Time compilation and state-aware orchestration enable companies to scale data development without breaking the bank.

Sign up for an account and walk through our dbt Fusion quickstart today.