Using dbt with Databricks: Architecture decisions that determine success

This guest post comes from Keith Ludeman, a solution architect at Analytics8.

Databricks gives you the platform. dbt gives you the structure, speed, and operating layer required to make it work for your whole organization. But getting the combination right requires decisions most teams underestimate, and the cost of getting them wrong compounds over time.

I work with organizations at every stage of their Databricks journey—teams standing it up for the first time, and teams that have been running on it for years and are hitting walls they didn't expect. The question I hear most often: do we really need dbt?

The answer, in almost every case, is yes. But when you introduce it, and how, depends on where you are.

Below I’ll walk you through:

• The cost of going Databricks-native without a transformation framework
• What dbt adds to a Databricks environment
• Starting fresh: Why dbt belongs early in a Databricks implementation
• Signals it's time to introduce dbt in an existing Databricks environment
• Common questions and objections About dbt with Databricks
• Practical lessons from real implementations

The cost of going Databricks-native without a transformation framework

Databricks gives you a lot of power right out of the gate. You can land data, shape it, orchestrate pipelines, run models, and serve analytics, all in one place. It is flexible enough to support almost any approach, which is exactly why teams lean into building things their own way early on.

That usually starts small. An engineer writes a few transformations in notebooks. Someone wires up orchestration to keep things moving. As more use cases come in, more logic gets added, often by different people, each solving for the problem in front of them. Nothing feels wrong in the moment. The system is working.

The problems show up later.

As the environment grows, it becomes harder to answer basic questions. What does this model do? Where is this logic defined? If I change this, what breaks? There is no single way of doing things, so every answer depends on who built it and when. What used to feel flexible now feels unpredictable.

That is where the friction starts to build:

• You lose consistency across the environment. Different engineers structure transformations in different ways. Over time, it becomes difficult to follow the logic, compare approaches, or confidently reuse anything.
• Technical debt builds quietly. What began as a handful of workflows turns into a network of dependencies that no one fully documents or owns. Testing becomes an afterthought; teams end up with basic row counts to confirm something ran, rather than automated, granular validation. Even small changes require extra caution, and new engineers need time just to understand how things fit together.
• Your team spends time maintaining the system instead of improving it. Instead of focusing on analytics, you end up managing orchestration, testing patterns, and documentation standards on your own. The platform starts to demand attention rather than enable progress.
• Issues surface late, and often without clear signals. Without consistent testing, lineage, and dependency tracking, problems do not show up where they start. They show up downstream, in dashboards, reports, or decisions, where they are harder to trace and fix.

Most teams do not decide to build their own transformation framework. It happens gradually, through a series of reasonable choices. By the time it becomes a problem, you are not just dealing with messy code. You are dealing with slower delivery, rising costs, and a growing lack of confidence in the data.

What dbt adds to a Databricks environment

Before getting into what dbt adds, it’s worth clearing up a common misconception. dbt does not replace Databricks, and it is not another engine doing the work somewhere else. Your data still lives in Databricks, and your transformations still run there. dbt is simply the layer that defines how those transformations are written, tested, and maintained over time.

That distinction matters, because most of the issues that show up in a Databricks-native environment are not about the platform itself. They come from how transformation logic evolves as more people start contributing to it.

Without structure, that logic tends to live wherever it was first created. A notebook here, a pipeline there, maybe a few different patterns depending on who built what. It works, but it does not give the team a consistent way to understand or extend what is already in place. dbt changes that by giving you a single, shared way to define transformations:

• You can follow how data moves from one model to the next.
• You can see dependencies instead of guessing what might break
• You have testing, documentation, and version control built into how the work gets done

The shift is subtle at first, but it changes how teams operate.

Engineers spend less time figuring out what already exists and more time building on top of it. Analysts are not blocked by how the data was originally created. And when something looks wrong, they can trace the logic themselves, propose a fix, and have it reviewed before anything changes in production. New team members do not have to reverse-engineer the environment just to contribute. The work becomes easier to reason about, which makes it easier to scale across teams.

That same structure becomes critical when you layer AI or advanced analytics on top of your data. Weak points show up quickly, and if transformations are hard to trace or validate, those issues do not stay contained. They surface in downstream outputs, where they are harder to catch and more expensive to fix.

A structured transformation layer gives AI systems the context they need to return trustworthy answers and makes it easier to identify sensitive fields before they surface somewhere they shouldn't.

Where the dbt platform advances this further

As teams grow, the challenge usually shifts from how transformations are written to how they are run.

In many dbt Core setups, orchestration becomes something you have to solve alongside everything else. Jobs are scheduled outside of dbt, dependencies are managed across tools, and over time you end up with another layer of logic that someone on the team is responsible for maintaining.

The dbt platform changes that dynamic by pulling orchestration into the same place where transformations are defined. More importantly, it adds awareness of the data itself.

If nothing upstream has changed, there is no reason to run the same transformations again. Instead of executing full pipelines on a fixed schedule, you run only what needs to run based on the state of the data. At smaller scale, that might feel like an optimization. At larger scale, it directly affects both cost and reliability.

It also changes the day-to-day experience in ways that are harder to quantify but easy to feel over time. When you can trace dependencies more precisely, make changes across multiple models with confidence, and catch issues before anything runs in Databricks, the work becomes less about managing risk and more about moving forward.

Why dbt belongs early on a Databricks implementation

If your organization is implementing Databricks for the first time, the instinct is to start simple. Get Databricks running, prove it out, and add tooling later. That instinct makes sense. It is also where the retrofit tax begins.

Most engineering teams have deeper SQL expertise than Spark or Python expertise. dbt is SQL-native, which means your team can start contributing immediately without a steep learning curve or relying on a small group of Spark specialists. It also changes your hiring options. SQL talent is easier and more affordable to find, and that advantage grows as your team scales.

Most organizations are not starting from scratch. They are migrating from something else, whether that is SQL Server and SSIS, Google Dataform, or another SQL-based transformation environment. In those cases, the transformation logic is already written in SQL, and it ports cleanly into dbt.

We have agentic migration accelerators that automate 70–80% of that work, compressing what would otherwise take years into weeks. That advantage depends on staying in a SQL-native model. If you go Databricks-native, you are rewriting that logic in a different paradigm.

If you are building the platform now, this is the least expensive moment to make the right architectural decisions. A few foundational choices made at the start will determine whether your platform is easy to scale or expensive to untangle:

• Adopt Unity Catalog from day one. Don’t take shortcuts. It is far easier to implement correctly upfront than to retrofit after you have already built on top of a structure that does not support it.
• Separate compute by workload from the start. Segregating compute for ingestion, transformation, and reporting gives you cost visibility and performance control. You can see where costs are coming from and tune each workload independently. Without that separation, costs become harder to interpret and performance becomes harder to improve.
• Don’t reinvent orchestration with dbt Core. Starting with the free version and building orchestration, testing, and observability yourself shifts the cost into engineering time. In practice, that effort often exceeds the cost of using a managed solution.

The decisions that are cheap to get right at the start are expensive to fix later. Starting with dbt isn’t adding complexity. It’s avoiding it.

Signals it's time to introduce dbt in an existing Databricks environment

Some organizations bring dbt in from day one. Others start Databricks-native, run a successful small team, production environment, or proof of concept, and then hit a wall. These are the signals that tell you it is time to introduce, or reinforce, structure.

• Your team has grown past two or three people. A small team can move quickly in Databricks without much formal structure. That breaks down once the team grows or when business analysts start contributing as analytics engineers. At that point, you are dealing with concurrency and change control. Multiple people working in the same environment without proper CI/CD and version control leads to conflicts, overwritten work, and compounding risk. dbt provides the structure that makes that collaboration manageable.
• You’ve added a second data domain. A proof of concept built on sales data works. Leadership pushes to bring in supply chain. The moment you move beyond a single domain, both your data structures and your team structures need to evolve. Without that shift, each new use case becomes harder to add, maintaining what already exists gets more expensive, and both technical and organizational debt start to build. It is far easier to address this before the second domain is fully in production.
• The data team is still a bottleneck. If business users are still waiting on IT for analytics, or if analysts are exporting data and rebuilding logic on their own instead of working from shared, governed definitions, your operating model has not kept pace with your platform.

“I don’t want to be the chokepoint. I want to activate the business.” That is what you hear from data leaders at this stage.

dbt makes it possible to bring business analysts into the analytics engineering process. Business analysts already know SQL. Very few know Spark. A SQL-based transformation layer helps close the gap between IT and the business, both technically and organizationally.

When teams make the switch, the first thing they feel is speed—not in terms of compute, but in how quickly someone can answer a question. When an analyst asks why a number looks off, the engineer is not launching a research project. The logic is traceable from the final output all the way back to the source, model by model, in a way that is easy to follow and fast to navigate. Engineers also feel the difference in how safely they can make changes. With CI/CD guardrails baked into how dbt is structured, a change gets tested and reviewed before it touches production. That security changes how the team works—less caution, more momentum.

Common questions and objections about dbt with Databricks

Can't we just do this natively in Databricks?

You can. Databricks is flexible enough to support almost any pattern. The tradeoff shows up over time in development effort, maintenance, velocity, and risk.

dbt establishes the many patterns that teams often try (but fail) to build for themselves. Testing, documentation, version control, and standardized transformations are not new problems. dbt gives you a consistent way to handle them without reinventing that layer inside notebooks and pipelines.

When does dbt become necessary?

It rarely comes down to a single moment. What changes is the cost of continuing without it.

The longer a team operates without structure, the more expensive it becomes to introduce it later. Patterns that are easy to establish early require rework once pipelines, dependencies, and team processes are already in place.

If you are seeing the signals from the previous section, the cost has already started to compound. And if your organization is starting to think seriously about AI, that is another clear signal. The foundation of clean data, traceable logic, documented definitions that AI requires, is the same foundation that dbt helps you build.

Isn't dbt expensive?

dbt Core is free and can even be run within a Databricks job. That makes it a practical starting point. The real question is where the surrounding work lives. Orchestration, testing, observability, and maintenance do not go away with Core. They move to your team.

The dbt platform shifts responsibility off your engineers. In many cases, the total cost balances out, while the operational burden drops.

How do teams decide between dbt Core and the dbt platform?

Most teams start with dbt Core because it is free and easy to spin up. That is a reasonable starting point, and for smaller teams it may be all you need. The conversation shifts, however, when complexity grows. When orchestration requires a separate tool, when you need more sophisticated dependency management, when the surrounding work starts pulling your engineers away from the analytics itself. At that point, the dbt platform tends to make sense. The licensing cost is relatively modest, and what you get in return (orchestration, testing, observability, all in one place), typically offsets what you were spending in engineering time to maintain everything separately.

Practical lessons from real implementations

Based on our experience implementing dbt with Databricks, here are a few practical recommendations to consider:

• Use each tool for what it does best. Databricks is the right place to land and organize raw data. dbt is the right place to transform it into something the business can use. Teams that blur that line end up with an environment that is harder to maintain and harder to explain.
• Keep all transformation logic in dbt. This is the single most common pattern we see in struggling implementations: logic split between notebooks and dbt models. It starts with one exception and compounds from there. Once logic lives in multiple places, lineage breaks down, debugging becomes a research project, and onboarding new team members takes significantly longer.
• Don't let logic creep into orchestration. Orchestration should trigger work, not define it. When business logic gets embedded in orchestration tools, it becomes invisible to the rest of the pipeline. Changes to source systems break things in ways no one anticipates, because the logic affecting the data isn't where anyone thinks to look.
• Adopt Unity Catalog from the start. Teams that skip it consistently pay for it later. It is one of those decisions that is low cost to get right early and high cost to retrofit once an environment has been built on top of something else.

When these things are in place, the team stops spending its time maintaining pipelines and debugging failures. It starts spending that time on what matters—expanding capabilities, supporting new use cases, and engaging with the AI and advanced analytics work the business is asking for.

Keith Ludeman is a Solution Architect at Analytics8 (a dbt Labs Visionary Consulting & Services Partner) who specializes in designing and recovering complex data platforms across Databricks, Snowflake, and dbt, from migrations to governed transformation layers. He focuses on uncovering the root causes behind delivery challenges and putting the right structure in place so teams can scale without rework or technical debt. Known for stepping into high-risk situations and bringing clarity, he approaches each engagement as a long-term partner rather than a tool-driven implementer.