Best of both worlds: dbt + Databricks Lakeflow Jobs

Picture this: your data engineering team builds ingestion pipelines in Databricks, your analytics team transforms data with dbt, and your ML engineers train models—all in completely separate workflows. Sound familiar? Those days are officially numbered.

The combination of dbt and Databricks has already proven transformative for countless data teams globally, streamlining transformations while leveraging the lakehouse's power and scale. But what if you could take that winning combination even further? What if you could seamlessly weave dbt jobs directly into your Databricks workflows, creating one unified pipeline that speaks every team's language?

That's exactly what we’ve built.

What's new: dbt Platform Task Type in Databricks Lakeflow

The dbt Platform Task Type is now available in Private Preview, allowing you to orchestrate dbt Platform jobs directly within Databricks Lakeflow Jobs.⁠ This represents a big step forward in how data teams collaborate.

With Public Preview expected in time for Coalesce,⁠ this integration represents the first time you can truly unify your ingestion, transformation, and AI pipeline orchestration in one place.

Why it matters

End-to-end pipeline orchestration made simple

Constantly switching between interfaces can disrupt ‘flow state’, and in so doing, detract from developer productivity. It’s one of our convictions that good developer tools should promote flow state.

Gone are the days of managing separate orchestrators for different parts of your data pipeline, and no more custom API calls to get Databricks and dbt orchestration working together. Now, you can manage raw data ingestion, streaming, batch transformation, and AI/BI in a single Databricks Lakeflow Jobs interface

Meeting your teams where they are

dbt offers three developer interfaces – Studio, Canvas, and CLI – and now your teams can productionize their dbt pipelines within Databricks Lakeflow Jobs without losing any of that rich development experience.⁠

Unlock the power of dbt Mesh

Here's where it gets really exciting: cross-project and cross-platform orchestration capabilities are only available through dbt Platform jobs.⁠ This means you can respect team-specific orchestration (per domain project) while integrating those into global pipelines, delivering true dbt Mesh architecture at scale.

Centralized control without compromise

Organizations standardizing on Databricks and dbt get the centralized scheduling, retries, alerting, and audit trails they need,⁠ while development teams retain the flexibility and power of dbt features like enhanced IDE integration and cost optimization.⁠

The technical sweet spot

Because dbt deeply integrates with Unity Catalog, you also get full metadata syncing and governance,⁠ while enhanced developer experience through dbt Platform's Jobs integration with Databricks⁠ means your teams get the best of both worlds: enterprise-grade orchestration with developer-friendly transformation tools. Plus, with improved data lineage, documentation, and performance metrics flowing seamlessly between platforms, you finally get that single source of truth.

What's next?

If you're using Databricks with dbt, this integration isn't just a nice-to-have; it's the missing piece that transforms how your entire data organization collaborates. Private Preview customers are already testing this,⁠ with Public Preview launching by Coalesce in October.⁠

Ready to see what unified data and ML orchestration looks like in practice? The future of collaborative data development is here, and it's running on the platform you already know and love.

Want to learn more about the dbt Platform to Databricks Lakeflow Jobs integration? Contact your dbt Labs or Databricks representative to discuss joining the Public Preview program launching at Coalesce.