What's new in dbt Cloud - April 2024

Our EPD team has been hard at work building new features, user experiences, and integrations for dbt Cloud. From column-level lineage in dbt Explorer to the GA of our Tableau integration for the dbt Semantic Layer to features like unit testing that make analytics engineering more powerful…there’s a lot of newness in our platform! We consolidated it all for you in one easy-to-read blog post, so let’s dive in to what’s new 👇.

🔎 dbt Explorer

Our vision is for dbt Explorer to be the best place for data teams to discover, understand, and optimize their dbt Cloud projects so downstream teams can leverage them with confidence. Here’s what’s new:

• 🕵️ Model performance evaluator. Quickly identify models across your dbt projects that could be optimized so you can keep your dbt estate performing as efficiently (and cost effectively) as possible. Check out the blog post to learn more

• 💡 Project recommendations. Use dbt Cloud metadata to proactively surface ways to improve the test coverage, documentation, and overall project health of your dbt models. Now, data teams can tackle project improvements on their own time to ensure that pipelines are performant and data trust remains solid. Read the docs to learn more.

• 🪲 Column-level lineage. dbt Explorer and the Discovery API now provide column-level lineage for models, sources, and snapshots within a dbt Cloud project. Column-level lineage can be used to improve many data development workflows including auditing, root-cause analysis, and impact analysis. It’s available in beta today for dbt Cloud Enterprise plan customers. Read our technical blog post to learn more.

• 🪄 Better search and lineage experience. We’ve improved the keyword search interface to make it more intuitive to find the resources and columns you need and filter down results. Lineage is also more performant at scale, providing more navigation options and supporting all common dbt selector methods.

• 🕶️ Lineage lenses. Think of lineage lenses as map layers for your DAG: they make it easier to understand your project’s contextual metadata at scale. Lenses currently supported include resource type (default), materialization type (e.g. identify incremental model dependencies), last execution status (e.g. diagnose a failed DAG region), and model layer (e.g. discover marts models to analyze). Read the docs to learn more.

📈 dbt Semantic Layer

The dbt Semantic Layer team has been hard at work since our GA at Coalesce 2023. Keep scrolling for what’s new, catch the recent webinar, or peruse customer FAQs here.

• 💾 Saved queries & exports. Exports allow you to materialize a saved query as a table or view in your data platform. This lets you unify metric definitions in your data platform and query them as you would any other table or view, making it possible to make your metrics available in any analytics platform—even if we don’t yet have a native integration with it. Check out this video of Jordan demoing the feature, or read the blog!
• 🗄️ Result caching. Improve load times and reduce compute costs by leveraging your data engine’s built-in cache for semantic layer queries.
• 🔒 SSO and PrivateLink support. You can now develop against and test your dbt Semantic Layer in the Cloud CLI if your developer credential uses SSO 🙌. The dbt Semantic Layer also now supports using PrivateLink.
• 🌎 Integrations: The Tableau connector is now GA for Tableau Server and Desktop! You can get the latest on all of the dbt Semantic layer integrations here.

💻 Developer experience

• 🚀 Trigger on job completion. You can now set up downstream jobs to kick off as soon as an upstream job finishes — even across dbt projects. This lets you orchestrate dbt jobs more flexibly, and easily break up long-running jobs into multiple parts. This is available on dbt Cloud Team and Enterprise plans today.

• 🏎️ Improvements to parse speed. Every time you type dbt build and hit enter, dbt goes through a process we call “parsing.” We’ve rewritten some of the internals of dbt Cloud so that your projects now parse 30% faster than on an M1 Mac. (This requires you to be running “Keep on latest version” in dbt Cloud.) … But that’s not all! You can also now enable partial parsing on your project to parse only changed files, making this process even faster.
• 🧪 Unit tests. Validate your model logic before you materialize your full model in production. This means you can improve your test coverage without driving up data platform spend. Available in Preview for dbt Cloud customers who have opted to “Keep on latest version.” Read the docs or check out this blog post to learn more about how to approach data testing in dbt.

🤓 Platform

• 💫 Keep on latest dbt version. dbt Cloud should feel and function like the other SaaS apps your team uses: you shouldn’t have to manually upgrade versions under the hood. Now in Preview, just select “Keep on latest version” in your environments and jobs to get immediate access to the latest and greatest. Read our recent blog post for more!

• 📊 Cell-based architecture. As your dbt Cloud workloads grow, we’re committed to meeting those increasing demands—with a new architecture designed for improved reliability and scale. We’ll reach out to account admins over the course of 2024 with info on specific migration steps, if any are necessary for your account. More in this blog post.
• 🦸 Git repo caching. Sometimes the things that cause jobs to fail are external to dbt: things like Git provider outages or package dependencies. dbt Cloud can now cache your project’s Git repository, ensuring that your jobs continue to run seamlessly, even if external services are temporarily down. This is available for dbt Cloud Enterprise plans only.
• 🤐 Account-scoped Personal Access Tokens. These provide a more granular level of access control, scoped specifically at the account level rather than the user profile. This prevents reuse of tokens across accounts, reducing the risk of compromised credentials. Read our docs page for more information.
• 🏃 New “job runner” permission. As the name implies, this limited permission-set can kick off a run for a job or cancel an in-flight run. You may find this helpful to assign more fine-grained access controls in larger dbt deployments. See the docs for an updated map of available permissions on the dbt Cloud Enterprise plan.

👭 Partnerships and integrations

• 👋 MS Fabric adapter. Hello Microsoft ecosystem! dbt Cloud is now available for Microsoft Fabric and support for Microsoft Azure Synapse Analytics is coming soon. Check out the blog to learn more.
• 🔀 Fivetran dbt Cloud integration. Our friends at Fivetran built an integration into dbt Cloud so you can automate your end-to-end ELT pipelines. Configure the integration to trigger dbt Cloud jobs to run as soon as new data is loaded into your data platform. Read the blog to learn more.

Wrapping up

Phew…that was a ton! Looking forward to hearing your feedback on these latest features and continuing to build new functionality that helps you build and deliver trusted data.