Blog Introducing the revamped dbt Semantic Layer Beta

Introducing the revamped dbt Semantic Layer Beta

Nick Handel

Aug 02, 2023

Product News

In February, we shared that dbt Labs was acquiring Transform. That moment marked the beginning of a new chapter for both organizations. dbt Labs exists to help data teams ship high-quality, reliable data products faster; Transform’s technology was mature and would help extend that vision to metrics with an improved Semantic Layer.

Now just a few months later, we're excited to share that the revamped dbt Semantic Layer is available as a Public Beta, bringing consistent metrics to the forefront of our shared vision.

When Tristan and I first spoke about the possibility of an acquisition of Transform, my mind immediately began to brim with product ideas. Building metrics is the final step of data transformation, and dbt was ubiquitous as the input to our work on MetricFlow. The possibility of joining together represented to me the opportunity to build a seamless end-to-end experience, from data transformation to consumption.

This Beta is our first step toward that vision for the universal semantic layer at dbt — critical logic governed from ingestion and transformation to consumption and knowledge creation.

The new dbt Semantic Layer: powered by MetricFlow

Now powered by MetricFlow, a state-of-the-art query generation engine, the new Semantic Layer has undergone a step-change in functionality from the previous iteration.

We’re launching many of the most requested features from the previous dbt Semantic Layer, enabling more complex metric definition and querying to be done efficiently at scale:

1. Join navigation. Support for joins is critical to a successful semantic layer. You’ll now be able to maintain DRY, reusable components to construct metrics and join them to dimensions, which can save a huge amount of time in aggregate. To enable joins, we’ve added a concept called entities, which allows us to establish join paths between your dbt models.
2. Expanded data platform support. The Semantic Layer now supports Databricks, Snowflake, BigQuery, and Redshift. This includes performance optimizations for each data platform.
3. Powerful new integrations. Starting today, you can use the new Semantic Layer in Mode, and a Hex integration is set to follow shortly. Both teams have been great partners from the early days of our previous-generation product. We're collaborating closely with them to enhance their integrations, ensuring compatibility with the new Semantic Layer.
4. Optimized query plans and SQL generation. With a focus on producing legible and performant SQL, the new Semantic Layer generates joins, filters, and aggregations just as an analyst would — without the need for complicated tricks that yield illegible SQL, such as symmetric aggregates.
5. Semantic Layer APIs. We’ve also built new APIs, including an entirely rebuilt JDBC interface built with ArrowFlight, allowing for more seamless integrations and applications to be built on top of the new Semantic Layer.
6. More complex metric types out of the box. Enabling new aggregations and more flexible metric definitions means you can define more of the metrics that are critical to measuring your business in the dbt Semantic Layer.

Continued investment

We’ve made a significant investment in the foundation that enables us to build exciting features at an exciting pace in the coming months. The team is already working on several features which bring light to the long-term potential of the new dbt Semantic Layer:

1. Caching. We’re working on using the data platform to cache data that is most commonly queried through the Semantic Layer, leading to cost savings as well as faster load times in downstream applications.
2. Metric types. Our new metrics spec will enable complicated metric types like conversion rates, extending the Semantic Layer to more use cases.
3. Query flexibility. We will continue to make our querying interfaces more flexible by adding additional data transformation capabilities like time-over-time changes, metrics in where clauses, and more.
4. Additional APIs and integrations - We plan to release a GraphQL API in the coming weeks. We're also actively working on some very exciting new integrations: we aim to integrate with all your favorite data and analytics tools, so please let us know where you’d like to use dbt metrics.

If you’re interested in this work, please reach out in the dbt Community Slack.

Getting Started

We've revamped the spec for defining metrics, and you can learn more about that in our blog post announcing that work. With these changes, we will be deprecating the original metric spec, along with the dbt_metrics package, which you can read about here.

If you're eager to use the Semantic Layer you can check out our new quickstart guide to get your organization up and running fast. We've also prepared a detailed guide for those who've been using the original Semantic Layer to help you transition smoothly to the new version.

If you’d like to see a production example of the new metrics spec in action, we’ve released a public example repo. We encourage you to join the conversation for all things Semantic Layer in the #dbt-cloud-semantic-layer channel in the dbt Community Slack.

dbt Cloud and the Semantic Layer

The dbt Semantic Layer will be available as a paid component of dbt Cloud at the end of our Public Beta in October. For any post-Beta pricing questions, please reach out to your dbt Cloud account representative or reach out here. There is no cost to using the Semantic Layer during our Beta.

It’s available on the Team and Enterprise plan for all customers on North America, multi-tenant deployments of dbt Cloud. We're working to expand availability of the Beta as quickly as possible to other regions in the coming months.

Looking Forward

You can start using the new-look dbt Semantic Layer in dbt Cloud today. We couldn’t be more excited to announce this milestone and hope you’re just as excited to get started using it.

We believe this technology has the potential to unlock new data applications, deepen trust in data teams, and reduce inequality in access to data across organizations. We have our work cut out for us in the coming months. We're eager to hear your feedback or answer any questions you may have, so get started today with our quickstart guide, and please reach out in the dbt Community Slack to let us know what you think!

Last modified on: Feb 21, 2024