Blog Frontiers of the Semantic Layer: Extending the dbt Workflow

Frontiers of the Semantic Layer: Extending the dbt Workflow

The dbt Semantic Layer extends the portfolio of information managed in dbt to move dbt Labs beyond the data team. Read now
Frontiers of the Semantic Layer: Extending the dbt Workflow

Hi, I’m Cameron Afzal, dbt Labs’ new Product Manager for the dbt Semantic Layer.

Before joining dbt Labs, I helped build an analytics product that enabled users to understand changes in their metrics. I saw how dbt transformed the way customers prepared data for analysis so they could use the product. We integrated with dbt Cloud to make it easier for them. 

Then I read Drew’s post about the dbt Semantic Layer and knew I wanted to join dbt Labs to help organizations make better decisions. I saw an opportunity to help people move up the stack by increasing their efficiency and contribute to the knowledge loop by expanding the portfolio of information they contribute to and access. 

Today, I’m introducing how we equip data practitioners to capture more business logic (metrics!) in dbt projects and enable data consumers using downstream applications to leverage their work. 

We’re laying the foundation with the Public Preview release of the dbt Semantic Layer and select partner integrations at Coalesce, and are excited to keep building upon it. 

Extending the value of dbt

Organizations need easier, faster ways for people to ask questions on their own and trust that they’ve arrived at consistent and correct answers. I hear these challenges from dbt Cloud customers:

  • “When I get asked a question about revenue… one time I’ll use this definition, another time another… it gets confusing for a lot of people.”
    – Pieter Klinkenberg, Data Analyst at GO Sharing
  • “I dislike having… multiple places to update because I am bound to miss one.”
    – Trish Pham, Head of Analytics at Payjoy
  • “The discoverability is a thing that we’re trying to solve… more on the end-user/stakeholder side.”
    – Erik Jones, Head of BI and Analytics at Hyperscience
  • ”[We want] a controlled interface for models…a portal into measurements, hard to query events…[with] core dimensions to answer more questions.”
    – Data Engineer at an online marketplace company

The complexity of these challenges is only increasing as data sources and use cases multiply. Organizations often define business logic like metrics in various downstream tools, resulting in inconsistency, rework, and siloes. 

dbt streamlines governance and manages the complexity of analytics data pipelines and business logic. The dbt Semantic Layer extends the portfolio of information managed in dbt. 

With this, the value of dbt is extending even further beyond the data team. The dbt Semantic Layer will help more people find the information they need to make quality decisions: consistency and precision in every consuming experience. As dbt Semantic Layer beta users said:

  • “Everyone wants data in our company…we keep adding a lot more…we want to build a good foundation…this is a really cool feature that can help us a lot.”
    – Pieter Klinkenberg, Data Analyst at GO Sharing
  • “My hope is everyone who touches dbt… start[s] to leverage metrics.”
    – Trish Pham, Head of Analytics at Payjoy
  • “I see us moving everything towards the Semantic Layer…instead of relying on LookML.”
    – Erik Jones, Head of BI and Analytics at Hyperscience
  • “The Semantic Layer + Hex integration helps to bring more of the data processing work into the tightly-managed dbt layer. This reduces the speed to valuable data products…[and] improves their long-term governance, making them more reliable.”
    – Data Engineer at online marketplace company

Extending the dbt workflow

The dbt Semantic Layer is the interface between your data and your analyses: A platform for compiling and accessing dbt assets in downstream tools.

Data practitioners can define metrics in their dbt projects, then data consumers can query consistently defined metrics in downstream tools.

We’re working with ecosystem partners to ensure the ubiquity and utility of their integrations to meet organizations’ needs. Here are some use cases the dbt Semantic Layer powers:

  • Prepare via data integration tool: A data engineer uses packages to load metrics from common data sources, saving time as they migrate their organization’s logic and analysis to the Semantic Layer.

    • Fivetran packages enable easy metric definition from common sources.
  • Discover via data catalog: An analytics engineer explores a metric by viewing its definition, if it’s verified, and how it’s used; then previews its values to determine whether to use it or create a new one.
  • Analyze via notebook: A data analyst diagnoses why a metric changed by instantly generating a dataset from the metric, incorporating more datasets and analyses, and quickly iterating towards the answer for their stakeholders.
    • Hex and Deepnote power fast and trustworthy iterative analysis.
  • Report via BI tool: A business user selects from a list of pre-defined metrics and analyzes them with relevant dimensions for their weekly business review without waiting for a data analyst to prepare a report
  • Monitor via observability tool: A data scientist proactively monitors a metric for unusual and dramatic movements, analyzes root causes, and uses alerting to monitor lineage and freshness.
    • Anomalo automatically generates daily anomaly checks for dbt Metrics.

See our Semantic Layer and Metrics Ready integrations page for a complete list of current partner integrations. 

Opportunities on the horizon

Creating an interoperable, scalable semantic layer has been a longtime dream. We are committed to working with the community to build the dbt Semantic Layer as a universal standard that data practitioners, data consumers, and ecosystem partners rely on for critical workflows. Getting there will take time. Here’s how we see the road ahead. 

Enable efficient and comprehensive metric and data modeling

We’re aiming to equip data practitioners making use of dbt metrics with improved ergonomics and capabilities.

  • Provide more efficient ways to specify metric dimensions, time grains, and filters with modular and reusable code.
  • Introduce advanced options for metrics versioning.

We called it the “Semantic” Layer for a reason. 

  • Unlock the dimensional richness of entire dbt projects via model relationships (joins). 
  • Increase discoverability with structures to streamline metrics organization.

Provide reliable infrastructure and an accessible analytics engineering experience

We’re focusing on enhancing dbt Cloud for customers to get up and running at scale.

  • Improve onboarding flow and environment management. 
  • Build a cohesive experience to develop and test metrics in the IDE, fitting into existing dbt workflows.
  • Enable granular, enterprise-grade access management.
  • Optimize the performance of query compilation (100 ms p95 goal) and the reliability of our highly available services (99.9% SLO).

Nurture the ecosystem to unlock new and improved workflows

The first slate of dbt Semantic Layer integrations is rolling out at Coalesce. We are working with our partners to ensure that the dbt Semantic Layer is as useful to you and your stakeholders as possible. Here are some ways we’re aiming to accomplish that:

  • Provide increased assistance for partners to build integrations (e.g. new and richer APIs).
  • Enable users to manage and access integrations directly through dbt Cloud.
  • Support additional data platforms to query the Semantic Layer.
  • Encourage community contributions to dbt Core and packages.

Getting started

The journey of the dbt Semantic Layer is just beginning. The Public Preview period allows dbt Cloud customers to put it through its paces with real-world, production use cases and provide feedback. Team, Enterprise, and trial customers can use it with partner integrations that leverage the Proxy Server and Metadata API. Developer accounts are able to query the Proxy Server using dbt-SQL. We’ll announce a timeline for the  Generally Available release in the future. 

If this sounds interesting, please see the docs to get started and engage with technical partners to start using their integrations. If you’re with a current or potential technology partner and would like to develop an integration, please reach out. And to the dbt Community more broadly: my door is always open in #dbt-cloud-semantic-layer or via email if you have questions or thoughts on our direction. I’d love to hear from you.

Last modified on: Oct 18, 2022