Blog Navigate and understand your dbt Cloud projects with dbt Explorer

Navigate and understand your dbt Cloud projects with dbt Explorer

Cameron Afzal

Nov 28, 2023

Product News

We’re excited to share more details about dbt Explorer, a new dbt Cloud feature we launched last month at Coalesce. Built on our rich and performant metadata platform, dbt Explorer is designed to help our customers visualize, understand, and manage their dbt projects at scale.

As our customers’ dbt projects scaled, so did the complexity of their data estate. They became mired in visibility roadblocks, reactive development workflows, and impaired performance. That’s why we built dbt Explorer: to help you visualize and control your data estate—at scale—so you can spend less time duplicating work, deciphering lineage, and debugging issues, and spend more time shipping quality data products faster.

Check out this quick video and keep reading on to learn more.

We’re hosting a webinar on December 5th to dive into dbt Explorer (and dbt Mesh!) in action. Sign up here.

Unlock metadata to manage complexity

dbt Explorer offers a birds'-eye view of your data pipelines so you can quickly understand your dbt projects and dependencies and orient to available resources to use for development or analysis. dbt Explorer provides documentation enriched after each dbt Cloud run and uses the Discovery API as its robust source of truth for production metadata.

Using dbt Explorer, you can visualize your dbt lineage and zoom in to interact with specific nodes. When you click on a specific node, you’ll immediately get served relevant context like run status, description, and more. You can also use global search to find specific resources and understand their upstream and downstream dependencies.

Given the size and complexity of Sunrun's dbt infrastructure, dbt Explorer is an ideal tool to provide our data engineers and analysts with quick, detailed visibility into all aspects of our environment.



– Gavin Mackinlay, senior data engineer at Sunrun

dbt Explorer empowers you to:

1. Discover data assets and intuitively navigate them within dbt Cloud.
2. Streamline data development and analysis workflows with an intuitive user interface.
3. Enhance organizational awareness of available resources across all dbt projects.
4. Understand data assets holistically by viewing their quality, governance, and relationships.

dbt Explorer is also an essential tool for organizations leveraging multi-project development through dbt Mesh. It powers the central visibility essential for distributed collaboration at scale.

Data discovery that scales with your organization

dbt Docs is a valuable and widely adopted product—nearly half of dbt Cloud customers have used it in the past month. And for good reason: auto-generating documentation alongside code development and deployment streamlines workflows and provides critical context as teams build and improve pipelines.

However, dbt Docs hits roadblocks when applied at enterprise scale. Once an organization has a critical mass of dbt project contributors and models, it’s hard to make sense of what’s available to leverage existing assets, let alone develop new ones. And this is a real and growing concern for a large contingent of our customer base: there are thousands of big projects, each with more than one thousand models, who struggle to use documentation that's 'single-player' mode. This inevitably slows down development, creates development siloes, and introduces risk in stakeholder relationships with untrustworthy and slow data pipelines.

“Explorer has a better UI experience than dbt Docs and the data model lineage is clearer…[it] has better model documentation and model quality testing which is useful to users that are not familiar with the data models.”



– Yuna Tang, senior analytics engineer at SafetyCulture

dbt Explorer was built for enterprise scale. In addition to supporting dbt Mesh with multi-project lineage and model governance features built-in, dbt Explorer performs better than legacy Docs and provides a foundation for future capabilities. Here’s how they compare:

Usage patterns across teams

Have you ever gotten lost trying to find the right data to answer your question or help your stakeholder answer theirs? dbt Explorer gives you the visibility and shared context you need to find the answers you’re looking for and helps data teams:

• Work faster. Easily find and understand data assets, see how data assets relate, troubleshoot and fix problems faster, and share context with ease.
• Trust your data assets. Check data freshness, examine test coverage and results, and understand who owns what and how it should be used.
• Save money and time. Find and use existing data assets instead of recreating them. Future releases will include tips on how to optimize your project (more on this below).

"Explorer has allowed us to surface valuable metadata about tests in a consumable and discoverable way. It's important to how we think about scaling our alerting efforts and enabling a best-in-class self-service experience." – Chris Davis, senior staff data engineer at Udemy

Below are just a few workflows that dbt Explorer enables to avoid directionless discovery.

“dbt Explorer simplifies collaboration between our data engineers and data analysts and empowers our data analysts to independently access vital information, reducing their dependency on the development team. This intuitive and inclusive experience lays the foundation to further bridge the gap between data developers and consumers, and ultimately the business and its data.”



– Mahtab Rahman, senior director of architecture at Cox Auto

For data developers

dbt Explorer empowers data developers to work more efficiently and confidently by reusing existing data assets and identifying areas for improvement.

Observe: Save time by viewing available public models and how they’re used across your account.

Audit: Understand the quality of modeling and input data within your project to drive improvements.

Enhance: Find existing dbt models defined in your team’s project to improve and build new models atop them.

Diagnose: Quickly pinpoint and get to the bottom of model run or test failures or stale source freshness issues.

For data analysts

dbt Explorer opens the door to analysts to ask questions through self-serve query construction and helps them trust the data or request changes.

Orient: Find the metric you’re analyzing to understand what it means, who owns it, and how it should be used.

Validate: Check that the models used in your mission-critical dashboard are properly tested and their sources are fresh.

Ask: Find a model and columns to construct a query and answer a question using your team’s data.

Roadmap details

We’ll continue building new capabilities in dbt Explorer and the Discovery API, so you can leverage metadata to help your organization make the most of your data and keep your pipelines running smoothly.

Our vision for dbt Explorer is to enable data teams to not just understand their dbt projects, but to also have the breadcrumbs to actively improve them. Coming soon, dbt Explorer will help users optimize deployments with insights into historical model execution performance, specific recommendations for project optimization, and column-level lineage across sources and models.

Upcoming dbt Explorer features include:

Model performance: Analyze historical trends of model executions and identify opportunities to reduce infrastructure costs and save data team time.

Project recommendations: Improve your dbt project’s code, modeling, and configs given best practices to provide quality data for stakeholders.

Column-level lineage: Pinpoint data quality issues and trace impacts with a detailed view into where columns come from and how they’re used.

“It’s exciting to see the direction of mesh for cross-project lineage break team silos, reduce node boundaries for parsing speed improvements, and enable the possibility of one view for an account.”



– Alex Garbarini, solutions leader at Cisco

Getting Started

dbt Explorer is available in public preview to dbt Cloud Team and Enterprise customers. The multi-project capabilities are also available to Enterprise plan customers. To get started, simply set a production environment and click the “Explore” tab in the top navigation bar.

We’re hosting a webinar on December 5th to dive into dbt Explorer (and dbt Mesh!) in action. Sign up here.

Last modified on: Feb 23, 2024