Unlocking model governance and multi-project deployments with dbt-meshify from Coalesce 2023

“dbt Meshify is a brand new, command-line tool to help you add model governance and cross-project lineage features to your dbt project...”

- Grace Goheen, product manager at dbt Labs

Team members at dbt Labs, Grace Goheen, product manager, and Dave Connors, senior DX advocate, join Nicholas Yager, principal analytics engineer at HubSpot, to discuss dbt Meshify, a tool that helps to manage and organize large dbt projects. They describe a fictional data team facing issues such as the undefined ownership of models, unclear interfaces between teams, and an overload of models slowing down their development. The team uses dbt Meshify to add model governance and cross-project lineage features to their dbt project.

dbt Meshify provides solutions to common problems in developing large dbt projects with multiple contributors

The participants discuss common challenges in dbt projects, including undefined ownership of models, unclear interfaces between teams, and an overwhelming number of models that can slow down development. This was followed by the introduction of dbt Meshify, a new command-line tool designed to tackle these issues by adding model governance and cross-project lineage features to their dbt project.

"dbt Meshify is ready to be pip installed anywhere that you run Python,” says Grace. As part of the demonstration, the team shows how you can dbt Meshify to add a model contract to a project, protecting your dbt projects from unexpected changes. "A model contract is a set of upfront guidelines that define the shape of your model," adds Grace.

dbt Meshify enables the addition of model versions to manage changes and expectations

Another significant feature of dbt Meshify is the ability to add versions to models. This allows developers to define new logic for their dbt model and have two outputs consumers can choose from. This provides a way to upgrade logic without disrupting anyone's work and gives consumers time to adapt to the new logic.

"Model versions allow you to define brand new logic for your dbt model and produce two outputs that consumers can choose between," Grace explains. "It's a pathway to upgrade your logic without breaking anyone's work and gives consumers time to migrate to the new logic as they're ready."

dbt Meshify can group models together and set their access levels automatically

dbt Meshify also comes with a feature to group models together. It can then set the access levels of these models automatically. This feature allows for easier understanding of the models in the project and better management of the relationships between them.

The participants demonstrate this feature by using the dbt Meshify group command to create a new group in their project, set the owner of the group, and select which models belong to the group. The tool automatically sets the access levels of the models according to their interfaces within the group.

"dbt Meshify understands how your models are being used and then sets the access levels accordingly," Grace explains. "For models in your project that are only going to be referenced by other models in the same group, their access is going to be private. But for models that are referenced by models outside of your group, or maybe leaf models of your project, those are going to be set to protected."

dbt Meshify can split a monolithic project into multiple, cleaner projects

The participants also demonstrate how dbt Meshify can be used to split a large, monolithic project into multiple, cleaner projects. The tool allows for a more manageable interface between contributors, reduces manifest size and parse times, and enables cross-project lineage without duplicating nodes.

Using the dbt Meshify split command, the participants were able to partition their dbt project into two separate projects. This resulted in a more organized and manageable structure, making it easier for the team to work on their respective parts of the project without stepping on each other's toes.

"For more documentation, check out the dbt Meshify GitHub page," Grace adds.

Grace, Dave, and Nicholas' key insights

• dbt Meshify is a command-line tool that helps add model governance and cross-project lineage features to a dbt project
• The tool helps solve problems such as undefined ownership of models, unclear interfaces between teams, and an overload of models slowing down development
• dbt Meshify can automate the creation of configurations, making it easier to manage large dbt projects
• The tool can also help split a monolithic project into two smaller projects, making it easier to manage and reducing parse times
• dbt Meshify can be installed anywhere Python is run