Practical guidance for initiating continuous integration in dbt Cloud: Enhancing data workflows at scale from Coalesce 2023

"dbt Cloud has made it easier to implement CI with smart defaults and new features."

Joel Labes, Senior Developer Experience Advocate at dbt Labs, explains the concept of continuous integration (CI) and why it's important in dbt projects. He also discusses the history of CI, how it works, and its benefits.

CI enables efficiency and mitigates risks in dbt projects

CI is crucial in dbt projects to ensure efficiency and mitigate risks. It allows developers to integrate their changes back into the main branch and identifies errors early in the development process.

Joel states, "CI is good for you. It makes sure that you don't release dumb changes by mistake. It's not niche anymore. If you're not using it, the time to start is now." He also discusses the evolution of CI and how it has become easier to implement in dbt Cloud, making it accessible to all developers.

Slim CI facilitates efficient and effective resource use

Joel explains the importance of Slim CI, a new method of doing CI that dbt Labs introduced in 2021. Slim CI enables developers to build only the models that have changed since the last production run, promoting efficient use of resources.

"The only thing you need to check [are] the things that have changed in your PR. And so, I would say that Slim CI is the right way to do all of your CI," states Joel. He further elaborates that the full build of every single node in the project is unnecessary and Slim CI is the preferred method to adopt.

dbt Project Evaluator package aids in maintaining project hygiene

Joel also highlights the dbt Project Evaluator package during the webinar. It was developed by the dbt Labs' Professional Services team and aids developers in maintaining good project hygiene by warning them of potential issues such as missing tests on primary keys, undocumented models, and direct queries on sources.

"... a lot of the logic and the warnings and recommendations are going to be made available to you from inside of dbt Explorer," Joel explains. He also advocates for the possibility of enforcing these recommendations programmatically through CI.

dbt Cloud simplifies the CI experience

Joel explains the recent changes in dbt Cloud that simplifies the CI experience for developers. dbt Cloud now offers a new CI job type that sets smart defaults for developers and allows CI runs for different PRs concurrently.

"dbt Cloud will kick off a run, and we'll see that it has succeeded, and then we can merge that code quite happily, and we can feel confident that everything that needed to happen has happened," says Joel. He emphasizes that these updates have made implementing CI in dbt Cloud easier than ever.

Unit testing is a significant, upcoming feature in dbt

Joel hints at the introduction of unit testing in the upcoming 1.8 version of dbt. This new feature, which is being eagerly awaited by the dbt community, is expected to be a major innovation in the tool.

"Unit testing is coming to dbt…That's where the next round of innovation is coming," shares Joel. He urged the developers to actively participate in discussions and contribute to this new feature.

Insights surfaced

• CI is important to prevent the release of erroneous changes and is no longer niche. Every dbt project should use it
• Slim CI is a more efficient form of CI, building only the models that have changed in your project since your last production run. It is recommended over full CI
• dbt Cloud has made it easier to implement CI with smart defaults and new features like the ability to run CI runs for different PRs concurrently and automatic cancellation of stale code
• The dbt Project Evaluator package can help maintain good hygiene for your project by warning you about potential issues. These include a lack of tests on your primary keys or undocumented models
• Unit testing is coming to dbt in version 1.8 and will provide an additional layer of code validation