Implementing dbt in public sector data services: A comparative study of Boston and California's experiences from Coalesce 2023

"If you choose not to use dbt, then you'll probably waste time building a less fully featured, buggy implementation of it yourself."

- Jenna Jordan, Data Engineer for the City of Boston

Jenna Jordan, Ian Rose, and Laurie Merrel discuss the use of dbt in the public sector. They share their insights on the differences between public and private sector data work and present case studies of dbt implementation in their respective organizations.

dbt is a potential game-changer for public sector data work

Jenna, Ian, and Laurie discuss the potential of dbt to revolutionize data work in the public sector, given its wide range of applications, flexibility, and the possibility to expedite workflows. They highlight how dbt, an open-source project, could become an industry standard since it’s been adopted by over 30,000 companies.

"If you choose not to use dbt, then you'll probably waste time building a less fully featured, buggy implementation of it yourself," said Jenna, Data Engineer for the City of Boston. She explains that dbt provided a data catalog, making the data warehouse less of a "black box" and helping with change management.

Jenna also highlights the speed with which dbt could deliver value, citing an example of how she was able to migrate existing SQL transformations from airflow into dbt within just six weeks. "dbt is only really going to be useful for your city or state data team if you meet these three criteria..." she explains, outlining the requirement for a data warehouse, SQL as a shared language, and regular data ingestion.

Implementation of dbt requires careful planning and consideration

Implementing dbt is not a one-size-fits-all solution, and it’s necessary to tailor the approach to the specific needs and structure of an organization. Laurie, Senior Analytics Engineer at Jarvus Innovations explains how she had to redesign certain aspects of her team’s workflow to accommodate dbt, such as restructuring schemas and overhauling their data processing pipeline.

"Be flexible and meet people where they are...culture shifts take time...balance incremental adoption versus leveraging synergy with other things, " she advises. Laurie also stresses the importance of building consensus and adopting an iterative approach.

"One of the culture shifts might be the idea that iteration is good. Folks may be much more comfortable and familiar with waterfall style practices..." she says, highlighting the need for a mindset shift when adopting new tools like dbt.

dbt can enhance downstream workflows and improve data governance

Jenna shares her experiences of how dbt has enhanced their downstream workflows and improved data governance. She discusses the automation of their open data publishing, which was facilitated by dbt.

"Our engineering team has three major goals where dbt helped to solve some pain points... the biggest one: we now have a data catalog…" says Jenna. She explains how this has been instrumental in jumpstarting data governance strategy and facilitating better cooperation with other teams.

Laurie also mentions the benefits of dbt's iterative approach, explaining that it allows for the growth and addition of new features. She advises, "Don't be afraid to break the rules…Best practices are great, but make sure that you're adapting them to your context…”

Insights surfaced

• dbt is a mature, well-documented, and actively maintained open-source project that has become an industry standard–used by over 30,000 companies
• dbt can be a valuable tool for public sector data workers who have a data warehouse, use SQL as their shared language for interacting with data, and regularly ingest data into their database
• Implementing dbt can lead to a more efficient development process, improved data governance, and a better understanding of data pipelines
• The public sector often deals with siloed data and engineering decisions can be affected by data governance policies
• The implementation of dbt can be tailored to fit the governance culture of an organization, whether it is top-down or bottom-up