dbt Cloud for distributed domain-driven development in a data mesh environment from Coalesce 2023

"With this transformation, moving to this new stack…we were able to start considering some big changes."

Holly Burch, Data Architect at Sharp HealthCare, explains how Sharp Health is leveraging dbt for distributed domain-driven development. She discusses the business problems that led them to this approach, their vision for change, the implementation process, and their current state of domain development practice.

Decentralizing the analytics approach to manage resources better

Sharp HealthCare was facing a challenge with its centralized enterprise analytics team being limited by resources. They aimed to switch up their workflow by decentralizing and allowing more domain-driven development. "We had a team of nine developers, three architects, and about eight to ten business analysts on our team, supporting preparing data for analytics across the entire enterprise at Sharp HealthCare. We were limited by team size and also by tool licensing costs," says Holly.

Holly explains that the domains (or service lines) at Sharp, including clinical, revenue cycle, marketing, and foundation, among others, were all supported by the centralized enterprise analytics team. She elaborates that the goal was to involve these domain groups in the software development lifecycle (SDLC), giving them access to the analytics tools and thereby changing the analytics development in the organization.

Talking about the benefits of domain-driven development, Holly says, "The domain groups really know their data best. They know the KPIs. They know what data might be missing, what data they're looking for…they take that knowledge then into the BI tool."

Implementing a unified security model across the analytics tools

To manage the decentralization of the analytics team effectively, Sharp Healthcare implemented a unified security model across all their analytics tools. This was driven by Active Directory (AD) group membership, which helped control licensing costs and ensure that the right data was accessible to the right teams.

Holly explains, "We have one cloud project per domain, and in dbt Cloud, the developer licenses are driven by AD group membership. We have one Mart schema per domain, so we have marketing schemas, clinical schemas, HR schemas set up…the security allows those domain developers to work within their sandbox schemas."

She also mentions the importance of data governance in managing this change, stating that they enforce peer reviews before merging into the main and master branch. This approach ensures they’re following the same SQL standards and guidelines and meeting the contract when creating data products.

Training and onboarding domain developers

To build a successful domain-driven development environment, Sharp HealthCare identified potential domain developers by having them pass a simple SQL test. Once identified, they were then trained on Snowflake, dbt, and Looker, with a focus on their specific domain's data.

Holly explains, "We start training on Snowflake. We give them a personal schema that they can use in the dev environment. We start with a domain-focused group, and it's usually one to six people at a time. The reason why I think it works is because we're using a dataset of their own. It's data that they're interested in."

Holly notes that the velocity of development was impressive. She explains, "The outputs that are coming out are really exciting. The velocity of development…it does take a while to get the domain developers there and working, and it's just a pleasure to see them really driving forward and making progress in moving quickly to produce outputs that are important to the organization."

Insights surfaced

• Sharp HealthCare faced resource limitations with their centralized enterprise analytics team, which led them to consider domain-distributed development
• The company decided to begin major platform migrations to the cloud
• They set up one dbt Cloud project per domain from the start, which proved beneficial
• They identified potential domain developers through a simple SQL test and then trained them on Snowflake and dbt
• The company is now in a state where they have fully franchised teams that are working independently across the data stack