MuleSoft's use of dbt in overcoming data transformation challenges within their modern data analytics framework from Coalesce 2023

“From my personal experience with working with dbt…it’s super powerful…I can finally work on the data as a software engineer, and it's so easy to develop with SQL SELECT statements…”

- Yijun Cao, Data Platform Engineer at Salesforce

Yijun Cao, Data Platform Engineer at Salesforce, and Dakota Kelley, Solution Architect for phData discuss Salesforce's use of dbt for their modern data annex framework. They cover their challenges, how they integrated dbt with their framework, and the benefits they experienced from implementation.

Decentralizing the data team structure improves efficiency and encourages collaboration

Salesforce transitioned from a centralized data team to a decentralized structure, which improved efficiency and encouraged more collaborative work. The centralized structure had posed challenges, as team members were only experts in some business areas. They had limited domain knowledge, leading to delays in data delivery. Decentralizing the team allowed for more focused work on specific business areas.

"Each domain team was relying on us to get the data they need from a single data warehouse," says Yijun. "We found this centralized data team structure brought some challenges and issues to us." She explains that they didn't want to be the "bottleneck" anymore and wanted to build a reusable and self-service data platform that could easily be shipped to domain teams.

"We want to build a reusable and self-service data platform that can easily be shipped over to the domain team so that they can use our capabilities and functionalities easily, and they can focus more on producing all the data products, and they will be the owner of the data products," Yijun emphasizes.

Integrating dbt with other tools can streamline the data pipeline

Integrating dbt with other tools was key in streamlining the data pipeline for Salesforce. This integration helped the company address challenges with collaboration, data lineage, and efficiency in producing data. Dakota explains how they integrated dbt with the existing framework to enhance technical details and benefits.

"So with us handling ingestion with Matillion, a lot of the processes that were built within this team was to couple transformations with ingestion," Dakota explains. He also highlights the importance of integrating dbt with cloud IDEs, mentioning the creation of a Python connector that would call dbt Cloud, making it easier for business users to work within their familiar environments.

Implementing automated testing and version control increases productivity

Implementing automated testing and version control systems increased productivity for the Salesforce data team. By using dbt, developers were able to work on the data as software engineers, making collaboration easier and resolving merge conflicts simpler.

"We focused on trying to make [the delivery of data products] as automated as possible so the team didn't need to do anything," Dakota says, explaining the benefits of continuous integration and continuous deployment (CI/CD). He also mentions how version control made it easier to work on the same data model and resolve merge conflicts.

Yijun points out the benefits of automatic data lineage, saying, "I don't need to manually maintain the data mappings anymore within our database. The data lineage automatically pops up within dbt."

Yijun’s key insights

• dbt can be integrated with other tools to streamline data pipelines
• dbt's advanced macros can be leveraged for customized solutions
• dbt's data governance features can be used for data lineage and integration with data catalog tools
• Implementing continuous integration/continuous deployment (CI/CD) with dbt can greatly improve the delivery of data products
• dbt can improve runtime efficiency and developer productivity