Improving data modeling for effective growth marketing: Insights from Deputy's data team from Coalesce 2023

"After we did build the confidence, it actually allowed us to work a lot faster and our pace increased."

Huss Afzal, Data Director of Deputy, explains how Deputy, a workforce management software company, integrated dbt into their data engineering processes. Huss discusses the company's challenges before dbt, how they built confidence in their data, and the rolling out of a tool called ThoughtSpot.

dbt significantly improved Deputy's data management processes and boosted team morale

Before the implementation of dbt, Deputy was struggling with an overdependence on Snowflake tasks, complicated security measures, and a lack of adaptability. This led to a lot of rework, ad hoc queries, and a high risk of regression bugs. Huss described the team as being overworked, burnt out, and underappreciated.

"We had discussions with domain experts…we had clearly metric owners…we introduced data governance meetings…we were really open about the communication of data-related issues and how we were going to fix it," Huss explains. These strategies were instrumental in building confidence in their data.

As a result of implementing dbt, Deputy was able to work at a faster pace, assess potential Upstream issues, complete reiterations on their datasets much faster, and stabilize their foundational elements of data. This enabled them to achieve timely decision-making analytics and reporting.

Deputy introduced ThoughtSpot to open up dbt to the rest of the company

Despite having a robust and reliable data model, Deputy encountered the challenge of making it accessible to the rest of the company. The company adopted a tool called ThoughtSpot to address this issue. ThoughtSpot allowed the rest of the company to extract their own insights and visualize data in a way they saw fit.

"What thought spot did for us was opening up dbt to the rest of the company, so letting them get their own insights, letting them visualize the way they see fit, and essentially maximizing the effectiveness of our data model," Huss details.

They also revealed that the company had written an integration script that automatically converted the YAML file out of dbt into ThoughtSpot's native language. This made it a seamless and automated process for column descriptions and metrics to be updated in ThoughtSpot whenever they were defined within dbt.

Python integration in dbt offered Deputy more flexibility and freedom

Huss highlights the importance of Python integration in dbt, which allowed Deputy to create custom logic and implement advanced analytics, especially in the areas of machine learning and AI. They were also able to reuse complex logic across multiple dbt models thanks to the usage of macros.

"Being a SaaS, it's sometimes necessary for us to create custom logic...we can mix and match SQL and Python and write more complex logic for our models," he explains.

Huss also shares their plans for the future, which include customer reporting and data as a service. He’s excited about offering customers the opportunity to build their own dashboards and reports. That way, they can use their extensive data to provide insights to the world in an aggregated way.

Insights surfaced

• Before the integration of dbt, Deputy faced several challenges including a lack of adaptability, over-complicated security measures, and data integrity issues
• Deputy built confidence in their data by having discussions with domain experts, introducing data governance meetings, and being open about the communication of data-related issues
• Deputy increased their pace of work due to lineage and dbt. The company was able to assess potential upstream issues and stabilize the foundational elements of its data
• The roll-out of ThoughtSpot allowed the company to open up dbt to the rest of the company, allowing users to get their own insights and visualize data as they see fit
• Deputy integrated Python into dbt, allowing them to create custom logic and implement advanced analytics
• Deputy plans to grow into the next phase, exploring different avenues including customer reporting and data as a service