dbt

Driving strategic decisions with dbt: The finance data story at Snowflake from Coalesce 2023

Team members from Snowflake, Sandra Herchen and Jack Peele, share the story of finance data at Snowflake and explain how the company uses dbt.

"Ever since we migrated to dbt, we've experienced development velocity like never before."

- Sandra Herchen, Analytics Engineering Manager at Snowflake

Team members from Snowflake, Sandra Herchen, Analytics Engineering Manager, and Jack Peele, Business Intelligence Analyst, share the story of finance data at Snowflake and how the company uses dbt to drive quick and strategic decision-making. They discuss the evolution of dbt at Snowflake, the structure of their finance data team, and how dbt has helped them overcome challenges.

Snowflake's success with rapid and strategic decision-making using dbt and Snowflake

Snowflake's finance data team discusses how they utilize dbt and Snowflake's database to drive quick and strategic decision-making. The team is comprised of three distinct sections; the analytics engineers, the BI analysts, and the data scientists. Each group plays a specific role in maintaining a robust financial data system.

Jack states, "We're going to talk about how dbt really helps us develop quickly and cost-effectively." He elaborates, "So, my role as the BI Analyst and support of this team is to build and maintain all the data products that go into making the most informed decision possible...these actions derive in a deep understanding of the Snowflake product and are backed by timely, accurate, and reliable data models."

The team shared that dbt has significantly impacted three main areas: development velocity, enabling real-time data-driven decisions, and governance. Jack explains, "dbt has been incredibly impactful for our team in three main ways...By bringing software engineering best practices to data, we've gained incredible oversight over our BI development, and this really helps our team iterate quickly."

Snowflake's shift from Excel to Looker to dbt for data management

Jack and Sandra share that their data management system transitioned from Excel to Looker to dbt as the company scaled and their data volume increased. dbt was chosen for its performance within Snowflake and its ability to use tables in various ways.

Jack explains, "In the early days of Snowflake, we really tried to leverage the product internally as much as possible... but as Snowflake continued to grow and scale, we received more and more cost data from AWS, Azure, GCP…that Excel really couldn't handle this volume... That's when we started looking for the next best thing, and that's how we ended up on dbt."

Jack adds, "dbt is incredibly performant inside of Snowflake... ever since we made the switch back in 2019, it's enabled us to do so much more than we initially even expected."

dbt's role in enhancing data governance and cost optimization

"dbt incremental models led to significant improvements from both a cost and performance perspective."

- Jack Peele, Business Intelligence Analyst at Snowflake

The finance data team successfully used dbt to reduce their cloud spend and improve their data governance. Their focus in 2023 is to leverage dbt's features to make their data operations even more cost-efficient.

Jack states, "2023 is really the year of cost optimization, and fortunately, dbt offers a number of tools that really help us be the most cost-efficient data team possible." Sandra adds, "Our next goal is to explore the relationship between cluster keys and incremental filters... We're also excited to leverage Snowpark to facilitate new machine learning workloads and continue to champion data-driven decision-making alongside our stakeholders."

Jack and Sandra’s key Insights

  • The finance data team at Snowflake is made up of analytics engineers, BI analysts, and data scientists–with each having distinct roles
  • dbt has been instrumental in increasing development velocity, enabling real-time data-driven decisions, and improving governance.
  • The team faced challenges in selecting the best incremental filter and defining a unique key that aligns with their business logic
  • By converting their tables to incremental models, they managed to reduce the data volume significantly, leading to cost savings and improved performance
  • The team plans to leverage more dbt features to reduce their cloud spend further