Overcoming hurdles in tech debt and complex data infrastructure migration at Unbounce from Coalesce 2023

"We made the decision to adopt dbt at the same time. This is actually a short list of the reasons we really like dbt. That list sort of keeps growing."

Morgan Cabot, Analytics Engineering Technical Lead at Unbounce, shares her experience overhauling tech debt and migrating to a modern data stack. She covers the challenges Unbounce faced during the transition, the process of migration, and the benefits Unbounce gained from the process.

Modernizing the data stack can significantly improve efficiency and reduce costs

Morgan highlights the journey of Unbounce, a landing page-building company, in modernizing their data stack. Despite the process being time-consuming and demanding, Morgan stresses that the move was justified by the significant improvements in efficiency and cost reduction.

Morgan mentions, "We wanted to move from an ETL approach to ELT...we have all of our transformations happening in the same place, and we want to then push the data from that single source of truth out to wherever we need to use it." The migration process involved examining and cleaning up old code, reducing data warehouses from six to one, cutting down Python scripts from 450 to 70 dbt models, and reducing job run times by 86%.

She also points out the surprising cost savings, stating, "Our infrastructure costs went way down... we just assumed. We pitched, ‘This is going to be so much faster. It's going to be so much easier.’ And like okay, what's that going to cost us? And we were really surprised that it actually was costing us way less than what we were paying for our pain spaghetti." The cost savings amounted to a reduction of 60%.

The switch to a unified data warehouse improved data accessibility and insight

The transition to a unified data warehouse significantly enhanced data accessibility and insight across the company. By moving to AWS Redshift and implementing dbt, the company consolidated its data, which made it much easier to answer business-related queries.

"...maybe we should have thought harder about which data warehouse to use, but it works for us, and we're really happy with it," Morgan shares. She highlights that the switch made the query process faster and simplified the code. This ultimately reduced the turnaround time for business queries from weeks to days.

Morgan also points out how the unified data warehouse improved self-service, saying, "Nothing really beats the feeling of being in a leadership presentation, and they're showing us all these business-impacting metrics, and they actually self-serve those." This meant business queries could be answered without the constant need for engineering support, making the system more efficient.

The process of modernizing the data stack demands patience, resources, and cross-team collaboration

Despite the clear benefits, modernizing the data stack was not without challenges. Morgan emphasizes the need for patience, dedication of resources, and cross-team collaboration throughout the process.

"There were moments of like, ‘Why isn't this done yet?’" she admits. She also mentions the time-consuming process of pitching the migration, proving its value, coordinating with various teams, and handling the complexity of the migration alongside other business priorities.

Morgan notes the importance of stakeholder communication during the migration, sharing, "We have a whole data updates channel... we would make sure to tag the right stakeholders... really making sure everybody was comfortable with these changes." This ensured there was minimal disruption to ongoing business processes and a smooth transition to the new data stack.

Morgan's key insights

• Unbounce decided to migrate to a modern data stack due to the limitations of their old data stack, which was built in the early 2010s
• The migration process was time-consuming and required coordination across multiple teams
• The migration resulted in significant improvements in job run times, a reduction in code complexity, faster turnaround times, and cost savings
• The migration also reduced the dependency on engineers to answer business questions
• Unbounce is now capable of supporting its data needs and is looking to scale its data infrastructure