Automating the impossible: migrating 40,000+ objects to dbt in 9 months

When we faced migrating 40,000+ WhereScape-generated procedures to dbt, the math was brutal: manual work would take years. We had 9 months. Automation wasn't a nice-to-have; it was the only path forward.

We built a system that handled environment setup with production-scoped parallels, SQL conversion to dbt-ready code, dependency mapping, and full testing with results published to Snowflake dashboards. The result: over 40 well-managed data products delivered on time.

But automation was only half the problem. Keeping data quality intact during an active migration was harder. Nondeterministic models, duplicate data, and standing up proper CI/CD while the business kept running—these weren't edge cases, they were the job. 

We'll share what we learned: when to automate and when manual work is non-negotiable, how to scale testing, and how to manage the organizational change that comes with a migration this size.