Revolutionizing the staging layer: automating dbt generation for 700+ sources

Migrating a large enterprise data warehouse from SQL Server to dbt meant confronting an uncomfortable truth early: manually building staging models, source YAMLs, and SQL files for hundreds of sources wasn't going to work. To make the migration feasible, we built an automation framework that generates dbt staging assets at scale across 700+ enterprise sources. 

We'll walk through how we automated source definitions, model SQL, YAML configuration, and documentation, plus how we used that framework to embed standards and governance into the migration itself, not bolt them on afterward. The result was a path through a large legacy warehouse that was faster and more consistent than we expected, and one other teams can learn from.