Beyond layers and medallions: An explicit data modeling framework

Using dbt as a tool does not guarantee a good data model. A clear vision of the necessary pieces does.

This is a story of standard data layering failing our growing analytics engineering team, leading to different implementations between domains, discouraged contributors and slow progress. We had data modeling rules, yet too many use cases did not have obvious implementation paths.

The cure is not more ad hoc support. The solution is going deeper into the patterns and explaining them better. What exactly should an intermediate model contain, how it should be named and how does it relate to other int models. Which tables do we need in the mart layer, how users are expected to interact with them, which columns do they need and how do we prepare them. Where do we use automation to ensure consistency. What exactly guarantees that the mart layer remains maintainable at scale, across business lines and horizontal support teams. Come learn from our mistakes instead of making your own.