dbt without the warehouse (or the bill): DuckDB end to end
DuckDB co-founder Hannes Mühleisen runs a full dbt build against 10 gigabytes of Wikipedia data with his laptop's Wi-Fi switched off, live onstage.
Overview
Challenges the assumption that transforming a few gigabytes requires a rented distributed system. Shows DuckDB, a free in-process database, running full dbt projects on a laptop with sub-second iteration and CI on standard GitHub runners, plus DuckLake for open-format publishing and DuckDB-WASM for browser querying.
What you'll take away
- 01
dbt v2 ships with a built-in DuckDB adapter, so connecting dbt to a fully local, offline warehouse takes only a small profile change, no authentication or credit card required.
- 02
Citing Fivetran's own analysis, the median data warehouse query scans about 100 megabytes and the 99.9th percentile scans about 300 gigabytes, sizes that fit comfortably on a single laptop.
- 03
In a live demo, a dbt build ran fully offline against 10 gigabytes of compressed Wikipedia page-view data, including a group-by-all aggregation over roughly 300 million rows, finishing in about three seconds with zero warehouse compute cost.
Session speaker
Hannes Mühleisen
DuckLabs