ClickHouse + dbt: Building a GitOps workflow at Twilio
Twilio replaced three distributed systems, Kafka, Spark, and Elasticsearch, with one ClickHouse pipeline governed end to end by dbt and GitOps.
Overview
Twilio's Voice Insights team migrated from legacy streaming infrastructure to ClickHouse, then needed to manage schema changes safely across teams. Covers integrating dbt with ClickHouse for a GitOps workflow: schemas as code, isolated preview environments, and CI/CD validation before production.
What you'll take away
- 01
Twilio consolidated three separate distributed systems, Kafka, Spark Streaming, and Elasticsearch, into a single ClickHouse-backed pipeline governed by dbt and the open-source dbt-clickhouse adapter.
- 02
Because ClickHouse is open source, engineers could spin up real test containers for CI instead of relying on mocks, and each developer can deploy a personal preview environment against a slice of production data.
- 03
The result: three systems became one, custom Spark applications became SQL models and materialized views, and manual production changes became version-controlled and audited through dbt.
Session speakers
Pavan Kotikalapudi
Twilio Inc.Marek Radonsky
Twilio Inc.