How Data Culture measured its way from zero trust to a shippable AI analytics agent

At a dbt Summit 2026 breakout session titled "Don't ship what you can't measure: Evaluation-driven development for AI analytics agents," Opeyemi Fabiyi, director of analytics engineering at Data Culture, laid out a three-step framework for measuring whether an AI analytics agent is trustworthy before it reaches a stakeholder.

Fabiyi borrowed the term evaluation-driven development from Chip Huyen's book AI Engineering, and distilled it into a single rule his team now follows for every agentic analytics rollout: don't ship what you can't measure.

Data Culture is a boutique data and AI consultancy that spent its first five years on classic modern data stack work, including data modeling, BI migrations, and platform migration. Over the past year, client demand shifted toward operationalizing workflows with AI and letting stakeholders chat with their data.

That shift pushed the team to build its own methodology, evaluation framework, and playbook for deploying agentic analytics, which it has since used on two production clients: a healthcare provider building clinical operations analytics, and an e-commerce retailer building product and sales analytics.

The trigger for the framework was a question Data Culture couldn't answer during its first client deployment: what is the accuracy of this agent? The team had connected an agent to a Snowflake MCP server for a group of test users, but had no number to offer, not even a rough one.

Months later, dbt Labs' 2026 State of Analytics Engineering report surfaced the same problem industry-wide. Trust remained the top concern for teams deploying AI agents to stakeholders, with 71 percent of respondents worried about hallucinated or incorrect data reaching decision-makers and 83 percent naming trust a top organizational priority.

Fabiyi said the real bottleneck had shifted from building an analytics agent to proving that agent's output could be trusted before a stakeholder relied on it.

When Data Culture built a first lightweight evaluation pipeline and reviewed the results by eye, it found a pattern: several "failures" were not real errors, but cosmetic mismatches a human analyst would have called correct.

In one case, the agent returned the exact right numbers for monthly gross revenue, just under a differently named column. In another, it returned the correct three product categories plus a fourth row of legitimate uncategorized products. In a third, it returned the right numbers plus one additional supporting column.

A naive exact-match comparison marked all three as wrong, even though the underlying answer was correct in every case.

To fix that, Fabiyi's team built evaluation-driven development around three steps: define what "correct" means, define how to compare results, and build layered evaluation logic.

The first step is writing "golden questions" specific enough that two analysts on the same team would write the same SQL to answer them, then validating the expected answer with a domain expert before using it as a benchmark.

The second step compares the results of a query rather than the query itself, since two different SQL statements can produce an identical answer. Data Culture built a four-tier comparison cascade: exact match, then a value-only comparison that ignores column names, then an overlapping-column comparison that tolerates extra context columns, and finally an approximate-numeric match that tolerates rounding differences.

The third step splits evaluation into two layers. A deterministic layer handles roughly 80 percent of golden questions with fast, free, reproducible comparisons, and a large language model acts as a judge for the remaining 20 percent, scoring responses on groundedness, relevance to the question, and comprehensiveness.

With that framework in place, Data Culture could run controlled experiments to see which kind of added context actually improved agent accuracy, using the same golden questions and the same model at every step.

On the e-commerce deployment, accuracy climbed from 0 percent with no semantic context, to 50 percent once the agent was pointed at modeled dbt tables instead of raw source data, to 52.6 percent after adding column descriptions, to 78.9 percent after adding business rules and default filters, and to 94.7 percent after adding verified queries and named metrics.

The healthcare deployment followed a similar curve, starting at 0 percent on raw data and eventually clearing Data Culture's internal 90 percent production threshold once the same layers of business rules and verified queries were added.

Across both deployments, the model stayed the same. The accuracy gains came entirely from added context layers: modeled tables, column descriptions, business rules, and verified queries, each one measured against the same evaluation suite.

Fabiyi pointed to similar patterns reported elsewhere in the industry, citing Anthropic's published account of taking an agent from 21 percent to 95 percent accuracy after building a curated data foundation and an offline validation step.

He closed by arguing that a dbt project's modeled tables and semantic context are the durable asset in any agentic analytics deployment, while the agent or LLM layer underneath can be swapped out later. Data Culture open sourced its evaluation pipeline, golden question templates, comparison logic, and a technical walkthrough at github.com/datacult/dbt-agent-trust.

Watch every keynote and breakout from dbt Summit 2026 on demand. Watch dbt Summit 2026 on demand →