Showing our work: a controlled experiment in context engineering

dbt's internal library offers an ever-expanding array of AI skills that users in every branch of the org rely on for their daily work. Almost all of these skills, however, pull context straight from vendor tools like Gong; over time, this led to increasing API constraints and ever-higher token spend.

Earlier posts in this series debuted a new pattern for modeling context in the warehouse, demonstrated through compressing Gong transcripts with warehouse-native AI functions and then serving the compressed form to agents through the dbt MCP server. This pattern helped us shrink data volume by 20x and slash token consumption on Gong transcript analysis from tens of thousands of tokens to a few hundred.

This pattern originated from our belief that engineering context happens in the same way analytics engineers engineer data, and now it’s time to show our work. We designed a controlled experiment that followed the classic scientific method many of us learned in middle school science classes: observe the problem, ask a question, do research, create a hypothesis, run an experiment, form a conclusion.

The test: 1 skill, 4 ways of gathering the same customer context, 80 runs, and 3 measurements (cost, speed, and output quality), all on real customer data.

The headline result: serving an engineered summary through dbt MCP cut cost 48.5% and run time 43.2%, and kept most of the dates, numbers, and next steps the full transcript carried. It missed decisions and job titles, and we’ll show you where. This post is our lab report.

The problem

Teams across Fivetran + dbt Labs rely on AI skills: packaged instructions that tell an agent how to complete a specific task, such as building a recap PDF from a customer's sales calls. Each skill calls the vendor's MCP connector (like Gong, Salesforce, or Slack), pulling context directly from the source. Different users using different skills all the time, all across the org, was exactly the plan — but it ultimately caused some unfortunate side effects:

• Shared API caps. Vendor connectors enforce org-wide rate limits. Every skill and every user draws on the same allowance, so heavy demand from one team’s workflow eats into everyone else's.
• MCP tool parameter limitations. A vendor's MCP exposes only the filters the vendor chose to build; Gong's MCP, for example, has no parameter for finding calls by account name. Agents work around that gap through token-heavy work like cross-referencing other systems for a match or bulk-reading all calls on a certain date.
• Duplicated retrieval logic. Each skill author independently specifies how to find and fetch the same kinds of context. The logic gets rewritten slightly differently in every skill, with no shared standard behind it.

These factors were driving our token spend higher every month. The skills themselves weren’t the problem. It was that each was separately requesting the same types of context (a rep's recent calls, an account's open tickets, a thread's email history), then handing the raw data to the LLM to synthesize from scratch. A single Gong call transcript, for example, can run past 70,000 characters, and every skill pulling up that transcript was reading all of it, every time.

The question

What if we did the digesting once, upstream in the warehouse? Every skill can then pull the already-curated version through dbt's own MCP using the same "build it once, reuse it everywhere" pattern that dbt originally pioneered for data transformation.

The research

Call transcripts, support tickets, and email bodies have been sitting in our warehouse for years; we just had no practical way to reason over that unstructured data at scale. Parsing it required rigid, rule-based structures which routinely missed or mis-measured things like sentiment, tone, and other contextual signals that don't parse well into fixed logic. So the data mostly sat there, until two things happened.

First, warehouses now ship AI functions that run inside SQL. We dug into that toolkit in our recent Context engineering is already possible in your warehouse post. SQL is great at querying and transforming, and LLMs are great at reasoning. Putting them in the same layer allows the warehouse to reason about the context it already holds. Second, the dbt MCP server lets agents query that context directly: the transcripts, documents, and ticket and email bodies that sit alongside the semantic layer metrics, not only the metrics themselves.

Ad hoc, per-skill retrieval logic with no shared standard is exactly what dbt was originally built to solve. Earlier posts in this series showed how much compressing Gong data in the warehouse can save per read. This time we wanted a rigorous, end-to-end answer to a narrower question: compared to letting each skill go straight to the vendor, what does it cost to curate and retrieve our own engineered context, using our own warehouse and our own MCP?

Our Internal Analytics Snowflake project already ingests Gong data and models this data up to supply Gong’s full transcripts and briefs, so we had a ready-made test bed. It does not context engineer the data (pre-gather important context from the full transcripts) for LLMs, but we can still use this model to compare using Gong’s MCP against querying dbt MCP to accomplish the same tasks. Measuring the benefits required evaluating variants against across three metrics against the control (Gong’s MCP baseline that we’d already established):

• The token cost / spend of the workflow
• The quality of the LLM’s output
• The speed at which the output can be produced

The hypothesis

We set up 3 hypotheses, each changing one thing in sequence. The first swaps where the context comes from, the second swaps raw transcripts for a summary, and the third swaps who writes that summary:

Hypothesis 1: If we change Gong MCP to use dbt MCP, the cost/spend, quality, and speed is expected to be roughly the same as calling Gong MCP.

The idea we’re exploring in our first hypothesis is that switching the source shouldn't change much. If a skill pulls full transcripts from our warehouse through dbt MCP instead of from Gong's MCP, cost, quality, and speed should come out about the same. Both the control and the variant read the same transcripts with about the same number of tool calls.

Hypothesis 2: If we change dbt MCP to retrieve a Gong brief, ingested directly from Gong into our warehouse, the cost and speed is expected to go down dramatically, but we’ll also experience a drop in quality.

The theory behind this particular hypothesis is that a vendor’s internal AI summary already condenses the context to important points — the selection of which we have no control over. This engineered context will be shorter than the full transcript and therefore faster and cheaper to retrieve. However, since we can’t control the definition of a “good” summary tailored to our specific needs, we expect we will be missing some context that’s important for our business.

Hypothesis 3: If we change then the instructions to retrieve data from a model which has read the full transcripts and engineered narrative summaries for us, the cost, speed, and quality will be balanced.

By engineering summaries using our own rules, we're reducing the amount of context dramatically, which will cut costs and increase speed. The definitions responsible for generating these summaries are within our control, which allows us to fine-tune the summary for information relevant to our business use cases. The resulting context should be quality comparable to that of full transcript context.

Test design

In order to keep the tests controlled we chose a single skill, sales-pdf-builder, for its relative simplicity. This skill calls Gong’s MCP to retrieve call transcripts and then generates a branded recap pdf based on whatever was requested in the prompt.

The Gong MCP tool call in this skill became our isolated variable: because every variant used the same skill and the same prompt, the only difference was the context-gathering step (i.e., where the context came from). Having this as the single change kept the tests tightly scoped against confounded results.

The controlled prompt

This prompt was the exact prompt used to trigger the skill for every one of the 80 sessions completed:

We tested four context-gathering paths:

The control

This is the skill's original approach, using Gong's MCP to find and read the transcripts. The retrieval mechanism for calling the Gong MCP stayed the same. To ensure that each run’s processes are idempotent, the original skill was tightened up to make sure that the LLM’s inherent creative freedom and reasoning didn’t confound results.

Variant A: dbt MCP on full transcript column

Same skill, but the instructions point the agent at our warehouse instead. It queries the conversation_transcript column in our fct_gong_call_transcripts model through dbt MCP's execute_sql tool. The skill instructions we used:

Construct a query to get the transcript with filters based on the user’s prompt. Use `analytics.analytics.fct_gong_call_transcripts` as the base table and retrieve `conversation_transcript` for the full transcript. Run the query using `execute_sql`. Example query: 
SELECT call_id, title, scheduled_at, conversation_transcript
FROM analytics.analytics.fct_gong_call_transcripts
-- WHERE clause on account_name, scheduled_at, internal_participants,
-- external_participants, or title

fig 1: An example of the conversation_transcript column in dbt Labs’ fct_gong_call_transcripts model, along with the number of characters in each transcript.

Variant B: dbt MCP, Gong's brief

Same query, but it retrieves the brief column instead of the full transcript (conversation_transcript, the full text transcript in fct_gong_call_transcripts ). Gong generates the brief itself, and we ingest it into the warehouse unchanged. Briefs usually run about 400–600 characters. Example skill instructions:

Construct a query to get the transcript summary with filters based
on the
user’s prompt. Use `analytics.analytics.fct_gong_call_transcripts`
as the
base table and retrieve `brief` for the transcript summary. Run th
e query
using `execute_sql`. Example:
```
SELECT call_id, title, scheduled_at, brief
FROM analytics.analytics.fct_gong_call_transcript
-- WHERE clause on account_name, scheduled_at, internal_participan
ts, external_participants, or title
```

fig 2: An example of fct_gong_call_transcripts’ brief column, which is ingested from Gong and pulled straight from the Gong source, along with the number of characters in each brief.

fig 2: An example of fct_gong_call_transcripts’ brief column, which is ingested from Gong and pulled straight from the Gong source, along with the number of characters in each brief.

Variant C: dbt MCP, our own engineered summary

This variant reads from a model we built specifically for the test. It runs each full transcript through Snowflake's AI_COMPLETE() with a prompt that sets our own extraction rules. The model is lightly engineered to use only AI_COMPLETE(), with none of the chunking, embedding, or classification techniques that could improve it further. The prompt asks for a factual, evidence-based narrative with no length limit, and sets a few hard rules:

• Only include what was actually said, with no inference or editorializing
• State every date, count, and figure explicitly ("May 4," not "soon"; "45," not "several")
• Name every tool, platform, and competitor specifically
• Call out every decision and next step as such, rather than folding it into the narrative
• When the transcript is ambiguous or contradicts itself, report both versions and who said each

Example skill instructions:

Construct a query to get the call summary with filters based on the user’s prompt. Use `analytics_dev.dbt_cberger.sales_pdf_builder_te sting__cleaned_calls_v2` as the base table and retrieve `call_summary` for the transcript summary. Run the query using `execute_sql`. Example: 

``` 
SELECT call_id, call_title, date(started_at), call_summary FROM analytics.dbt_cberger.sales_pdf_builder_testing__cleaned_call s_v2 
-- WHERE clause on account_name, date(started_at), internal_partic ipants, external_participants, or call_title 
``` 

fig 3: sales_pdf_builder_testing__cleaned_calls_v2’s call_summary, showing the content generated from the AI_COMPLETE() function, along with the character lengths of the engineered summaries.

Variant C’s resulting summaries are much longer than Gong's briefs (roughly 12,000–20,000 characters for our test calls) because the goal was to retain all relevant details that matter.

Experiment parameters

Here's how much context each variant gave the agent for the same two calls:

We ran every variant 20 times, 80 sessions in all, each in its own contained session. Every run used the exact same prompt:

“Create a branded PDF recapping two [redacted] calls: Jess Williams' pre-service call with [redacted] on December 4, 2025, and Christine Berger's RAA Kickoff call on January 8, 2026 — covering what was discussed pre-service and at the start of the engagement”

For each run, we tracked token cost and run time, and scored the output on six categories of specific detail it should have captured: dates, numbers, people's titles, decisions, next steps, and tooling.

The results

With all 80 runs complete, we had three things to compare for every variant: what it cost, how long it took, and how much useful detail made it into the recap.

Cost and speed

The control and variant A were the costliest and slowest variants by a wide margin. Reading full transcripts is expensive no matter where they come from. Variant A’s results are ahead of the control; the retrieval from Gong's MCP has no way to find calls by account name, so the LLM needed to parse through every call on a given day to make a match. If it couldn’t find it, it needed to reach out to other MCPs for clues it could use in its Gong MCP filters. In our warehouse, on the other hand, the account name is just a column — a standardized way the LLM can filter to retrieve the right transcripts using a constructed WHERE clause.

Variant B was the cheapest and fastest, cutting cost and run time by more than half over the control. Variant C came in close behind. In order to understand how they fared overall, we need to analyze the output quality.

Note: If you've read the earlier posts in this series, those savings might seem small next to the 98% token reduction we reported for compressed transcripts but they’re two separate things: The token reduction figure compares the size of the context itself, a raw transcript versus its summary. In this experiment we are measuring the whole skill run, including instructions, tool calls, and generating the PDF output; these cost the same no matter where the context came from. C's summaries were also deliberately long, because we wanted them to keep the details.

Quality

Cost and speed only matter when the output itself is actually useful. To measure this, we scored every run on whether it captured six kinds of specific detail from the calls. The skill's instructions asked for specifics when available but didn't demand them, so we didn't expect every run to hit every category. The scores show how often each variant surfaced the metric details across its 20 runs.

*B named only one tool in every run, out of five discussed on the calls.

• The control and dbt MCP (A) variants both produced a good ratio of hits because they had access to the full transcript text.
• The variant B Gong-supplied Brief showed a drastic difference in quality. In all 20 runs, it didn’t provide any specific dates, numbers, job title, decision, or next step (except for dates that were explicitly supplied in the prompt, not in the context). It did mention a single tool across all runs, out of a total of five tools discussed during the call. Variant B works if you only want an overall sense of how a call went; it falls short if you need to plan around a deadline, size the work, or need deeper context about the customer’s priorities.
• Variant C’s curated context performed within a few runs of the full-transcript variants on dates and numbers, matched them on tooling, and was the best of all four variants at naming the next steps. It fell short on two categories: titles (9/20, compared to A's 19/20) and decisions, where it scored 0/20.

The gap between variant B and variant C is easiest to see in the quality of the recaps themselves. For example, here's how each variant described the next step from the kickoff call:

B: Next step: the team will start with a deep dive into the customer's setup.

C: Next session: the customer's director of BI walks through their Salesforce workflow, and their BI lead covers a product-usage workflow that spans multiple Redshift clusters.

While both are technically accurate, only variant C gives you detailed and actionable information for preparing for the next customer call.

The conclusion

Back to our three hypotheses.

Hypothesis 1: If we change Gong MCP to use dbt MCP, the cost/spend, quality, and speed is expected to be roughly the same as calling Gong MCP.

Result: Mostly true. Hypothesis 1 tests that switching the source shouldn't change much. Full transcripts from our warehouse performed about the same as full transcripts from Gong on quality. But variant A’s dbt MCP approach was about 14% cheaper and faster, entirely because our model let the agent filter calls by account instead of hunting for them. Same data, better access.

Hypothesis 2: If we change dbt MCP to retrieve a Gong brief, ingested directly from Gong into our warehouse, the cost and speed is expected to go down dramatically, but we’ll also experience a drop in quality.

Result: True. Gong's brief cut cost and run time by more than half. It also missed every date, number, title, decision, and next step in all 20 runs, and named only one of the five tools discussed. For someone who only needs the gist of a call, this does the job. Anyone who needs to plan around a deadline, understand who's who on the customer side, or spot an opportunity, though, needs more context to work from.

Hypothesis 3: If we change then the instructions to retrieve data from a model which has read the full transcripts and engineered narrative summaries for us, the cost, speed, and quality will be balanced.

Result: Largely true. Variant C cut cost nearly in half and cut run time by more than 40% relative to the control. It also retained most of the detail variant B dropped. It was the only variant that balanced all three measures. It wasn't a clean sweep: variant C's summaries came up short on titles and missed some decisions entirely, which tells us our extraction rules need another pass. But fixing this simply requires editing our own dbt model (instead of petitioning a vendor for a new feature).

This is the true finding behind the numbers: Once context lives in the warehouse, every skill draws from the same modeled, versioned, tested source instead of each one rebuilding its own path to Gong. The reads hit Snowflake instead of Gong's API limits, the retrieval logic lives in one place, and — once we improve the summary — every skill that uses it gets better at once.

Our experiment puts numbers behind the dbt philosophy fundamental to every post in this series: context is data, and it should be engineered like data:

• Ingest (Fivetran, cSDK)
• Add vector indices to the key unstructured columns
• Model with dbt
• Annotate with a semantic layer (Looker, Omni, dbt semantic layer)
• Connect your AI tool to the Fivetran context service (ChatGPT, Claude)

This lab report shows this new pattern at work using real-world data, and demonstrates that even a lightly engineered model was able to cut our costs nearly in half while keeping most of the detail our teams rely on — and every improvement we make to it from here pays off for every skill in our library.

Try it yourself. The dbt_context_engineering package gives you the building blocks for this pattern, starting with chunking for the call transcripts, emails, and other unstructured text already in your warehouse. Read dbt_context_engineering: modeling the context your AI agents read to get started.