Context engineering is already possible in your warehouse. Here's how to get started.

Analytics engineers already know how to build enterprise data architecture: take raw data, clean it, model it, and serve it in a trusted form. For years the destination was a dashboard and the input was structured tables. Now, teams are moving from dashboards to AI as their primary interface, it’s time for context engineering.

The largest data user has quickly become AI agents and the context they need to deliver business results extends beyond structured data to include transcripts, tickets, PDFs, and emails. We now have technology that can read large bodies of text, then understand, interpret and deliver a response faster than it took someone to get up to grab a coffee.

These context engineering techniques come with some new vocabulary and technical implementation details, like vectors and ontologies. But they also benefit from the same engineering patterns teams already rely on from dbt: transform data to build logic into models, enrich data by joining it to related information from other systems, describe what the different fields mean and how to reliably query them, and govern who can see what information.

The stack stays the same too: Fivetran moves data in and dbt models it. The part most people haven’t tapped into fully is that warehouse providers have been heavily investing in handling the unstructured data that’s essential for context engineering.

Context engineering dramatically increases the data team’s potential for impact. The analytics engineer used to mainly serve users who could write SQL or build charts. The context engineer serves every person, through their AI agents, across the entire organization. This post teaches some quick-start terms and techniques that allow analytics engineers to unleash their inner context engineer.

Join our session, From Analytics Engineer to Context Engineer: A dbt Playbook, where we’ll walk through the exact patterns dbt Labs uses internally to turn unstructured sources into governed, versioned context, and introduce the dbt_context_engineering package that puts those patterns into practice. Save your seat here.

Your data warehouse is already ready for context engineering. Now it’s your turn.

When AI started to become a thing in 2023, the big data warehouses like Databricks, BigQuery, and Snowflake all started investing to become AI infrastructure. They began adding useful AI-native capabilities: the ability to call AI directly inside a SQL query, and support for embeddings and vector cosine similarity search as a SQL function.

Now, all the major data warehouses include these native AI functions for their users as part of the system. For example, Snowflake Cortex offers AI_EMBED, AI_COMPLETE, and AI_PARSE_DOCUMENT; Databricks’ task-specific functions include ai_extract (structured fields from a schema you define), ai_classify, and ai_summarize. No matter what the warehouse calls them, dbt inherits the functionality because it’s deployed as a service to run those systems.

So how do you context engineer? You just use the AI tools the warehouses provide! You don’t need to invest in new providers and you don’t need to build agent infrastructure in separate products like a graph database, a vector database, or a separate ML stack. You can call functions and thoughtfully architect your warehouse with the tooling it already exposes. The problem is, most people haven’t thought about developing these patterns.

While a handful of people have been able to transition to running AI on the frontier through trying out what the warehouses added in, the majority of data practitioners haven’t realized the magnitude of this moment — and their potential to capture it. We think this is mainly due to lacking some basic vocabulary and clear examples.

The first missing link: vocabulary

When the big data warehouses announce things like, ‘We now have vector stores!’ a lot of analytics engineers, who have traditionally built for analytics use-cases with metrics and dimensions, may not understand the full potential these warehouse AI capabilities are offering. These functions extend the possibility of what they can deliver to their organization.

Built-in warehouse functions are important because they are the path for analytics engineers becoming the key driver of AI for your business. They are incredibly powerful building blocks for designing AI systems, callable from any SQL model. Unfortunately, the data warehouses did not do a great job educating and enabling users on how and why they should learn these functions. Here are the concepts you need to know so you too can implement the craft.

Starter concepts that turn raw data into modeled context

1. Chunking: making a large body of text addressable by splitting it into pieces (ie, chunks of text) small enough to embed and search over. Or, alternatively, combining many small pieces of text into larger, more useful and meaningful sets.

There is not a single chunking methodology that is universally agreed on, but the general principle is to assemble text into meaningful units so that when they are embedded (the next vocabulary term), they are going to represent a valuable piece of the search space for agents. If they are too large, the meaning can get messy, or if they are too small, they can become meaningless noise. A simple, naive chunking method would be to set a minimum amount of text for a chunk and either split or combine text to have enough to be meaningful. Typically, chunks usually overlap with each other so that the search space isn’t awkwardly cut off in the middle of a chunk and meaning is lost.

1. Embedding: turning text into vectors to allow for searching.

Embedding is how you take the chunks and create the vector search space for the meaning of text, so that agents can use semantic search to find the most relevant context. An embedding model reads a chunk and returns its coordinates in a vector space, which is a highly dimensional representation that is a list of numbers representing relative distance from one of a thousand or more dimensions. Embeddings don’t replace source data; they sit alongside the source data as a lookup key for the meaning, which is why retrieval pipelines always store the chunk text next to the vector. Also, it’s a one-way process: you can’t recover text from a vector.

1. Vectors: an array of numbers that positions a piece of text in space, so that closeness in that space means closeness in meaning.

Vectors are, functionally, coordinates for meaning. A single vector is a list of numbers that places a specific piece of text in space relative to every other piece from that same source. The other vectors closest to it are the ones that are most related. Example: a query about "can't log in" will rank highly in similarity to a ticket that says "password reset loop," even though they share no words. A vector doesn't make any sense to a human being because it's literally just a bunch of numbers. But to an LLM it's a search space.

1. Classification (Ontology): attaching meaning to each chunk with a shared vocabulary.

In computer science, ontologies are structured frameworks that define concepts, attributes, and relationships within a specific domain to help machines understand data. In context engineering, classification uses AI to say, ‘Okay, what does this chunk of text mean and how does it relate to the context of my broader organization?’

There are many more techniques that we’ll be publishing, and we'd love to hear from the community how you are approaching context engineering in your own work. Join in the conversation on the dbt community: https://www.getdbt.com/community/join-the-community.

The second missing link: What this looks like in a DAG

Vectors, chunking, embedding, and other AI data-modeling techniques are fairly new concepts, but they don’t require a new stack. They all work within the stack you already know: source data lands, then a staging layer, then foundational modeling into marts. Every dbt user has built this a thousand times.

Now, AI functions, embeddings, and vector search have arrived as SQL primitives. Anything that’s available in a SQL function, including AI tools, fits inside a dbt model, and it also inherits everything dbt already does: version control, lineage, testing, incremental runs, scheduling. You don't need a vector database or an orchestration layer for embeddings. This all happens without materially changing your data stack.

The new layer: chunk, classify, embed

Taking sales call data as an example: Chunking turns a long single-row call transcript into many rows per call. Embedding adds a vector column so that you can search through a large corpus of calls. And classification is able to relate the information to the broader context of your organization. By the end you have a table where every row is a passage, described and searchable, giving you a search space so you can return only the most relevant pieces of information.

For example: Looking at a large body of customer calls and asking, ‘How often are users talking about login issues across our customer base?’ Chunking will have turned those calls into the meaningful units for search; embedding makes a search space so that if you want to search over top of the large corpus of calls, you are able to find just the chunks that matter. Then, the similarity of “login issues” can be searched within the vector store, within the warehouse.

Context engineering patterns for enriching the calls with additional data from across the business are already well built within dbt. Meaning, this context will already also have other related pieces of information that you also probably want, like which call did it happen on, who was the customer, what exactly was said, and who said it.

See our Model for the token, not the table post for details about how we applied vectors, chunking, and embedding in the dbt warehouse to mine our own library of Gong transcripts, shrinking data volume by 20x and slashing token consumption on a 60-minute Gong call from tens of thousands to just a few hundred.

The byproduct of us thinking about context engineering as a data modeling problem is that we are able to make better context structures for AI, starting with creating vector columns as the search space for whatever question you are asking. The result is higher quality context and smaller token counts.

A philosophy, not a feature

The pattern is a transformation pipeline that ends in retrievable context, going beyond what’s in a dashboard: source → staging → marts → chunk, classify, embed → agent. Context gets staged, transformed, tested, and versioned the same way any other data product does.

What makes this a philosophy is that nothing in this requires new tooling for dbt users. Same warehouse, same dbt project, same modeling discipline. The pipeline doesn't change, it extends: chunk, classify, embed and more all sit where the modeling used to hand off, and it runs on native warehouse AI functions called from SQL models.

The dbt Gong example is only one instance of this pattern, which applies to any durable, token-heavy source like support tickets, contract PDFs, email archives, Slack logs, or marketing collateral. The source changes; the techniques stay the same.

The vocabulary, the practices, and the shared patterns for context engineering are still being built. Running these techniques against real data surfaces a second set of questions: tracing an output back to the prompt that produced it, bounding inference spend as the corpus grows, proving that the context an agent used traces back to source data.

Once that context is modeled and annotated with a semantic layer, an AI tool can query it the same way a person would, with meaning, not just keywords. The full pattern looks like this: ingest → index the unstructured columns → model with dbt → annotate with a semantic layer → connect your AI tool to that context.

That last step is where Claude and ChatGPT come in. This handoff is what the Fivetran Context Layer (currently in limited public preview) is built for, so the modeled, governed context you've just built in your warehouse becomes something an AI tool can reason over right away, rather than a separate integration project. Stay tuned for more.

Join our session, From Analytics Engineer to Context Engineer: A dbt Playbook, where we make the case that context engineering is analytics engineering with a new last mile. We'll walk through the exact patterns dbt Labs uses internally to turn unstructured sources into governed, versioned context, and introduce the dbt_context_engineering package that puts those patterns into practice. Save your seat here.