How NASDAQ built a governed intelligence layer with dbt and Databricks

The stakes in financial services data are different from almost any other industry. In most business cases, the cost of a data error comes down to wasted time or a slightly wrong metric. In financial markets, a data break doesn't surface as an error message. It surfaces as a wrong regulatory filing, an incorrect client bill, or a risk desk making calls from numbers that were already stale.

Jamie Nemeroff leads value engineering at dbt Labs, where he quantifies what good data infrastructure is worth. In financial services, that calculation is easier to make than elsewhere because the cost of getting it wrong is so traceable. It's also what makes the story of what NASDAQ has built worth examining closely.

Michael Weiss, AVP of Product at NASDAQ, leads NASDAQ Eclipse Intelligence, an end-to-end data platform built on dbt and Databricks that NASDAQ originally developed for its own markets and now delivers to its financial market infrastructure (FMI) customers: the exchanges, clearinghouses, and central securities depositories (CSDs) that operate on NASDAQ's Eclipse product suite. Jamie spoke recently with Michael and Andrea DeSosa, who leads go-to-market for capital markets at Databricks, to discuss what they built, how they built it, and what it makes possible.

Four themes came up that Jamie tracks across every large financial services engagement he works on: scale, removing engineering bottlenecks, regulatory governance, and AI readiness. NASDAQ's story connects all four. And it's increasingly a product story: the architecture they spent a decade proving internally is now available to their customers in six months.

NASDAQ Eclipse Intelligence: architecture and origins

Most people know NASDAQ as a market operator. A less-visible part of the company is its financial technology arm, which delivers infrastructure software to financial services firms globally, covering anti-financial crime, regulatory technology, and capital markets operations. NASDAQ's Eclipse product suite focuses specifically on FMI customers.

Michael described the intelligence platform as a data layer spanning the full lifecycle: ingestion from financial market protocols (NASDAQ-provided and third-party), transformation and mapping, validation and reconciliation, and business applications for reporting, analytics, and billing. Databricks serves as the primary computation layer.

dbt plays two roles: transformation engine and semantic layer. On top sit three customer-facing products: InsightsHQ for visual dashboards, ReportHQ for structured report outputs like CSVs and PDFs, and RevenueHQ for managing billing and fees.

The platform didn't arrive fully formed. NASDAQ has been on this path since 2012, starting with a regulatory data product for US broker-dealers. In 2014, the company moved all of its market data from on-premises warehouses to AWS. The intelligence platform was formally launched in 2021, extended to external FMI customers in 2024, and is now in active delivery to its first customers.

What's notable about that timeline is the direction of proof. NASDAQ didn't build a product and then go find buyers. They built it for themselves, ran it at scale for years, and then recognized that the FMI customers they spoke with had the same problems they had already worked through.

Scale at a trillion messages per day

The volume NASDAQ manages helps explain why every architecture choice at this level carries weight.

Michael described ingestion of anywhere from 750 billion to a trillion messages per day across NASDAQ's US and Nordic businesses, from 5,000 to 7,000 data sources, across 26 internal business lines. The regulatory consequences of errors are immediate.

US programs like the Consolidated Audit Trail (CAT) run on roughly a three-day window for getting data out. Billing has to reconcile to the exact contract.

As Michael put it: "When we go to generate the bill at the end of the month, we can't have an unaccounted for or an extra contract in the bill we're sending. We need to make sure that information is right, both in terms of the input and the output."

The approach NASDAQ uses to maintain consistency at this scale is what Michael described as a dbt Mesh structure: a family of dbt projects organized from foundational models upward. The foundational layer defines contracts for raw data. Every message on an order chain looks the same, regardless of whether it comes from a NASDAQ market, a Southeast Asian exchange, or a non-NASDAQ trading platform. That consistent contract makes it possible to define metrics and KPIs once and deploy them everywhere.

It also shapes how the team responds to errors. "Our stance," Michael said, "is we'd rather have the pipeline break with an error on the test and be late on sending data than send wrong data."

Andrea framed scale from the Databricks side as a trust problem as much as a volume problem. "Every time a team builds its own pipeline and defines its own metrics," she said, "you've created a future audit finding, a future reconciliation break, or a future AI model trained on the wrong truth."

Databricks' Unity Catalog addresses this at the infrastructure level. The same definition of settlement finality or net position applies across all of NASDAQ's internal teams and FMI customers. That’s not because people agreed to use the same spreadsheet, but because the platform enforces it.

Getting data to the people who need it

Engineering bottlenecks are the second theme Jamie consistently work through in financial services business cases. The data exists. The business teams need it. But access runs through ticket queues, and the lag compounds.

Michael described what changed at NASDAQ when they put dbt in front of non-engineering teams. Because SQL is broadly understood across the business, governed modeling tools could go directly to analysts and business users. The result was a 30 to 40 percent reduction in time to market for new data products and insights.

The example he gave was concrete. NASDAQ's options sales team, working alongside the options business team that owned the underlying dbt models, built client-specific visuals to distribute as part of their sales process. No engineering involvement required.

NASDAQ is now looking to offer the same model to its FMI customers: access to NASDAQ's foundational dbt models alongside governed tooling that lets customers' business teams build on top. "We're looking to let them take our foundational models and rebuild things from a business point of view that complement or supplement the models they need to make their business go."

Andrea described why this approach compounds rather than complicates. "The biggest tax on delivery isn't talent," she said. "It's starting from scratch every time." A consistent foundation changes the unit of work: teams inherit a proven architecture and configure it to their markets. The people at FMIs who understand the business best, the clearing workflow specialists and surveillance officers, can start contributing to model development rather than waiting on engineering queues.

The condition that makes this safe is governance. As Andrea said: "You can only safely put those tools in non-engineering hands when you have a governance layer enforcing the boundaries underneath."

Unity Catalog handles that at the data level. dbt handles it at the model level. Together, they let more people build without fragmenting the foundation.

Governance that holds up to regulators

"Governance" can become abstract fast. In financial services, it's specific.

Michael walked through two concrete examples:

• Rule 17A for broker-dealer compliance in the US establishes a write-once, read-many (WORM) obligation. Firms must be able to prove that data wasn't manipulated after the fact.
• SOC 2 compliance for billing means being able to show an auditor the full path from source data through every transformation to the fee applied and the invoice sent. "I can't go to a customer and say, here's a non-SOC 2 compliant billing solution," Michael said. "Every other billing solution on the planet is SOC 2 compliant."

When compliance gaps appear, the cost runs beyond fines. There are regulatory filings that have to be amended, legal overhead, and time spent explaining and correcting.

For NASDAQ's FMI customers, these same obligations apply. That means the platform NASDAQ delivers has to make defensible answers available on demand.

Andrea described Databricks' role here as infrastructure-level auditability. Unity Catalog tracks every transformation, model, and dashboard: where the data came from, who touched it, when it changed, and what depends on it, automatically, without anyone needing to document it manually. "When an auditor or regulator asks," she said, "you have the answer."

dbt adds the logic layer. Michael described pulling a data lineage snapshot from dbt to show a regulator the flow of a model from source to output. If there's a question about a specific calculation, the answer is in the documentation or the code itself.

"dbt makes that part pretty effective and pretty easy," he said. It also helps customers building on top of NASDAQ's foundational models understand what those models mean and how they're calculated before making any modifications.

Trusted data as the path to AI

The AI conversation in financial services carries a particular weight. As Andrea framed it: "In financial markets, a confident but wrong answer is a risk event. It's not just an inconvenience."

If a generative AI model hallucinates a product recommendation in a consumer app, a user gets a bad suggestion. If it hallucinates a position, a margin requirement, or a regulatory classification at a clearinghouse, that's a compliance breach.

This is the gap the semantic layer is built to close. Michael described the root cause: AI gets things wrong not because models are incapable but because they lack context about the data they're working with.

The dbt Semantic Layer, which Michael called NASDAQ's "context layer," provides structured meaning alongside the data. When an agent accesses a metric, it knows what the metric means and how to apply it because the definition is explicit and governed, not inferred.

Dbt Labs recently published a study on text-to-SQL accuracy, comparing results with and without a semantic layer in place. The difference was stark: close to or at 100% accuracy when using Semantic Layer with both ChatGPT and Claude. Without it, agents querying raw tables produce confident answers that can be wrong in ways that are difficult to detect.

NASDAQ is now building what Michael called an "agentic surface layer," investing in Model Context Protocol (MCP) tooling and skills that let customers build their own agentic workflows on top of the intelligence platform. The longer-term vision is a marketplace where NASDAQ, its partners, and its customers can share those workflows in a governed, verifiable way.

"How do you do everything in a controlled environment such that you always know the result is guaranteed?" Michael asked. The foundation they've built is designed to answer that question, whatever the use case.

When Jamie works through a financial services business case, the four themes he tracks—scale, productivity, governance, AI readiness—keep pointing back to the same thing: the work that satisfies a regulator today is the same work that makes AI trustworthy tomorrow. Lineage, contracts, semantic definitions, etc., aren't AI features. They're data infrastructure decisions. But they're the infrastructure decisions that determine whether AI agents can be trusted with the calculations financial markets depend on.

Michael put it directly near the end of our conversation: "Delaying these initiatives is just becoming a bigger issue. There's a lot of focus on just trying to get there as quickly as possible."

For NASDAQ's FMI customers, the option now exists to skip the three-year build and get there in six months. The combination of dbt and Databricks underneath the intelligence platform is a significant part of why that acceleration is real.

Watch the full recording to hear more of what Michael and Andrea covered. And if you're working toward the same kind of trusted data foundation, talk to our team.