The data practitioner for the AI era

The following is an excerpt from MIT Technology Review's "The Data Practitioner for the AI Era," a report co-sponsored by dbt Labs and Databricks. Read the full report here.

In today’s rapidly evolving business landscape, the integration of artificial intelligence (AI) into operations is no longer a luxury but an outright necessity. Companies worldwide are increasingly adopting AI initiatives to drive innovation, improve decision-making, enhance customer experiences, and deliver a competitive edge.

But for all its promise and investment, AI adoption is fraught with challenges, particularly in managing and leveraging the vast amounts of data that fuel these initiatives. The enthusiasm for incorporating AI into business operations is often dampened by the complexity of managing the underlying data infrastructure. A common challenge is the integration of large language models (LLMs) with cloud data platforms.

Without high-quality, continuously updated data connected to clear semantic definitions, businesses risk exposing inaccurate or outdated information to their LLM applications. Further, strong data governance practices are required to ensure that LLMs operate on data appropriately and that sensitive or regulated data is not misused or misconstrued by LLMs.

Ultimately, LLMs are great at internalizing mountains of data and spitting out answers; if we prompt them with rich, high-quality inputs, we can expect to see high-quality outputs. Likewise, if we prompt with low-quality or non-compliant inputs, they may jeopardize business integrity and erode customer trust by outputting incorrect, invalid, or otherwise inappropriate outputs.

dbt Cloud is a powerful ally in this context, providing a suite of tools designed to maintain the integrity and trustworthiness of data powering AI initiatives.

Semantic layer implementation

The creation of a semantic layer, which maps data to business concepts, ensures that the data exposed to AI models is accurate, relevant, and consistent. A semantic layer can significantly reduce the risk of hallucinations and inaccuracies.

Metadata framework

A built-in metadata framework enables data to be enriched with a wealth of context and meaning, and it magnifies AI’s ability to yield reliable answers to critical business questions.

Data quality with contracts

Data contracts enforce clear definitions of data quality, structure, and relationships across teams. This ensures that only compliant data feeds into AI projects, even if that data crosses team boundaries, data stores, or domains.

Version control and testing

With built-in version control and testing capabilities, teams can track changes, test data models rigorously, and ensure that the data infrastructure remains stable and reliable.

Alerting and continuous integration

Real-time alerting mechanisms to flag data quality issues, paired with continuous integration processes to catch issues before they hit production, ensure that data quality or model performance problems are promptly identified and addressed, preventing potential setbacks in AI initiatives.

Accelerating AI projects

AI initiatives can only move as fast as the development of the data that underlies them. dbt Cloud fortifies the data foundation of AI projects and accelerates development and deployment by streamlining data transformation and modeling processes.

At dbt Labs, our mission is to empower data practitioners to safely create and disseminate organizational knowledge. These data practitioners will shape how AI is deployed in the enterprise and drive the strategy that leads to higher-quality results and designing data workflows that reduce risks in AI implementations. For them to be successful, we believe practitioners must adopt a structured and reliable approach to data management.

By ensuring data integrity, fostering trust, and facilitating rapid development, dbt Cloud empowers businesses and data practitioners to leverage AI with confidence on top of their cloud data platforms.