Blog A closer look at the newly-launched dbt Cloud CLI

A closer look at the newly-launched dbt Cloud CLI

Tristan Handy

Mar 22, 2024

Product News

Last Coalesce, in October 2023, we launched the dbt Cloud CLI in Preview. In a short period of time, it’s quickly become a critical part of the analytics engineering toolkit for hundreds of organizations.

Today, let’s talk about what the Cloud CLI can help you accomplish, and why users are loving it.

The developer experience gap today

I want to state it as clearly as I can: dbt Cloud CLI provides a superior developer experience when compared with dbt Core. Today. And that developer experience gap is going to continue to widen, fast.

It’s able to provide this better experience because it takes advantage of the stateful backend services provided by dbt Cloud, because parts of the internals have been rewritten to provide better performance, and because its footprint on your local device is much smaller and easier to manage.

Let’s dig into some of the details.

Save money and time with deferral

As a user, the biggest win that smacks me in the face as soon as I switched from Core to Cloud CLI was a feature we call “deferral.”

Imagine you’re working in a 1000-model project. You’re working on some models in the marts layer to implement a new feature. When you dig into this project, do you start by re-initializing your upstream models so that they’re current with production code? This process can take a long time and be quite cost-intensive (especially if you’re one of tens or hundreds of developers doing this at your company!)

With deferral, you can skip that step. dbt Cloud will automatically reference upstream models in production without you even having to think about it. This results in a massively better developer experience (totally eliminates a major source of wait time) and saves real costs simultaneously. You get all of this out of the box with Cloud CLI, and it’s all powered by dbt Cloud’s comprehensive metadata platform and Discovery API.

Deferral is the same underlying technology that has powered Slim CI for some time; now we’re bringing it to your development workflow too.

30% faster parse times vs. your M1-Mac

If you’re an enterprise that has made a real investment in dbt, you likely have written thousands of dbt models. Large projects are challenging to operate in for a bunch of reasons; one of these reasons is high parse times.

Every time you type dbt build and hit enter, dbt goes through a process we call “parsing.” It reads your dbt code into memory and compiles it into SQL that will run on your data platform. For medium-sized projects, this can take tens of seconds, for large projects it can take multiple minutes.

Part of the solution to this problem is to move towards a more modular project architecture using dbt Mesh (more on this below!) But part of the solution just needs to be continued investments in faster parse times. And this is where dbt Cloud CLI delivers.

Over the past few quarters we’ve rewritten some of the internals of dbt Cloud to optimize performance on exactly the types of very large projects that we see enterprises tend to build. As of today, if you’ve set your environment or job to “Keep on latest version” in dbt Cloud, parse times are roughly 30% faster on Cloud CLI vs. dbt Core. As we continue to invest in these types of optimizations, we anticipate that gap will continue to grow.

Parse times are a major contributing factor to developer experience—writing dbt code should be highly interactive, and at modest project sizes it very much is. dbt Cloud CLI is now a far better developer experience for those folks working in large projects.

Build for scale with a multi-project, dbt Mesh pattern

dbt Mesh is how we believe sophisticated data organizations should build out their at-scale dbt DAG. It focuses on distributing domain ownership to the edges of an organization while maintaining centralized governance and control over the platform.

This allows domain teams to build quickly and autonomously, without compromising governance. If two-pizza teams and microservices were a big part of the answer that has allowed software to scale to the levels of complexity it has achieved today, dbt Mesh will be a big part of that answer on the data side. And dbt Mesh is powered by dbt Cloud.

With dbt Mesh, data engineers on a Platform team (as an example) can develop via the Cloud CLI, while BI analysts on the Finance team develop via the browser-based IDE — each operating within their own dbt projects, but able to seamlessly reference and build on models from each other’s work.

Spend less time on maintenance, and onboard new developers faster

Do you support a large dbt Core install base? If you do, you know what it’s like to act as “tech support” to dozens or hundreds of users:

• “My local environment got screwed up, I’m getting an error. Can you help?”
• “I’m getting a version conflict; it says I need to upgrade. Can you help?”
• “I think I installed dbt but maybe it was in the wrong venv because it seems to be gone. Can you help?”

…etc. Often times installing, maintaining, and upgrading local software is actually harder than using it! And while dbt Core can be a great user experience, the installation and upgrade experience can be hairy, especially users who don’t go deep on Python.

With the dbt Cloud CLI, dbt version upgrades are a thing of the past—we’re doing away with minor versions for all dbt Cloud surfaces. This means you’ll always be developing with the most up-to-date version and will no longer have to choose between a) being stuck on a dbt version from 1+ years ago missing out on a ton of new features and fixes, or b) coordinating massive upgrade efforts across your team.

Not only that, the dbt Cloud CLI is a far more lightweight application—it’s essentially a wrapper over dbt Cloud APIs—which means its install process is far easier and upgrades are seamless. This install and upgrade process is already easier than with dbt Core, and it will continue to get ever-better as we continue to refine it.

dbt Cloud CLI lets you stop acting as tech support and start spending your time focusing on building data products that deliver value to the business.

Enjoy improved security

The dbt Cloud platform allows access to your business logic to be centrally managed, audited, and controlled. This reduces the risk associated with distributing sensitive information (such as credentials!) across various local environments or team members, like is often done with dbt Core. This centralized management system allows for easier rotation of credentials and ensures that access can be quickly revoked when necessary, enhancing the security posture of your data projects.

“The dbt Cloud CLI has allowed our core analytics engineering team to continue using the command-line workflows they’re comfortable with. For them, the dbt Cloud CLI has made the transition from dbt Core seamless. Their workflows look and feel the same, even though the mechanics have changed behind the scenes.”
— Katie Claiborne, Staff Analytics Engineer, Cityblock Health

Our roadmap for dbt Cloud’s development experience

The introduction of the dbt Cloud CLI marked a pivotal moment, representing—for the first time ever—a second way to develop natively within dbt Cloud.

Some users love writing code in a browser. There is nothing to set up, no ecosystem of tooling to learn about and deal with, just log in and go.

Other users love developing locally—they love the power and control that the local dev tooling environment gives them.

There is no wrong answer here—these are just different users expressing their preferences. There are great software development environments in browsers and on the desktop.

Our goal is to bring the power of dbt Cloud to your dbt development workflow regardless of which you prefer. And in the future, we expect there will be additional development modalities! More info on this down the road, but you should expect to see a more graphical way to write dbt code as well as a natively-integrated VSCode experience, all powered by dbt Cloud, in the future.

Our belief is that there is no wrong way to write code, as long as it goes through mature governance practices (CI/CD, pull request reviews, linting, etc.). So the more options we give people to write dbt code, the more humans will be able to participate in creating and disseminating knowledge at their organizations.

What’s next for the Cloud CLI?

As a user, I already strongly prefer using Cloud CLI for all the many reasons I shared above. I’ve wanted this experience for years now and I’m never going back. But things only get better from here. Over the coming months, here’s what’s on the way:

• Invoke production jobs from the Cloud CLI. Soon, you will be able to run production jobs on demand from the Cloud CLI in addition to the currently supported development jobs. This will be a very powerful and flexible way to invoke dbt Cloud as a part of your orchestration workloads.
• VS Code extension support. Get all of the syntax highlighting, linting, and other convenience that you would expect from an IDE language extension.
• Richer integration with dbt Explorer. Visually interact with your development branch in dbt Explorer. See what you’ve built and easily demo it to others.
• Further improvements to parse performance. Expect large projects to continue to get snappier.

Look out for more announcements over the course of this year related to the dbt Cloud CLI. If you’re already on dbt Cloud, you can get started using the Cloud CLI today. If you’re not using dbt Cloud yet, the Cloud CLI is available to every dbt user for free, forever, on the dbt Cloud Developer tier. Get started today.

I’m confident that you, like me, will never want to go back.

Last modified on: Apr 04, 2024