Seamless scalability, effortless upgrades: the enhanced dbt Cloud platform
Feb 01, 2024
Product News
Over the past few months, dbt Cloud has grown into a more complete solution for businesses of all sizes. At Coalesce 2023, we unveiled some substantial new dbt Cloud features, which you can read about in our blog post. We added features like native support for multi-project architecture — what we’re calling dbt Mesh — along with an updated dbt Semantic Layer, the brand-new dbt Explorer, a dbt Cloud CLI, and more.
But we also understand that before your team can start using dbt Cloud, you need to be sure about one key question: “Is this a service we can build our business around?” In other words, is it reliable enough, secure enough, battle-tested enough?
We’ve long invested in making dbt Cloud a best-in-class platform, with robust SLA commitments and support for SSO, role-based access controls, fine-grained git permissions, audit logging, and multiple deployment regions. As a result, dbt Cloud is now trusted by more than 4,100 customers — including Anheuser-Busch Group, Boston Consulting Group, Fanduel, Fifth Third Bank, Roche, Siemens, Under Armour, and Urban Outfitters.
Today, we’re sharing several more enhancements that mark another step in dbt Cloud’s maturity as the enterprise standard for modern data transformation.
Here’s what’s coming your way:
- A new Cell-Based Architecture for improved scalability and reliability
- No more dbt version upgrades: get new features and fixes immediately, with no interruptions
- dbt Cloud jobs are now resilient to Git provider outages
- Elevated security with account-scoped Personal Access Tokens
Introducing our new Cell-Based Architecture
As our customers' dbt Cloud workloads grow, we remain dedicated to meeting these increasing demands with an architecture designed for efficiency and scale.
Our new Cell-Based Architecture will be the foundation of dbt Cloud for the indefinite future. And the best news is: it’s already in production today. A small group of customers, ranging from our free Developer tier to our business-critical Enterprise tier, are currently experiencing its benefits:
- Improved scalability, such that dbt Cloud is maximally performant regardless of the complexity of a customer’s deployment, or the number of customers in a given region.
- Improved reliability, ensuring we can continuously deliver new product innovation in dbt Cloud with even less risk to product stability than before.
Our engineering teams have been working on this project for some time, from initial design to prototyping, then validation, and finally, deploying this completely redesigned dbt Cloud architecture into production over the past few months.
dbt Cloud's Cell-Based Architecture has been deployed worldwide, featuring multiple redundant, fully-equipped regions. Each region is geographically isolated and contains its own set of cells, ensuring high performance and reliability across the globe.
Imagine your organization is located in Dublin, Ireland. With our updated architecture, you have the option to host your dbt Cloud account in either the US1 region (the default), or in the EU1 region, should your organization's compliance and regulatory requirements dictate. Regardless of your choice, your organization will benefit from a consistent, fault-tolerant dbt Cloud instance. As more dbt Cloud customers opt for your selected region, new cells will be added to accommodate them, operating independently from the cell hosting your account.
This type of scalable architecture is a hallmark of mature software infrastructure, and we’re excited to offer it to our customers. This also means that, in addition to delivering other product innovation more quickly, dbt Cloud can also expand into new deployment regions more easily in the future to support data residency needs in more locales — something we plan to do over time.
What does this mean for you?
Over the course of 2024, we'll gradually onboard new customers and migrate existing ones onto cells.
If you are a dbt Cloud user, there's no immediate action needed on your part. We will reach out to you and your account administrators with detailed information about specific migration steps, if any are necessary, to facilitate the migration of your account.
The migration process will be highly automated, and there will be no service disruption. We aim for this to be a seamless and unnoticeable process to the extent possible.
If you’d like to proactively schedule your account migration, please reach out to support.
We’re eagerly anticipating the future this creates for dbt Cloud, the thousands of organizations that rely on it, and for what we’ll be able to build on top of this platform.
The new Cell-Based Architecture will be available to all dbt Cloud accounts this year. A migration check-list is available today.
Never worry about dbt version upgrades again
One of the main goals of dbt Cloud is to abstract away the core maintenance work required to run dbt, by providing a fully managed SaaS solution built on top of it — and thus freeing up your team’s time and energy.
However, we recognize that coordinating dbt version upgrades in dbt Cloud can often feel tedious and disruptive. We know some customers have opted to stay on older versions due to this inconvenience and missed out on the latest features and fixes.
Going forward, we’re committing to making dbt Cloud feel and function like the other SaaS apps your team uses. Just like you never have to worry about which version of Databricks, Salesforce, or Snowflake you’re using, coordinating dbt versions upgrades should never be something you have to spend valuable cycles on.
Today we're introducing the option to automatically keep your dbt Cloud environments and jobs on the latest version of dbt. This is rolling out to some customers in beta now, and then will be generally available by April 2024, ahead of the planned release of dbt Core v1.8.
You’ll soon see this as an option in your version selection dropdown. Once you select this option for all your environments and jobs, you’ll be off the version treadmill for good: you’ll never have to upgrade dbt versions again.
Our commitment to you: the dbt Labs team will ensure that behind-the-scenes upgrades are a seamless experience, with zero breaking changes. We are building the platform for analytics engineering, and that means your analytics code keeps working.
Once we demonstrate the value and stability of this experience for our customers, we plan to ultimately remove the option to select a dbt version altogether. We will be doing this slowly, over many months, and we’ll remain open to your feedback throughout that process.
Managed dbt version upgrades will be available to all dbt Cloud accounts in April of this year.
dbt Cloud jobs are now resilient to Git provider outages
Sometimes the things that cause jobs to fail are external to dbt. Those can include Git provider outages or a third-party package dependency.
To mitigate the impact of these scenarios, dbt Cloud can now cache your project's Git repository, ensuring that your jobs continue to run seamlessly, even if external services are temporarily down. If there's an outage that causes the cloning operation to fail, dbt Cloud will use the cached copy of the repo so your jobs can continue running as if nothing happened.
dbt Cloud customers experienced this in action during GitHub's brief recent outage on January 9th. Despite GitHub’s unavailability, 29 scheduled dbt Cloud runs continued to execute that would previously have simply failed. This ensured that hundreds of downstream data products could continue to be built and relied on, helping dbt users maintain trust with stakeholders.
GitHub's brief January 9th 2024 outage.
Git repository caching is available to dbt Cloud Enterprise accounts today.
Elevating security with account-scoped Personal Access Tokens
Today, we're introducing account-scoped Personal Access Tokens, a significant enhancement to the security posture of dbt Cloud.
Our new Account-scoped Personal Access Tokens provide a more granular level of access control, scoped specifically at the account level rather than the user profile. This marks a substantial improvement in security by preventing the reuse of tokens across multiple accounts, reducing the risk associated with compromised credentials, and adhering to the principle of least privilege.
As part of our ongoing efforts to enhance security, we're moving in the direction of account-scoped access across the platform, reinforcing our commitment to safeguarding your data and workflows in dbt Cloud.
This feature is set to roll out next week to all dbt Cloud accounts. You'll have at least 90 days to transition from profile-scoped tokens before they are eventually deprecated.
Moving forward
In our ongoing efforts to refine dbt Cloud, we'll continue to introduce key features aimed at bolstering your workflow's security and continuity.
These updates reflect our dedication to providing a robust and reliable platform — one that quietly does its job, and that you never have to think about when you’re not building, testing, or exploring data products.
“Is this a service I can build my business around?” By continuously improving not just what dbt Cloud is capable of, but how it’s delivered, we aim to ensure the answer to this question is always a confident, resounding yes — regardless of the size and scale of your organization.
Last modified on: Feb 21, 2024
Achieve a 194% ROI with dbt Cloud. Access the Total Economic Impact™️ study to learn how. Download now ›