Blog Introducing Webhooks in dbt Cloud

Introducing Webhooks in dbt Cloud

Jeremy Hutt

Mar 01, 2023

Product News

Today, we're making it possible for dbt Cloud to notify other applications and tools when certain events take place in dbt Cloud, through outbound webhooks. Think: triggering real-time notifications or workflows in Slack, PagerDuty, or Jira when a dbt job fails, for instance.

This feature is available only for dbt Cloud users on either a Team or Enterprise plan.

Most of you are likely familiar with the concept behind webhooks, but as a quick refresher:

• Webhooks are conceptually similar to APIs, with the key difference being that an API must be polled by external tools, while webhooks can push information when an event occurs.
• This information is typically delivered in the form of a JSON payload with specific parameters that get sent to a unique URL, in order to communicate with an external application.

When to use webhooks

If your team could benefit from external applications being notified in real-time of dbt Cloud events, webhooks can be a much more effective solution than building a polling mechanism against the dbt Cloud API.

In our conversations with beta customers and technology partners, some common examples of use cases for webhooks have included:

• Sending real-time notifications (e.g. using Slack, Microsoft Teams, or PagerDuty) to alert relevant stakeholders when a dbt Cloud job run fails.
• Creating a task in a task management tool (e.g. Jira or Asana) based on when a specific event in dbt Cloud is triggered.
• Notifying external orchestration tools, BI, or reverse ETL tools to schedule their own workflows when a dbt Cloud job completes.

That said, in building this experience, we aimed to deliver a generic webhook feature that relies on raw JSON, allowing for as much flexibility as possible. I’d love to hear about other ways you’re thinking of using webhooks!

Getting started with webhooks in dbt Cloud

You can find webhooks in your dbt Cloud Account Settings. Once you select “Webhooks”, you can configure a new webhook subscription to trigger on specific events at either the account or job level. From there, it's as simple as selecting which events you want to have this webhook trigger on, and adding the receiving URL.

dbt Cloud webhooks are also accessible programmatically through an API endpoint. If you want to create, edit, or delete webhook subscriptions programmatically.

You can read up on how to do that – as well as learn more about how webhooks in dbt Cloud work more generally – in our docs here.

Here’s an example of what the JSON output of a dbt Cloud webhook looks like:

{
  "accountId": 1,
  "eventId": "wev_2L6ZDoilyiWzKkSA59Gmc2d7FDD",
  "timestamp": "2023-01-31T19:29:35.789265936Z",
  "eventType": "job.run.completed",
  "webhookName": "test",
  "data": {
    "jobId": "123",
    "jobName": "Daily Job (dbt build)",
    "runId": "12345",
    "environmentId": "1234",
    "environmentName": "Production",
    "dbtVersion": "1.0.0",
    "projectName": "Snowflake Github Demo",
    "projectId": "167194",
    "runStatus": "Success",
    "runStatusCode": 10,
    "runStatusMessage": "None",
    "runReason": "Kicked off from UI by test@test.com",
    "runStartedAt": "2023-01-31T19:28:07Z",
    "runFinishedAt": "2023-01-31T19:29:32Z"
    }
}

What happens next? The external (receiving) application will usually send a callback message over HTTP with a status code such as 200 or 404 to notify dbt Cloud of whether the webhook was received successfully or not.

Webhooks are typically secured by each user of an external application getting a unique, random URL to send webhook payloads to. This can also optionally be secured through a secret or key.

And with all of that said, if you're having any trouble getting started, please do reference these helpful guides our Community team has put together.

Leave us Feedback!

This is just the beginning. We hope to continue to build out an even more robust webhooks experience to make it possible to meet a broader variety of needs for more organizations. Stay tuned for more!

Don’t hesitate to let us know what you think of webhooks in dbt Cloud. Feel free to reach out in the dbt Slack by either posting questions in the #dbt-cloud channel or reaching out to me directly (@Jeremy Hutt).

Last modified on: Feb 28, 2024