class: title, center, middle # Packages --- # What are packages? -- Packages are dbt projects that you can install in your _own dbt project_ They have: * A `dbt_project.yml` file * And contain macros, models etc. ??? Teacher notes: * We're starting with a very literal example (using macros), and then going to talk more broadly about packages. Before we hit databases, there's a nice story of "woah that's a package, which is just a dbt project? you know what else has a dbt project? (A: dbt)." Or, "yo why is there 138 macros in our project when we only added one". And from there we can talk a little bit about writing custom tests + using custom schemas + custom aliases + custom overviews. Once you get that concept down, it's so easy to understand how you can get dbt to do things you want it to do --- <!-- Template for displaying the focus slides for each presentation --> <h1>Packages and projects | Focus</h1> <li> Explain the power of packages in a dbt project </li> <li> Install and configure packages for your dbt project </li> <li> Leverage macros and models from a package </li> <li> Explain how dbt decides between identically named macros and models </li> --- ## Use case: Using macros from a package 1. Add a `packages.yml` file to your project -- ```yml # packages.yml packages: - package: dbt-labs/dbt_utils version: 0.8.6 ``` .caption[Always check [hub.getdbt.com](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/) for latest version] Note: you can also install a package from a git URL, or from a local directory. --- ## Use case: Using macros from a package 1. Add a `packages.yml` file to your project -- 2. Run `dbt deps` to install the package (check out the `dbt_packages` directory!) ```shell $ dbt deps ``` -- <img style="width: 4%; position: absolute; bottom: 12%; right: 5%;" src="/ui/img/dbt-standalone-new.png"> </img> <p style="width: 4%; position: absolute; bottom: 4%; right: 5.5%;">demo</p> ??? Demo Time: * add package by navigating to https://hub.getdbt.com/dbt-labs/dbt_utils/latest/ and copying yml * run `dbt deps` command * Show the dbt_packages directory * Note that these are .gitignored --- ## Use case: Using macros from a package 1. Add a `packages.yml` file to your project 2. Run `dbt deps` to install the package -- 3. Access macros from the package ```sql -- models/utils/util_days.sql {{ dbt_utils.date_spine( datepart="day", start_date="to_date('01/01/2016', 'mm/dd/yyyy')", end_date="current_date" ) }} ``` -- <img style="width: 4%; position: absolute; bottom: 12%; right: 5%;" src="/ui/img/dbt-standalone-new.png"> </img> <p style="width: 4%; position: absolute; bottom: 4%; right: 5.5%;">demo</p> ??? Demo Time: * create date spine model `util_days.sql` * run date spine via "Preview" button * show compiled SQL --- ## Use case: Using macros from a package 1. Add a `packages.yml` file to your project 2. Run `dbt deps` to install the package 3. Access macros from the package ```yml # models/marts/core/resources.yml version: 2 models: - name: orders tests: - dbt_utils.expression_is_true: expression: "amount > 0" ``` ??? * Go through some of the most common macros in dbt-utils (there might be overlap with previous lesson) --- ## Q: What happens when you add a package that contains models? -- These models get added to your project and will be run when you execute the command: `dbt run`! --- background-color: #005E7A <img src="/ui/img/dbt-learn/lessons/packages/dbt_artifacts.png" style="width: 100%;"> --- ## Use case: models from a package Install [dbt_artifacts](https://hub.getdbt.com/brooklyn-data/dbt_artifacts/latest/) package: ```yml # packages.yml packages: - package: brooklyn-data/dbt_artifacts version: 0.8.0 ``` - `dbt deps` - `dbt run --select dbt_artifacts` ??? * For this specific package, you may need to run `dbt run-operation create_artifact_resources` after `dbt deps` and before `dbt run ...`. * It's already been run in the group training snowflake account and automatically created resources in the analytics.dbt_artifacts schema --- ## Use cases for packages: ### Macros: * Versatile macros, especially for “How to Write this in SQL” problems ([dbt-utils](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/)) * Ad hoc queries/snippets you find yourself writing over and over ([audit_helper](https://hub.getdbt.com/dbt-labs/audit_helper/latest/), [codegen](https://hub.getdbt.com/dbt-labs/codegen/latest/)) ### Models: * Common staging models for one data source ([facebook](https://hub.getdbt.com/dbt-labs/facebook_ads/latest/)) * Follow industry standards for boilerplate-yet-complex modeling ([snowplow](https://hub.getdbt.com/dbt-labs/snowplow/latest/), [segment](https://hub.getdbt.com/dbt-labs/segment/latest/)) --- class: subtitle, center, middle # Hold up --- ### What about all those _other_ macros? Even before we added `dbt_utils`, we had a ton of macros whenever we ran dbt. .center[ <img src="/ui/img/dbt-learn/lessons/packages/builtin-macros.png"> ] ??? This slide is truly for curious minds. --- # dbt is itself a package! .left-column-33[ * a.k.a "the dbt global project", named `dbt` * Schema tests are macros prefixed with `test_` * Materializations are macros, too [_Check it out!_](https://github.com/dbt-labs/dbt-core/tree/main/core/dbt/include/global_project) ] .right-column-66[ <img src="/ui/img/dbt-learn/lessons/packages/dbt-global-project.png" style="width: 100%;"> ] --- # Everything's a project! So: * You have your own dbt project * You also have the dbt global project * You can also install additional projects as packages --- ## What happens if you create a macro with the same name as a global macro? -- dbt will use _your_ version! This means you can: * Change the way tests are implemented ([docs](https://docs.getdbt.com/docs/writing-code-in-dbt/extending-dbts-programming-environment/custom-schema-tests/)) * Generate a custom overview for your documentation site ([docs](https://docs.getdbt.com/docs/building-a-dbt-project/documentation#setting-a-custom-overview)) * ⚠️ Change how materializations work ([Example](https://www.loom.com/share/1507ac7baee043d784a65c770d387de1)). .caption[ [Learn more](https://discourse.getdbt.com/t/did-you-know-dbt-ships-with-its-own-project/764) ] --- ## Checkpoint - **Share Out:** What ways can packages level up your analytics workflow at your organization? -- ## Questions?? --- class: subtitle # Knowledge check You should: * Be able to explain what a package is * Be able to install a package in your own project and use macros/models from the package * Understand that dbt itself is package --- class: subtitle # Zoom Out <!-- This lists all the presentations in the current day --> <li> Polish your dbt project</li> <li> Intro to Jinja</li> <li> dbtonic Jinja</li> <li> Packages and projects</li> -- <li class="orange"> dbt Materializations</li> <li class="grey">Incremental Models</li> <li class="grey">Techniques of the trade</li> <li class="grey">Closing ceremony</li> <!-- This adds a link to the first slide of the next presentation --> <a href="/dbt-learn/lessons/materializations"> <img style="width: 4%; position: absolute; bottom: 10%; right: 5%;" src="/ui/img/dbt-standalone-new.png"></img> </a> --- ## Jinja, Macros, and Packages Working Session Let's lock in the learning by completing the following: - Add the dbt_utils package to your project and install with dbt deps - Create a date spine using the dbt_utils macro - Create an `orders_by_date` - Explore other Jinja/Macros/Package techniques ### Resources: - [docs on importing packages](https://docs.getdbt.com/docs/building-a-dbt-project/package-management/) - [dbt_utils on the package hub](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/)