As dbt projects scale, running the entire DAG for every change quickly becomes inefficient—both in time and compute cost. To solve this, dbt offers state-based selection, which enables you to compare your current project against a previous state and run only the resources that have changed.

In order to attain this, it uses the manifest.json file and the --state method. Learn exactly how this works and how to customise state-based selection to your needs.

The manifest file

After executing a dbt command, a manifest.json file is created in the project’s target directory. Even if you’re only running specific nodes of the project, this file will contain a full description of your project’s resources and configurations.

The manifest file is used for analysing the project’s health, compiling the docs site, and allowing state comparison.

It’s important to remember that with every parsing, dbt overwrites the manifest file in the target directory. Therefore, you should never compare a project’s state to the target manifest, as it leads to incorrect or unreliable results (e.g., no changes detected or warnings about a missing manifest).

Instead, you can store artifacts from a previous run in a separate folder and use that path for state comparison.

State comparison

The state method enables you to select nodes according to a comparison with previous versions of the project.

The commands above are using state:new and state:modified, and dbt offers another two states.

• state:new will only select new nodes, that is, nodes whose unique_id are not already included in the comparison manifest. Bear in mind that a test is a resource, so new tests in existing models will also be selected.

• state:old is the opposite of new, that is, it will select nodes that already exist in the manifest.

• state:modified selects existing models that have been modified.

• state: unmodified is the opposite of modified and runs all nodes that had no changes.

Deep-dive into state:modified

Due to the complexity and diversity of dbt projects, there are aspects to consider when using state:modified to ensure it meets your specific needs.

Additional subselectors:

You can specify which aspect of the node needs to be compared against the manifest using the following subselectors:

• state:modified.body → changes to the node’s body like in a SQL model or seed values.

• state:modified.configs → selects resources whose configs have changed since the last manifest. These changes don’t include database, schema, and alias.

• state:modified.relation → changes to database, schema, and alias.

• state:modified.persisted_descriptions → changes to descriptions, but must have persist_docs enabled at each level.

• state:modified.macros → changes to upstream macros called directly or indirectly.

• state:modified.contract → resources whose contract has changed. In the case of breaking changes, an error will be thrown.

If no subselectors are used, dbt will enforce all of the above, plus other resource-specific criteria such as freshness, quoting, and maturity.

Impact of group changes:

While running on state:modified, dbt will track node-level config changes, including group definitions, access permissions, deprecation dates, and latest_version.

If a group definition changes, the models affected will be caught under the state:modified selection. However, as the group configuration can be used to control resource access, this may cause breakage downstream.

Even if you update the group everywhere, dbt still flags these nodes as modified, and any resulting access or reference issues are detected during parsing or execution.

Attention to partial parsing: with partial parsing enabled, dbt may not immediately detect downstream reference issues during parsing, but they will surface during execution (especially if downstream nodes are included in the run).

The “nothing to do” error and selecting downstream resources:

This error is thrown when dbt flags a node as modified, but nothing needs to be run (for instance, in the case of metadata-only changes).

The run will be successful, but things may still break downstream. That is why it is recommended to include downstream models using state:modified+.

Seed files behaviour:

State comparison differs for seed files according to their size:

• Small seeds (< 1 Mib): included only when the contents change,

• Large seeds (>= 1 Mib): included only when the seed file path changes.

In our last post dedicated to data tests, we are going to cover the test command and how to define how dbt handles tests in dependencies.

The dbt test command

Once tests were declared and configured, you need to run dbt test to apply them and identify potentially failing records.

You can use the --select flag to run specific tests according to their type (data, unit, generic, or singular) and/or the model name.

Defining how dbt tests dependencies

Unlike other resources, tests upstream or downstream can be selected indirectly. You can use modes to choose how dbt handles tests in other models.

Eager mode (default)

This is the most inclusive option and the default mode. Which tests will be built?

• Tests referencing the selected nodes as well as those of their upstream AND downstream models, even if they don’t directly reference the selected nodes.

Cautious mode

This is the most exclusive option. The tests that are built are:

• Only the tests that exclusively reference the selected nodes.

Buildable mode

This is a middle ground between Eager and Caution. Useful if you want to build a subset of your DAG without running into failures caused by unbuilt nodes in Eager mode. Includes the following:

• Tests that reference the current node as well as those of its ancestors, even if they don’t reference the selected nodes.

Empty mode

Ignores all tests.

Let’s cover the possibilities around configuring your tests and deep-dive into customisation options to handle large datasets and set default paths for stored failures.

Where to define data test configurations

Data tests can be configured in:

1. under the config: method nested inside properties - this is for generic tests only.

2. inside a config() block within the test’s SQL definition.

3. in the dbt_project.yml file.

The config inside the dbt_project.yml file is always the last precedence, however the hierarchy differs for singular and generic tests:

• Generic tests: (1) takes precedence over (2)

• Singular test: (2) takes precedence over (1)

Resource-specific configurations

The following configs are available specifically for data tests:

Let’s define each one of these configs:

fail_calc:

All tests return the set of records that fail the assertion. By default, dbt returns the count(*) of records.

This config allows you to change this calculation. For instance, for a test that checks for duplicate records, you can set fail_calc to "case when count(*) > 0 then sum(n_records) else 0 end".

If this calculation returns an error, like a data type error, the test won’t pass or fail, but it will return an error.

limit:

Limit the number of failures returned, specifically useful for large datasets and when failures are being stored in the database.

Example: “limit: 1000”

severity, error_if, warn_if:

By default, when even a single record fails the assertion, the tests will produce errors and prevent the model from running. This config allows you to add some leniency to your tests.

When you set up the severity: config to error, dbt will look for the error_if and warn_if configs, which will set up the conditions to bring an error or a warning:

If severity: is set to warn, then dbt skips the error_if config and only considers warn_if. As with warnings of any nature, dbt will not fail the run, but you can promote warnings to errors using the --warn-error flag when running the models.

store_failures:

Optionally set a test to always or never store failures in the database. By default, the failed results will be stored in a schema named _dbt_test__audit, but this can be customised.

Ensure you have the correct permissions to create schemas in the database.

where:

Allows you to limit the records being tested. For instance, you can tell dbt to only test the assertion on records for which column A is not null.

This configuration generates a macro called get_where_subquery that will replace the source reference with a subquery:

For this config, you have access to {{ var( ) }} and {{ env_var( ) }} but not custom macros. This is because configs are evaluated before custom macros are compiled.

To use custom macros in the where: config, you can override the default get_where_subquery.

For instance, say you want to test only the latest partition:

Create the custom macro latest_partition.sql that identifies records of the latest partition:

Now, create a macro named get_where_subquery to override the default macro and call the custom macro you just created. Don’t forget to specify the test name this should refer to if you don’t want the get_where_subquery to be modified for all tests.

General configurations

• enabled: if set to False, dbt does not consider this resource part of the project.

• tags: add common tags to resources to apply commands to all in one go using the “tag:” method.

• meta: adds custom metadata for the resource to the manifest.json file and documentation.

• database, schema, alias: relevant for store_failures only to set a custom path for failed records.

We’ve learned what data tests are and how to implement them logically in your DAG. Now, we are going to understand how to declare the properties for data tests.

These properties allow you to correctly set up your tests and apply them to multiple columns.

dbt also offers some options in terms of naming these tests, which can make logs, artifacts, and database table names more logical.

Description

Use the data_tests property to define assertions about a column, table, or view. It can be used to define generic data tests, including the 4 built-in tests.

It can also be optionally used for singular data tests for documentation purposes using the “description” parameter.

Test-specific properties

Out-of-the-box generic tests require specific arguments for the column it’s being applied to. It’s also possible to test using expressions involving multiple columns.

Built-in generic tests:

These tests have to be applied at column level.

Not_null

This test only requires the data_tests property to declare the test and no further arguments are needed.

Unique

Similar to not_null, unique only has to be declared. However, it takes an optional “where” config to filter the rows this is applied to.

Accepted_values

This test requires no further configuration for strings. However, if you want your accepted values to be non-string values (like integers or booleans), you need an additional “quote” argument to be set to false.

Relationships

This test takes an argument “to” and a “field” that define which child table it should look for corresponding values from the parent table.

Please note that this test excludes null values from validation. If you want nulls to cause failures, use the not_null test.

Testing an expression:

When a data test involves multiple columns, it doesn’t make sense to place it under the “columns” key. In these cases, you apply the test to the model.

For instance, when using the dbt_utils.expression_is_true:

Custom test names: the “name” property

dbt will define the test names in the artifacts by concatenating the test name, the resource name and, if relevant, column name and arguments.

You may want to define a custom name for your tests to obtain a more recognisable name in the artifacts. For that, you can use the “name” property.

Besides showing in artifacts and logs, this custom name can be used by the “select” method to run only a specific data test.

You cannot have duplicate custom names for the same column. However, you can repeat them across different columns or models. In this case, the “select” method will run all tests under the same custom name.

You can also use custom names when you need to apply the same data test type to one column. By giving them custom names, dbt will know how to differentiate them.

Alias for “store_failures” database table

When switching “store_failures” on, dbt will store test results in a database object. By default, it takes the name of the test, or the custom name if defined, as the name of the database table.

If you’d like the database table to have a different name, you can set an alias for the test.

Using “test_name” to flatten config

dbt offers the option to define test properties using the “test_name” property to make your file easier to read. The result is the same.

Data tests are assumptions we make about resources in our project. For instance, we can configure a model to fail or issue a warning when certain preset requirements are not met.

dbt offers a few out-of-the-box solutions, but it is also possible to create custom tests to reflect specific business logic.

In this post, we will cover the reasons why you should add tests to your project and suggest a gradual implementation of data testing. Finally, we’ll go into the nitty-gritty of adding data tests to your DAG.

In the next posts, we will cover data test properties, configurations, and the dbt test command.

The importance of data testing

Whether you deal with internal or external stakeholders, trust in the data is fundamental to getting buy-in and support. Therefore, data professionals cannot afford the risk of sharing incorrect data.

When we are engineering a complex project, we could end up running countless models successfully without ever noticing that something is off.

Data testing helps us ensure that certain assumptions are met before models are built successfully - or at least throw a warning for less critical issues.

The built-in data testing tools in dbt enable you to check if:

• The SQL code is doing what it’s meant to do

• Assumptions about your source data are correct

• Anything changed in your source data that is impacting your assumptions

Introducing data tests gradually and sustainably

You don’t need to set up every data test under the sun just because they are available. dbt’s general rule of thumb goes for data testing too: don’t make your project unnecessarily complex.

dbt suggests the gradual introduction of tests following the phases below:

1) Simple test use cases

Initially, you might not be testing your data.

However, if you’re consistently running into data quality issues or you’re preparing to enhance your project’s complexity, it’s time to consider adding tests.

• Primary Keys: Unique & not_null tests help you identify values that mess with filters, joins, or aggregate calculations.

• Accepted values or relationship tests: for columns with predictable values, this test flags any new values that appear in the source data.

2) Preempting issues

As your project grows, you might want to start preparing for outliers or lack of freshness.

Here you can go beyond dbt’s out-of-the-box tests and into custom testing to spot and treat these corner cases in your data.

You can also consider configuring freshness so dbt alerts you when a source is stale. Source freshness was addressed in Checkpoint 2.

3) Tackle workflow testing

Once you’ve safeguarded your project, it’s time to expand testing into the pull requests with CI/CD jobs. These will be discussed further in this Checkpoint.

dbt also recommends creating a testing culture within the team with data tests consistently being added to new code and processes in place to debug test failures. I’d also add the usage of a good PR template to ensure data quality, another aspect that will be discussed later in this Checkpoint.

4) Avoid over-testing

At this stage, you could potentially find yourself drowning in warnings and errors generated by data tests.

Consider removing those warnings that you are consistently ignoring so your project only flags those that really deserve your attention. A periodic audit for deprecated fields or tests can be a good tactic.

You can also evaluate switching errors to warnings if you find that your project is delayed over failures that are not that severe.

Finally, consider the value for money of your tests: data testing may incur computing and development costs.

Adding tests to your DAG

What are data tests in dbt?

Firstly, it’s important to understand that, like most things in dbt, data tests are SQL queries. Their statement will pull a list of records that fail the test you set up.

There are two types of data tests in dbt:

Singular data tests

Singular tests are a custom SQL query that returns failing rows saved in a .sql file in your test directory. These are useful to test assumptions that involve specific business logic.

The SQL query should return the list of failing records and their relevant attributes. Remember to omit semicolons at the end of the statement.

You can use Jinja in singular testing, as shown in the ref( ) function above. The name of the file becomes the test name.

Any singular tests included in the test directory will automatically be summoned by the dbt test command.

They don’t need to be referenced or configured in .yml files. However, you may want to document them under the data_test block with a description:

Generic data tests

These are parameterised queries defined by a test block, like a macro, that can be referenced throughout your project. dbt comes with four generic tests out of the box.

Once defined and saved in the tests directory, this test has to be defined as a property of a given resource in their respective yml file.

The example above is simply checking a column for nulls. However, dbt already includes this type of testing and three others as a native feature.

The out-of-the-box generic tests assert that:

• unique: a column (or combination of columns) contains no duplicate values.

• not_null: a column contains no null values.

• accepted_values: all values in a column are in a set of possible values.

• relationships: each value in a given column exists in another table.

There are additional configurations available for tests, which will be covered later in this Checkpoint.

Besides the native data tests, dbt also offers the ability to write custom generic tests and import them from Packages. The former is not part of the Certified Developer Exam, so it won’t be covered in more detail here. But we’ve explored key open-source packages in the Packages post that include testing macros.

We’ve arrived at the final part of our Practice Project - Checkpoint 4. This was definitely a heavy checkpoint, but at the end of this post you’ll see how mature our project looks now.

The next Checkpoint will cover data tests and deployment - lighter in content but no less important!

Python Models

Let’s create a simple Python model that calculates the lifetime value of each employee.

In real life, you wouldn’t use Python for such a banal calculation. SQL is always preferred on dbt. However, my Python skills are a bit rusty, so I’ll keep it simple.

The next step is to configure the newly created model. We are going to materialise it as a table.

Now, we are going to run the model. As usual, because I have billing disabled in BigQuery, I’m not able to run it successfully. There would be ways to run this locally, but it would involve more advanced features.

The important thing is that we understand the minimal requirements of Python model configuration, how to use the “model” function, and test the ability to reference sql models.

Exposures

We’re modeling all this data for our B2B company so that the end-users can access clean, consistent data via a visualisation tool such as Power BI or Tableau.

Creating the actual dashboard is out of scope here, but we can practice configuring a dashboard as an Exposure.

The first step is to configure our Exposure in an exposure.yml file:

Not all these configs are required and you can make it as robust or simple as you’d like.

Finally, we are going to run our new Exposure so it becomes a node in the DAG.

Please note I have a deprecation flag on, as dbt recently started requiring the docs config to be under a “config:” block.

After a successful run, you should now see the Exposure as part of your DAG. I’ll add a small detail: fact tables should be in a single layer, as per dbt’s best practices, so do what I say and not what I do!

Documentation

The last step of this very long Checkpoint is to add and configure documentation in our project.

Throughout our project, we’ve been adding descriptions to sources and other resources:

However, our fct_orders model is quite an important one and includes complex discount and VAT logic. Therefore, for that model, we’re going to include a doc block for a more thorough description.

For that, we need to create a docs.md (optionally in a docs folder within models), where we can store all doc blocks we may need.

In the fct_orders model, let’s replace the standard description we had in the properties.yml file with our doc bloc.

Finally, we can generate the docs (compile all the descriptions, metadata, and doc blocks) and then serve them to a bespoke webpage.

I’m using the IDE, so I don’t need to serve the docs. The next step is to click the docs button to see our docs website:

And there we have our brand new docs page. There are other ways to customise it, by creating an overview.md doc block to replace the text in the homepage, but we’ll keep it simple for now.

We can also see our doc block for fct_orders in action:

And finally, a complete visualisation of our DAG with colour-coded nodes.

Yes, yet another part of the Practice Project Checkpoint 4! And it’s not the last one.

I just started a new job and the time I have available for this Study Guide has drastically shrunk. So, forgive me for the multiple parts. I’m just glad I’ve been able to stay consistent.

In Part 4, we’ll tackle Grants and Snapshots. Next part, hopefully the last one, we’ll add Exposures, Python models, and documentation to the project.

Grants

For this Practice Project, we will define grant permissions in YAML files.

Let’s think of this scenario: we need to limit access to the “dim_employee” model as it shows confidential information on the employees.

Before we get started, it’s important to understand that each adapter will have specific terminology that needs to be used. For instance, in Big Query, which is the adapter I am using, the “select” role is called “roles/bigquery.dataViewer”.

Setting up grants

We’ll first set up the grant config for the “dim_employee” model in the dbt_project.yml file.

Below, I gave a fictitious user permission to query my table in BigQuery.

Conditional grant with Jinja

We can also use Jinja to set conditional permissions.

Even though the model was built successfully for both grants, when I queried the object privileges for “dim_employee” on BigQuery, I couldn’t get any data to return. I could not get this user to come up under my IAM permissions either. Not sure what this means, but I had to move on.

Snapshots

We are using a dataset created by Leo Godin on Medium, and I’m not entirely sure whether company or employee details change over time.

However, for the sake of our example, we are going to create 2 Snapshots to record changes to company and employee data.

Adding live employee and company data

First things first, we are going to connect our employee and company data to the live public database that Godin set up - so far, we’ve been keeping that data static. Don’t forget to run these models and their downstreams once done.

Defining Snapshot strategy

We don’t have an “updated_at” field we can go by in our data, so we’ll adopt the “check” strategy.

It is recommended to store your snapshots in a different schema, as once dropped, you cannot recover the historical changes lost. So we will define it to be built in our prod schema.

Also, dbt recommends basing snapshots on staging models that have already been cleaned rather than the raw data.

Run the “dbt snapshot” command and check that the two tables were created on dbt and in the warehouse.

Verifying Snapshots

Upon running the command, you will see the tables being created in the warehouse. On the first run, dbt simply copies the existing records and adds accessory columns like “dbt_scd_id”, “dbt_updated_at”, and “dbt_valid_from”.

Upon subsequent runs, dbt will check which records have changed or if any new records have been created. The updated record and any new records will be inserted into the snapshot table.

Frequent jobs for Snapshots

In a real-life scenario, you’d need to have a frequent, scheduled job to run the snapshots to ensure you’re capturing all changes.

In my case, since I don’t have billing enabled on BigQuery, the Snapshots wouldn’t quite work the way they’re supposed to, so I’ll skip this step for this Practice Project.

Some configurations look straightforward in the documentation but when you actually try to implement them in a complex project, you run into all sorts of bugs.

This was the case with Model Governance features. I had some trouble setting up the access modifier correctly and, unfortunately, could not implement model versioning (I explained the bugs below).

This goes to show the importance of practising. Running into bugs is very frustrating, but they help us really grasp the workings behind dbt.

Model Governance

For this practice step, we will add some model governance features to our project.

Here, we will add all features for practice, but in a real-life scenario, you would only add these features as needed. In data modelling, we always strive to reduce unnecessary complexity.

Model access

This is the ability to make models public or private to control who can ref them.

First, we need to define a group. We already have some groups defined by department, like the “customer_success” group, which holds the “fct_company” and “fct_company_trend” models.

By default, all groups are protected, so let’s change this group to private and try to ref one of its models from outside the group.

I then created a new model and tried to reference one of the models. Upon compiling, I got the expected error.

Model contracts

They restrict the shape of a model in terms of column names, data types, and constraints.

Following dbt’s recommendation, we will enforce some contracts in our fact tables. For instance, fct_orders could use some contracts, as we started feeding it with live data in the Practice Project 4 - Step 2.

Firstly, let’s add the contract and constraints to the dbt_project.yml file. Please note that I’m using BigQuery and, while it enforces schema contracts, it doesn’t enforce constraints.

I modified the fct_orders table by selecting only two of its columns to see it fail.

Upon building the model, I get the model contract error:

Model namespaces

Delineate ownership boundaries and responsibilities through naming structures.

We’ve been working with namespaces throughout the Practice Project. For instance, when we added groups and access modifiers to the customer_success subdirectories above, we were using namespaces.

Also, when we use the source function, we are declaring the source namespace. In the example below, it would be “dbt_fake_live”.

Model versions

Versioning for the models, allowing for adjustments to be made without breaking access for the end user.

Now, for this one, I must be honest: I could not get it to work.

I tried versioning fct_orders and set up the configuration as per the documentation. However, dbt refused to read “fct_orders” as the v1 model. I tried using the defined_in optional parameter, renaming fct_orders to fct_orders_v1. Nothing worked.

Upon trying to compile the v1 (dbt compile -s fct_orders:v1), dbt seemed to see two “fct_orders” models, one versioned and one unversioned.

Also, interestingly, as soon as I created the fct_orders_v2 file, I got a server status error. But when I edited the file name to fct_orders__v2 (with double underscore), the server status went back to Ready. Go figure.

I will revisit this eventually, but I needed to move on with the Practice Project. I wanted to be honest and show you how, even after reviewing documentation thoroughly, some configurations are tricky to set up in a real-life environment.

If anyone’s got any clue as to what could be causing this issue, send me a message or leave a comment!

Incremental materialisation is one of the trickier configurations of dbt. It requires attention to detail as the placement of the configs affects how and when the incremental rows will be processed.

I covered only the essentials in incremental materialisations in this post due to a limitation of my adapter’s free tier. I think a real-life scenario would present more challenges than I was able to replicate here.

Still, this is good practice to understand the intricacies of this type of materialisation and the many ways in which it can be configured.

Incorporating live data into our project:

So far, we have been working with static data in our Practice Project. Now, thanks to Leo Godin’s efforts in setting this fake database up, we are going to add live data.

Our project will now receive new orders every day. We will assume no new employees, products, or companies are added. Only new orders.

Setting up the new connection:

Godin’s data is hosted in a public BigQuery dataset that we can directly add as a source to our project. You can view documentation on the full dataset here.

Replacing the source of our orders table:

Now, we are going to replace the orders source in the “stg_orders” model. Optionally, you can also remove the static orders table from the source.yml file or use the “enable: false” property to remove it from the project.

Configuring the incremental materialisation:

You want to add the incremental strategy in the layer where the heaviest transformations take place. This way, these costly operations will only be applied to new rows at each run.

In our case, the optimal model is int_orders_detail. We are also going to implement a merge strategy and limit the scan of new records to orders from 2025. We are taking the assumption that orders from previous years never change.

Please note that I’m working on BigQuery’s free tier in this project, which doesn’t support DML queries. Therefore, even with the correct configurations, the “dbt build” command will still fail. However, the logs will show us that dbt tried to implement the incremental materialisation.

dbt_project.yml configuration

According to the documentation, you can add the incremental configurations to the dbt_project.yml or the model itself.

Upon trying to build the models, the command fails and the logs show that dbt tried to run a DML query, but my adapter has billing disabled.

In-model configuration

Using a config block, we can set all the configs for the incremental materialisation in the block itself. Remember that this block can be placed strategically in the code (for example, after the data has been filtered).

The logs are even more detailed in this case. You can see dbt tried to merge the new records, applied our predicates, and only searched for new records within orders from 2025.

We can also apply a simpler configuration to your incremental materialisation using the is_incremental macro:

While incremental_predicates filter the rows that the warehouse will consider during the MERGE step, is_incremental( ) filters the rows selected from your source before the merge happens.

The past 2 weeks have been hectic as I started a new job AND moved flats in the same week - I do not recommend it!

I did slow down with the posts a bit, but I have no intention of abandoning this project. With a full-time job, I’ll obviously need to adapt my schedule, but I aim to post at least once a week.

Speaking of the new job, in addition to the study notes from the past two weeks, I've included some valuable tips for landing a job in the age of AI recruiting below. Next week, I’ll share more about my new job itself.

Study notes added:

We finished the topics for Checkpoint 4 and began applying our learnings to the Practice Project. We covered a lot of topics in this Checkpoint, so the Practice posts will be broken into a few different parts.

A full list of the study notes is available in the study guide.

Here’s how to land a job in the age of AI recruiting

TL;DR: Work on side projects (like this dbt Study Guide) and talk about them on LinkedIn to increase your profile visibility.

I quit my data job in July 2024 to freelance and upskill. A year later, I felt the need to put my new skills into practice, so I began job hunting.

I started searching while in Europe, but the market was flat. After months of insisting, I made the tough call to return to Brazil—where I got hired in under a month.

However, even though the Brazilian tech market was booming, I never landed an interview for the dozens of jobs I proactively applied for. But every week, recruiters kept approaching me on LinkedIn. One of those messages led to the job I just started.

Sure, my 10-year on-and-off experience in data and a recent Big 4 contract helped to attract the recruiters. But it’s also worth noting that since launching my Study Guide, my LinkedIn visibility increased 9×.

The first big spike came when I announced the dbt Study Guide on 30th July (23.5k impressions). The next jump happened when I posted about speaking at the dbt Labs HQ in Dublin.

During this period, I gained 699 new connections—loads of aspiring data pros, but over half were senior professionals and people in IT services and consulting. LinkedIn won’t show how many recruiters follow me, but they definitely sit within these groups.

The bottom line:

AI has transformed job hunting. Candidates can now blast tailored CVs at scale, and HR teams use AI to filter the flood. Speaking to a human about your skills is harder than ever.

So, you need to let the recruiters come to you.

Show your work publicly. Share your learning, your projects, your Zoomcamps—whatever you’re building while you’re job searching.

Don’t stand still during your job search, applying mindlessly. Stand out and show yourself.

(Speaking of which, DataTalks Club has a new cohort for their Data Engineering zoomcamp coming up soon!)

Does this make sense to you? I would love to hear from you here or on LinkedIn!

We covered a fair amount of ground in Checkpoint 4, so we will need to break the Practice Project posts into multiple parts.

In Part 1, we will cover adding Seeds and installing Packages and Macros.

We will then continue with further posts covering Incremental Materialisations, Model Governance, Grants, Snapshots, Python models, Exposures, and Documentation.

Seeds

(All CSV files are in my git repo. Link to study note)

You are the Data Analyst in our B2B corporation (I should’ve made up a name for it, too late now), and you received requests from 3 different teams.

Requests

Request 1: The Sales Revenue department requested some additions to the fct_orders table. They asked for two new columns:

• A state_code column to understand where those orders were shipped to. They asked for the 2-letter code for the state of each employee who placed the order.

Let’s remember what our DAG looks like:

You could go ahead and add the new column only to the fct_orders table. But, if you want to future-proof your work, the best practice would be to add it to the stg_employees table so the column can be reused by other models, if needed.

In the case of a very large seed (which is not the case here), you could also consider adding it to the int_order_details model for performance.

Request 2: The User Support department needs to identify the countries that they need to call when contacting an employee by phone. They use the dim_employees table to find this information.

• Add an area_code column to dim_employees to identify the area code that each employee has on their phone numbers

Again, this is one to be added to the staging layer for reusability.

Request 3: A specific team from the Operations department said that some products are not produced in-house, so they don’t need to track them. They said they keep having to apply manual filters by prod_id to their Excel sheets, and it’s annoying. Other teams still need to have visibility of the external products.

• Use the prod_ids csv file provided by them to create a seed identifying in-house and external products. Then add a column to the fct_product_trend table with this information so they can filter by it.

The fct_product_trend also pulls from the int_order_details table. Similar to the first request, we are going to add the new column to the staging layer (stg_products) for reusability.

Configuring the new seeds

To fulfill the 3 requests above, we had to add 3 new seeds into our project. It’s time to configure them.

Let’s start by adding a properties.yml file to the seeds directory.

I added only documentation properties. In the future, we will spice them up with data tests.

You can also define configurations in dbt_project.yml for your seeds. In this case, I did not need quoting, set a different delimiter, or change data types, so I did not configure them in the project file to reduce clutter.

Packages & macros

(Link to study note)

This is not the first time we touch on Packages. In previous checkpoints, we added macros from dbt Hub (generate_surrogate_key and date_spine from dbt_utils).

In this step, we have a few tasks:

• Practice adding Packages from other sources: Git and internal Packages.

• Understand “dbt clean” in practice.

• Add our own macros using Jinja to improve our code reusability.

Adding Packages from Git and Internal Packages

Let’s start with a Git package.

In my project’s repo, I created a core_macros folder, which essentially acts as an external project I will add to my main project. Thus, it requires its own dbt_project.yml file and, of course, the macro we will be using.

I created a simple macro that outputs a string and stored it in this file structure:

The next step is to add the macro to the packages.yml and run “dbt deps”:

One thing to note is that dbt will pin this package to a specific commit in package_lock.yml. Which means that if you make changes to the macro in a new commit, you need to edit this information in the package_lock file:

Upon calling the macro, I need to use the subdirectory.macro_name reference so it finds the macro:

Now, let’s install an internally hosted macro:

We will use the package project_evaluator available on dbt Hub, but we will install it using the tar.gz link: https://github.com/dbt-labs/dbt-project-evaluator/archive/refs/tags/v1.1.2.tar.gz

After installing it with “dbt deps”, you can run the package’s models with the following command: dbt run --select package:dbt_project_evaluator

I’m running this project on a free tier of BigQuery, so my adapter won’t allow the tables to be created. But, inspecting the logs, I can see the package was successfully installed.

Understanding dbt clean: uninstalling project_evaluator

Since our adapter’s free tier doesn’t allow us to use the dbt_project_evaluator package, we should now remove it.

The steps for removing a package are:

• Remove the package from package.yml.

• Delete the package_lock.yml file.

• Run dbt clean to empty the dbt_packages folder.

• Run dbt deps to reinstall the previous packages.

If you inspect the newly created package_lock, you will see that dbt_project evaluator is no longer there.

Adding custom macros to our project

When we created the intermediate models for the orders, we added some case whens to define the level of discount based of time since joined as well as volume of orders. We also had a VAT that would vary according to the product category.

In order to ensure that the same logic is used across the company and to make this code easier to maintain, we should make them as re-usable macros.

Therefore, we are going to create 3 macros:

• disc_loyalty: companies with over 2 years of registering get a 5% discount, those with over 5 years get 10%. If less than 2 years, companies with over 200 orders placed get 4% discount.

• disc_volume: employees who placed over 100 orders get 5% discount, those with more than 50 orders get 3% and those with over 10 get a whopping 1% off.

• vat_category: certain categories have different VATs. Instead of the standard 18%, food has 5% VAT, electronics has 20% VAT, and clothing has 12% VAT.

I created a file for the 2 macros related to discount calculations:

And finally, a separate file for the vat calculation:

Now, we will call these macros in the intermediate models. For example:

We have covered macros in the context of Packages in a previous post. For this post, we will touch on how to create custom macros using Jinja.

We’ll also take the opportunity to understand how Jinja works in the dbt context and some best practices for using Jinja & Macros.

What is Jinja?

Jinja is a templating language that enhances the capabilities of SQL. dbt uses it in native functions like {{ config() }} and {{ ref() }}. But you can also use it to create bespoke macros or within the models.

These are some examples of what Jinja enables you to do:

• Using if statements and for loops

• Using environment variables for production deployments

• Change the build according to the target

• Pivot tables

• Abstract SQL codes into reusable macros

If you need support using Jinja, you can check out its own documentation and the dbt-specific functions, like ref and source. But the general syntax for Jinja is below:

• Expressions {{ … }} : when you want to output a string.

• Statements {{% … %}} : used for if statements, for loops, or variable control.

• Comments {{# … #}} : used for comments.

• You can also use a minus sign to remove unwanted whitespace in the compiled code, for example {{- … -}} or {{%- … -%}}

Jinja within models

Within models, you can use Jinja for injecting variables, environment-based logic, and references, before dbt runs the model. Jinja is processed at compile time, so you can use “dbt compile” to ensure that your code is compiling correctly.

Jinja within macros

Macros are the dbt equivalent of functions. Macros are defined in SQL files stored in the “macros” directory or any other directory defined in “macros-paths”.

They use Jinja to create reusable code to be used across multiple models.

Defining and using macros

This is how a macro would be defined. A file can contain more than one macro:

You can then use it as a function in the SQL model, entering any necessary arguments:

Finally, this is what the compiled code would look like:

Macros properties

Macros are not a resource like models, seeds, and snapshots. However, dbt allows them certain properties for documentation purposes.

Macro properties can only be defined in a properties.yml file stored within the “macros” subdirectory.

• name (required): must match the name of the macro file.

• description (optional): a string that describes the macro for dbt Docs

• docs (optional): shows or hides macros from dbt Docs

• arguments (optional): defines the arguments the macro accepts - again, for documentation purposes only, it does not affect how the macro runs.

Best practices for Jinja & Macros

Readability vs. reusability

It’s important to strike a balance between reusable code and readability. Too much Jinja in the model can make it hard for others to read.

Variables

Set the variables at the top of the macro/model.

SQL models serve the main purposes you may have within the dbt. However, some transformations cannot be performed in SQL, and at times, hundreds of lines of Jinja-SQL can be replaced by just a few lines of Python. In these cases, you may want to explore Python models.

Considering the limitations of Python models, which we will cover below, writing in SQL is still preferable. However, Python models enable developers to leverage Python’s open-source libraries for analysis while still being able to subject these models to dbt’s testing and documentation facilities.

The dbt Python models are run on the platform’s Python engine. Therefore, the adapter must support Python runtime.

The structure of a Python model

Each Python model is a .py file in the models folder. They begin with a function that takes two parameters: dbt and session. This function should return a final DataFrame.

• dbt: enables you to reference other models and sources, get environment info, project configs, and other dbt metadata.

• session: dbt sends the code to the warehouse, and this parameter allows you to interact with the objects there.

dbt uses Pandas DataFrames because it is the most common API for DataFrames. The downside is that Pandas works on a single node, which makes working on large datasets complex.

Referencing other models and contexts

Python models become a regular node in the DAG. You use the dbt.ref( ) or dbt.source( ) to reference models and sources. The Python model can also be ref’ed by SQL models.

Please note that Python models cannot reference ephemeral models. This type of materialisation inlines SQL code into the downstream model, and it is not possible to inline SQL code into a DataFrame.

For referencing contexts, you can use dbt.this(), dbt.this.database(), dbt.schema(), dbt.identifier().

You can also access var, env_var, and target names configured in the models config using the dbt.config.get( ), including the usage of f strings.

Configuring Python models

Similar to SQL models, Python models can be configured in the dbt_project.yml file, in the properties file, or in the model itself. Python models have access to almost all the configurations of SQL models. You can also select them using similar methods.

For on-model configuration, the {{ config() }} function is replaced by the dbt.config() method. While this is possible, more complex configurations should be done in a YAML file using the config property.

Materialisations

Python models can be materialised as tables or incremental models.

The incremental Python models support the same incremental strategies as their SQL counterparts. Just like in SQL, you need to filter the tables for rows with new data.

Python-specific functionality

Using and configuring PyPI packages

You can use Python packages as long as they are installed in the Python runtime within your data platform.

dbt recommends configuring these packages so they can be added to the project’s metadata. This can be done in a model config file or within the model itself:

User-defined functions (UDFs)

@udf functions are also supported in the Python models.

No code reuse

Currently, functions and general code cannot be reused across models.

Limitations of Python models

While Python models give you access to more complex transformations and Python libraries, certain dbt features and Python functionalities are not available.

The IDE Preview button does not work for Python models. For previewing the result, you should use methods such as .head() or .show().

The print() function is not supported within dbt. For debugging, you may want to check the platform’s logs, return a DataFrame that contains your logs and build it into the data warehouse, or write messages into a column:

And finally, Python models are also slower to run than SQL models and more costly to compute.

It’s important to keep a thorough, up-to-date documentation of your projects so users and colleagues can quickly understand the content of the datasets.

dbt enables developers to easily configure and generate documentation for their projects using descriptions and commands.

The documentation generated includes an overview of the project structure, descriptions of tables and columns, and information about the data warehouse.

dbt Docs is the feature available in the free plans, while paid users can use dbt Catalog. We’ll focus on dbt Docs here.

Adding descriptions to your project

Start by adding descriptions to tables and columns. They should be added to the same file where the data tests are declared.

If you need a description longer than one sentence, use “>”:

And to insert line breaks, use “|”:

You can also use a docs block to add more robust descriptions to your files.

Using docs blocks

Docs blocks allow developers to use Jinja and markdown in the descriptions.

You then point a description config to the relevant docs block using the doc( ) function:

Docs block files need to be uniquely named and use the “docs” Jinja tag. Their name can contain alphanumeric digits and underscores, but it cannot start with a number. The file type is .md.

dbt will look for docs blocks in all resource paths. Alternatively, you can place all docs blocks in one folder and point “docs-path” to it.

Using docs blocks for the overview page

Within your docs website, dbt creates a default overview page with helpful information about the project.

You can override this with specific information about your company's style guide, links to reports, or relevant contact information.

Using the dbt docs commands

dbt docs has two supported commands: generate and serve. They can be run in the command line or, if using the IDE, set to be run during a job.

dbt docs generate

This command generates the documentation website following these steps:

1. Copying an “index.html” into the “target/” directory.

2. Compiling the code and adding it to manifest.json.

3. Stores data warehouse metadata in the catalog.json file.

The combination of manifest.json and catalog.json makes up the data that powers the docs website.

Methods available for dbt docs generate:

You can use the “--select” method to only add specific nodes to catalog.json:

The “--no-compile” method skips step 2, that is, it skips compilation.

The “--empty-catalog” does not populate the catalog.json, which can speed up the command in development. However, in production, you can end up excluding valuable warehouse information from the docs, like column types, stats, sample values, etc.

Use the “--static” method to generate a static page for hosting in cloud providers. When you generate a docs site, you still need to serve it (more on that below), so the index.html, manifest.json, and catalog.json files get loaded into the front end.

With this method, dbt creates a standalone index.html file that can be used externally.

dbt docs serve

After generating documentation with “dbt docs generate”, this command starts a web server on port 8080 to serve your documentation locally and launches it using the default browser.

You can also use several flags to adjust how this server is created:

Grants can be defined for models, seeds, and snapshots to manage access to the datasets at build time. For resources with this config, dbt will run any grant or revoke statements when you run it. The grants will also be included in the JSON manifest.

Grants are usually defined in YAML files as a resource config. However, in specific cases, they might need to be defined in manual statements written in hooks:

• Apply grants to objects other than views and tables.

• More granular access permissions (row or column level, for instance)

• Apply them with advanced capabilities not offered out-of-the-box by dbt or that are adapter-specific.

Configuring grants

Like other configs, grants can be declared in dbt_project.yml, in a properties file, or in the resource itself.

The grants config also follows the config inheritance pattern of other configs: the more specific config takes precedence.

However, an exclusive feature of this config is the “add and merge” behaviour. Say you have a group of models that has a grant configuration at properties.yml level, but you need a particular model to have additional users. Here’s how you can achieve this:

Conditional grants

Like any other config, you can use Jinja for more specific configurations. In the example below, the user access depends on the environment:

Revoking grants

If you need to completely remove the access previously granted, deleting the grants configuration won’t do it. You need to give it an empty list, instead:

As I accidentally anticipated on Wednesday with an early publish (sorry about that!), this week we have gone deeper into a complex, but very important topic: Model Governance.

We have also covered simpler, but no less important features: Grants and Docs.

You can see the links to the new study notes below. And if you scroll down further, I have a bit of personal news!

Study notes added:

A full list of the study notes is available in the study guide.

Bye-bye, freelance life! Hello, full-time work!

Last year, I was feeling a bit stagnated at my full-time job. I was in a Data Analyst role that was largely non-technical. And the company’s data infrastructure was still in its early development days.

This led me to quit my full-time job to venture into the freelance world.

I did want to make a consulting career happen, and I landed some interesting contracts. However, I also wanted to take this time as an opportunity to upskill.

I learned the fundamentals of Data Engineering with the DataTalks Zoomcamp; I improved my SQL and Python skills; and I deep dived into data modelling and dbt (and hopefully will be certified soon!).

Just over 1 year later, the contracts started running a bit thin as life became harder for non-Europeans looking for work in Europe-based companies.

I felt like it was the right time to get back into full-time work and put all these new skills into practice.

I returned to my home country, where the job market was much better than in Europe, and landed a job in less than a month.

It’s a more technical Business Intelligence role within a large organisation with a mature data environment. The salary is fantastic too. I’m very excited!

In future weekly posts, I can share tips on how to land a data job (I’ve got a lot of experience in that!). But, for now, I just want to leave you with this message:

Don’t be afraid to make a career move and take a risk, especially if you don’t have a family to support, like me. Obviously, don’t be dumb. Think strategically, prepare yourself financially, and don’t take it as a holiday. But, do take that leap and go see what’s out there, if you feel like that’s the right thing to do.

Have the study notes been useful to you so far? How are you getting on with your self-paced learning journey? I would love to hear from you here or on LinkedIn!

In this post, we will dissect the definition, config, and best practices for Model Versions. This is part of the Model Governance section of the Study Guide, which also includes a post covering the other 3 features: Access Modifiers, Model Contracts, and Namespaces.

Table of contents

• What, why, and when: model versioning

• Creating a model version

• Configuring a model version

• Running a model version

• Optimising model versions

What, why, and when: model versioning

Why version a model?

Model versioning is a way to handle situations where you need to modify a model that’s already being used by downstream models or end-user tools. It serves 2 main purposes.

Handling breaking changes

When you run a model and it fails to build due to a breaking change (caused by a deleted column or changed data type, for example), all downstream models and end-user tools will be affected.

In smaller organisations, this may be an easy problem to solve. Fix the model, adjust downstream models and tools. Bob’s your uncle.

However, in larger organisations, a breaking change may affect tons of models and tools across different teams. Instead of making everyone react immediately, you can start working on a new version of the model while keeping the old version running without any breakage. This allows teams to migrate at their own pace without disrupting production workflows.

Controlled change management

Besides handling breaking changes, model versioning is also effective in deploying code changes in a controlled way. Model versioning enables developers to:

• Deploy a “pre-release” version in the same environment as the current one.

• Gradually migrate downstream users to the new version.

• Eventually, deprecate (using the “deprecation_date” config) and remove the old model when it’s no longer necessary.

When should you version a model?

Create a new version of a model when you’re making modifications that intentionally break the model’s contract. If you’re making a non-breaking change, you don’t need a new version.

Remember that the process of deprecating an old model and adopting a new version takes time, resources, and collaboration between teams. When you stick to non-breaking changes, you avoid all of that, but at the cost of models with lots of unused columns. Weigh pros and cons and make a judgment call of what’s best for your use case.

You can also think of a schedule of releases where models get updated to the latest version periodically to clean the unused columns.

Model versioning vs. Version control

Version control is linked to branching strategies. Your entire project goes to a separate branch for reviewing by the relevant teams. You can also leverage state to only rebuild changed resources. Changes can be rolled back by reverting to a previous commit.

Model versions live in the same branch and environment. They exist when people or processes outside of your team depend on the models. They enable you to offer a migration path with clear diffs and deprecation dates.

Model versioning vs. New model

What’s the difference between model versioning and duplicating an existing model and naming it “dim_customers_v2”?

Both have similar effects, but model versioning enables you to:

• Reuse model configuration.

• You can tag a versioned model as latest, prerelease, or old, and build models based on this designation, using the “version” method.

• deliver notification of deprecation or newer versions to consumers.

Creating a model version

The first step is to duplicate the relevant model, adjust the code as needed, and save it with a clear suffix (for instance, “_v2”).

Then, you are going to declare this new model as a version of a model, highlighting the changes made to it. You can also give it a full specification, but dbt recommends adding diffs only.

By adding the “v: 2” config, dbt will link it to a file named “dim_customers_v2”. dbt recommends following this convention, but if you need to override this, you can use the config “defined_in: file_name”.

Besides, dbt will automatically assign the “latest_version” to the greatest version number. In the config above, we assigned latest_version to v1, so dbt will automatically read v2 as a prerelease version. When v2 is ready to be deployed, we could change this config to v2 or remove it altogether.

Configuring a model version

Each version can take standard model configurations (materialized, description, contract, etc) as well as the versioning-specific configs below.

• V: denotes the version. It can an integer or a string. dbt sorts versions by ascending numeric or alphabetical order.

• defined in (optional): defines the model name related to this version. This is in case you don’t use the naming convention <model_name>_v<v>.

• alias (optional): a custom name for the table in the database, in case you don’t want to original file name.

• include (optional): lists the columns from the top-level property to include in this version. It can be a list or “all” or “*”.

• exclude (optional): columns to be removed from the top-level property. It can only be declared if “include” is declared.

• latest_version (optional): denotes the latest version which is the model dbt will refer do in an unpinned ref or in the “version:latest” selection method. If not defined, dbt will interpret the latest version as in the V definition.

  

In the example above:

• v1 includes none of the columns from “customers” and includes an id column.

• v2 takes all columns from “customers”, except for customer_country.

• v3 takes all columns from “customers” and puts customer_country back in.

• v4 takes all columns from “customers”

Setting up an alias for the data warehouse

You may want “dim_customers_v1” to continue populating the “dim_customers” table in the warehouse. For that, you can use the “alias” configuration:

Alternatively, you can create a post-hook with a macro that creates or updates dim_customers with the version you are currently building. That way, if someone is querying the table outside of dbt, they will always get the current version.

Running a model version

By selecting the main model name, in our example “dim_customers”, you run all versions of the model.

You can also run a specific version by selecting the specific file name or use the “.v2” suffix to run the model configured as “v: 2”.

Finally, you can use the “version:” method pointing to latest, prerelease, or old to select a specific method.

Optimising model versions

If the new version only makes simple modifications to the code, like removing a column, you can simply create a view and use Jinja to refer to the previous version and make the necessary changes using a select statement.

dbt has native features that help developers define model structure and visibility.

By using these tools, we can set who can see a model, what data it should contain, how it changes over time, and how it can be referenced by other models.

The features available in the open-source version of dbt are:

• Model access: the ability to make models public or private to control who can ref them.

• Model contracts: restrict the shape of a model in terms of column names, data types, and constraints.

• Model namespaces: delineate ownership boundaries and responsibilities through naming structures.

• Model versions: versioning for the models, allowing for adjustments to be made without breaking access for the end user.

We will cover the first 3 features in this post and dedicate a separate post to Model Versions.

Table of contents

• Model Access

• Model Contracts

• Namespaces

• When to adopt model governance features

Model access

We can define model access by using two configurations: access and group.

These can be declared in the dbt_project.yml file, in a properties.yml file, or in the model itself. They are also subject to the config hierarchy (model first, then properties.yml, and finally dbt_project.yml).

Access modifiers

This configuration allows the developer to limit which models can ref an individual model or a group of models.

There are 3 types of model access:

• Protected (default): can be ref’ed by models in the same project (or package).

• Private: can be ref’ed by models in the same group.

• Public: can be ref’ed by any model in any group, package, or project

If you change the access modifier to public, ensure to run a production job to apply the changes.

Things to note:

• Ephemeral models cannot be set to public.

• The access configuration doesn’t impact permissions at database level.

• Models in packages can be ref’ed by any model, regardless of the access config. Package maintainers can set the “restricted-access” config to true so that private and protected models can only be accessed by models within the package.

• dbt recommends setting models in development to private so they don’t get accidentally ref’ed by other models.

Groups

The main role of Groups is to restrict access to models. Every declared group comes with a default protected access configuration, which means these models can only be ref’ed by models in the same project.

Although we are specifically talking about models here, other resources (all except sources and exposures) can be assigned to a group. Each resource can only be part of one group.

Declaring groups

If you want to override the default access modifier, you can set it in the config. For example, by setting the access config to “private”, you are limiting ref’ing to models in the same group.

You can also add a “description” and a “meta” config to your groups:

Finally, another useful feature of groups is the ability to select all models in a group using the “group:” method. For instance, dbt run --select “group:finance”.

Model contracts

When you set up a model contract, dbt ensures that the dataset resulting from a model follows the defined shape, or it will fail to build.

Contracts can be enforced upon SQL models that are materialised as a table or incremental (with “append_new_columns” or “fail” on_schema_change config). Certain adapters may not support it or have constraint restrictions.

dbt recommends setting up contracts for models that feed into visualisation tools for the end user or models that are used by other teams or projects.

How to define a contract

Under a “contract” config, set “enforced” to true. Then, you are required to add every column name and data type, following your adapter’s types.

Finally, if your adapter supports it, you can add the optional config of “constraints”. The types and reinforceability available will depend on the adapter. Some adapters may allow the model to build even if it violates a constraint.

Defining constraints

The possible configurations for constraints are:

• Type (required): can be one of not_null, unique, primary_key, foreign_key, check and custom.

• Expression: free text required for some constraints.

• Name (optional): human-friendly name.

• Columns: list of column names to apply the constraint to.

• Warn_unenforced: set to false to skip warnings for constraints supported but unenforced by the adapter.

• Warn_unsuported: set to false to skip warnings for constraints not supported by the adapter.

Furthermore, the foreign_key constraint type accepts the input of “to:” with a ref or source function and “to_columns:” with the corresponding primary key.

Contracts specificities

Handling breaking changes

When a contract is enforced and a previous state is being used for comparison, dbt will check for changes in the schema that could break downstream models.

This includes:

• removing an existing column

• change the data type of a column

• removing or altering one of the constraints of a column

• removing or renaming a contracted model

The severity of the error will depend on whether the model is versioned. For versioned models, dbt will issue a contract error, and you must add a new version with the changes. For unversioned models, dbt will only issue a warning.

Contracts vs. Data Tests

Contracts are different from Data Tests in that they prevent a model from building if it doesn’t have a specific shape. Data Tests validate the data after the model has been built, and they also offer more flexibility in terms of configuration.

You should consider replacing some Data Tests with Constraints to save on compute costs and ensure the dataset is shaped correctly at build-time.

Contracts for incremental models

Since Contracts are supposed to ensure that the YAML definitions and the schema are the same, they can only be imposed in incremental tables with the on_schema_change config set to “fail” or “append_new_columns”.

In the other two possible configurations, “ignore” and “sync_all_columns”, dbt will either not check for schema changes or actively modify it. This contradicts the idea of a contract and potentially creates breaking changes.

Essentially, if dbt automatically changes the schema, the contract can no longer guarantee that downstream consumers are still compatible.

Model namespaces

Namespaces are meant to create a logical organization or grouping of objects in your project. There are several ways of creating namespaces.

dbt’s file structure

Your subdirectories are automatically namespaces.

For instance, the models under the “staging” folder can have commands applied to them as a group or have a separate schema defined for them.

These folders can be defined based on ownership. Besides, with a custom schema, developers can define who can see what data. Both these steps aid data governance.

Name

When using ref and source functions, you can use two arguments using the name set up in the configuration.

For the ref function, the namespace is optional. dbt recommends specifying it when the project uses models from other projects or packages. This adds clarity and avoids ambiguity, as dbt will first look for the model name in the current project.

When to adopt model governance features

It is important to note that, while adding these features will strengthen the trust and stability of your project, you need to consider that they often require a lot of additional setup and maintainability.

For that reason, ensure your models are ready to benefit from these features before you start adopting them. If your models are still changing significantly, maintaining these features can become tricky.

This week, we focused on deep diving into three more dbt resources: Seeds, Snapshots, and Exposures. We covered configuration, best practices, and the purpose of each. See the study links below.

Also, I shared 3 things I did to keep myself consistent during this hectic October, juggling client work and self-paced learning.

Study notes added:

We covered the remaining 3 resources of a dbt project:

This is a breeze, considering we are covering model governance and Python models next week!

A full list of the study notes is available in the study guide.

Juggling work and self-paced learning

This month, I was inundated with requests from a client. While my bank account appreciated it, my motivation to study dbt definitely took a hit.

Time was scarce, and when I did carve out a moment, I was often too mentally drained to focus. The workload has eased now, but every challenge brings a lesson.

Here are 3 things that helped me stay consistent:

1. Commit to doing at least a little bit every day:

   Busy days fly by, and skipping “just one” can easily snowball. Even on hectic days, I tried to write a short paragraph, read a few pages, or plan the next steps — anything to keep the momentum going.

2. Have a long-term plan in place:

   Since this is a side project, I didn’t have a strict timeline. However, when life got hectic, I felt like having a long-term schedule gave me a sense of direction and helped me stay motivated when life got busy.

3. Remember your purpose:

   I started this dbt study guide almost 3 months ago. After so many posts, it became easy to forget why I began. Reminding myself that this project helps me upskill — and helps others learn too — kept me motivated and inspired.

Have the study notes been useful to you so far? How are you getting on with your self-paced learning journey? I would love to hear from you here or on LinkedIn!

After learning about Seeds and Snapshots, the final resource we will analyse in this Checkpoint is Exposures.

Exposures is a metadata object that defines and describes the downstream use of your project. They are used to represent and document external resources like dashboards, reports, machine learning models, and APIs.

There are 3 main purposes for this:

1. End-to-end visibility: understanding what resources are using your dbt project so you know the impact of altering a model.

2. Documentation: show external assets in the dbt documentation.

3. Governance: help track ownership and accountability for data assets.

How to declare Exposures

Exposures can be defined manually with a .yml file or created automatically for supported integrations.

Declaring an Exposure

Exposures are declared under the “exposures:” key in a .yml file nested under models.

Exposure properties

Required properties:

• name: a unique name for the exposure in snake case (replacing spaces by _)

• type: it can be dashboard, notebook, analysis, ml, or application. For documentation purposes.

• owner: name or email required. Additional properties name or email required; additional properties allowed

Recommended properties:

• depends_on: defines the metric, ref, or source (although it is highly unlikely to have an exposure linked to a source) that the exposure depends on.

Optional properties:

• label: allows for a more human-friendly name than the “name” property, which doesn’t allow spaces.

• url: the link to the data asset used to populate the “View this exposure” button in the upper right corner of the documentation site.

• maturity: it can be “high”, “medium”, or “low” and refers to the level of confidence in the asset.

• enabled: if set to False, dbt does not consider this resource part of the project. Can also be set in the dbt_project.yml file.

• description: sets descriptions for documentation purposes.

• tags: add common tags to resources to apply commands to all in one go using the “tag:” method.

• meta: adds custom metadata for the resource to the manifest.json file and documentation.

Running commands with exposures

As Exposures are metadata objects, you cannot run or build them. To have them included in the documentation, just run “dbt docs generate” and the exposures will appear in the documentation.

Besides, you can use the “exposures:” method to run or test all the upstream models that the Exposure depends on.