The dbt_project.yml file
Dissecting required and optional configurations for the dbt_project.yml file
Part of the “Mastering dbt” series. Access to the full Study Guide. Let’s connect on LinkedIn!
Notes from this dbt documentation.
One of the first steps when starting a project on dbt is to create your dbt_project.yml file. In the 1st Checkpoint of our Practice Project, we added basic configurations. However, in this post, we are going to dive deeper into all the configs available.
Bear in mind, this is a very technical post. I categorised the configs to make them more easily digestible, but it’s essential to play with these configs in your Practice Project.
Update 22/9: a previous version of this post had a section about Flags. These are being moved to a separate post so we can fully cover them with all their complexity.
Table of contents
Defining the dbt_project.yml file
The dbt_project.yml file is a required file for all projects. It contains information on how dbt should operate the project.
It serves 2 main purposes:
Declares the name, version, and structure of the dbt project.
Provides global and scoped configurations to the project’s resources.
By default, dbt looks for the dbt_project.yml in the main project folder. A different directory can be set by using the - -project-dir flag or the DBT_PROJECT_DIR environment variable.
Basic configurations
Name
The name of a dbt project. Must be letters, digits, and underscores only, and cannot start with a digit. If your project name contains invalid characters, you will end up with the “Invalid project name” error:
Config-version
Sets the dbt syntax structure for your project. The v2 structure is the current one, so if you don’t specify this, dbt will assume you’re using v2.
Version
Within the dbt_project.yml file, this config defines the version of your dbt project. This is an optional parameter and defaults to None if not used.
Please note that, within the dbt_project.yml file, this is not the dbt version but rather a versioning system for your own reference. However, when this config is used in other .yml files, it refers to dbt versions.
There is also the versions property for models, which is a separate feature we will cover later.
Profile
If you’re using dbt Core, the free open-source, command-line version of dbt, you will need to set up a profiles.yml file. This file replaces the part of the dbt Cloud UI that establishes the connection to the data warehouse.
When you run dbt Core from the command line, it will look at the profile set on the dbt_project.yml file and find it in the profiles.yml file. It will then connect to the warehouse using the credentials under the profile according to the environment.
Path customisation configurations
When we use commands, dbt looks for resources in the folders specified in the dbt_project.yml file.
The default directories are:
We can change these specified directories, following some rules:
The directory path must be relative to the location of the dbt_project.yml file. Direct paths should be avoided.
Seeds and Models can be co-located because they are of different formats (seeds are .csv and models are .sql)
However, macros and models cannot be co-located because they are both .sql files.
Resources can be scanned for in multiple folders, but this is not recommended.
Additional path customisations
target-path
The target path defines where the JSON artifacts and SQL files will be written. By default, they go to a “target” directory, but this can be customised using this config.
clean-targets
In dbt, there is a “clean” command that deletes the paths specified under this config. By default, the target and dbt_packages directories are deleted when this command is invoked, but this can be customised.
If this config is not included, the “clean” command will remove files in the target directory.
asset-targets
The directories specified here will be copied into the target directory when the “docs generate” command is invoked. This is useful for including images in your descriptions. By default, this directory is empty.
packages-install-path
Specifies a custom directory for the packages to be installed when the “dbt deps” command is invoked. By default, packages are installed in the dbt_packages directory.
Resource-specific and test configurations
You can also add global configurations to each specific resource and data tests. We will cover these by resource, starting with Models and Sources in this Checkpoint and expanding to the others in future Checkpoints.
Please note that these configurations can also be defined in the specific .yml files and even in models, following the hierarchy defined in the Configs and Properties study notes.
Variables
dbt offers the ability to define variables that can be invoked by the {{ var('...') }} function in a model or macro, for example.
The context can be for all resources, a specific project, or a package.
Variables defined with the - -vars command line argument override variables defined in the dbt_project.yml file, following the usual hierarchy for configs and properties.
We will cover variables in more detail in future Checkpoints.
Other optional configurations
exposures:
Enables or disables the usage of exposures in your project. Default configuration is enabled. This feature will be covered in more detail in future Checkpoints.
query-comment:
As a default, dbt inserts a JSON comment in each query with metadata like dbt version, profile, target names, and node ID.
With this configuration, you can add a string of your choice. The string gets added to the beginning of the query (except for Snowflake, which adds it to the end).
If set to null, the default comments are disabled.
There are 3 parameters for this config:
comment: the string to be added to the query.
append: if switched to True, the comment is added to the bottom of the query, instead of the top.
(For BigQuery) job-label: if set to True, adds the comment as job labels to the query.
You can also refer to a macro or variables.
required-dbt-version:
Restricts your project to only work with a range of dbt versions. If the current version falls out of the range specified, dbt sends an error message. However, if you opted to receive ongoing dbt version upgrades, dbt will ignore this config.
dbt-cloud:
If you’re setting up dbt CLI, you will need to enter this config with the project ID into your dbt_project.yml file.
quoting:
Defines whether dbt will wrap database, schema, and/or identifier names in quotes in the SQL script. Quoting allows you to use reserved words and special characters, but this should be avoided.
For Snowflake, it should be set to false to avoid quoting model and column names unnecessarily. For other warehouses, leave the default values.
on-run-start & on-run-end:
A SQL statement to be run at the start or end of the following commands: dbt build, dbt compile, dbt docs generate, dbt run, dbt seed, dbt snapshot, and dbt test. It can also call macros that return SQL statements.
dispatch:
By default, dbt searches for macros first in your root project and then in the package specified in the macro’s namespace. For example, if you created your own version of a dbt_utils macro and you want dbt to prioritise using your version:
