Configurations and Properties
Learning key concepts, differentiating configs from properties, and understanding hierarchy in .yml files.
Part of the “Mastering dbt” series. Access to the full Study Guide. Let’s connect on LinkedIn!
Notes from the following dbt documentations: Configs and properties, Resource path, and Plus prefix.
From here onwards, we are going to start getting a bit more technical as we begin to deep-dive into configurations and properties. In this particular Checkpoint, we are going to focus on the setup of two resources: models and sources.
However, before we do that, we need to review key concepts that we are going to start using and understand how hierarchy works in configuration files.
Key definitions
Resources
In the context of a dbt project, a resource is each of the building blocks of the project.
Examples: models, seeds, snapshots, sources, tests, analyses, exposures, metrics, macros.
Properties
Properties are used to describe resources. They are used for documentation, tests, and metadata. They are declared in .yml files.
Configurations
The configurations define HOW dbt builds resources into the data warehouse. They are used to control dbt’s behaviour.
They can be declared in:
properties.yml nested under a config: property.
individual models inside a config( ) macro.
dbt_projects.yml for global defaults.
For example, here we see configurations in the dbt_project.yml file. Here they are controlling how a particular source is treated by dbt.
And here, we can see the config: property being used in a properties.yml file to control how sources and tables are treated:
And finally, here we have a config macro being used inside a model:
If the same configuration is applied in all three places, this is the hierarchy:
Model file overrides properties.yml and dbt_project.yml.
Properties.yml overrides dbt_project.yml.
If a config isn’t specified, dbt uses its default configuration.
Artifacts
Artifacts are JSON files generated after you run or compile your project. They describe the state of the project and execution.
dbt writes these files to the target/ directory after the commands “dbt run”, “dbt compile”, or “dbt docs generate”.
Examples: manifest.json, run_results.json, catalog.json
Resource paths:
You can apply configurations to the entire project, a particular folder, a model, etc. To define that, you can use the resource path below:
When the same configuration is applied to different levels, there is a hierarchy. The most specific config always takes precedence.
The + prefix:
dbt differentiates between a configuration and a folder name by using a + prefix for configs. It also helps disambiguate folder names that have the same name as configs (like a folder called “tags”).
