Setting up and using source freshness
Understanding source freshness along with its configuration and command.
Part of the “Mastering dbt” series. Access to the full Study Guide. Let’s connect on LinkedIn!
Notes from this documentation.
Using source freshness
The “freshness” command is useful to determine whether your Sources are up to date. You set up a benchmark for what is considered to be fresh data in your use case and dbt will throw a warning or an error, depending on the configuration, if your table is stale.
It can also be used to analyse freshness over time. You can run this command on a schedule in deployment, storing the results and setting up SLAs to ensure the data is being updated as expected. We’ll cover this in detail in Checkpoint 5.
Declaring source freshness
Source freshness can be declared in two places: 1) in your sources.yml file with a “freshness:” config either at the source or table level or 2) in your Job settings. The latter will be covered when we get to deployment in future Checkpoints.
In your sources.yml file, the declaration would look like this:
The “loaded_at_field” field is required and should point to the column in your data that represents a timestamp for each row. For example, if you are tracking clickstreams, the “loaded_at_field” field would be a “click_date” column.
You also need to set up a “warn_after” and “error_after” field. These are timeframes of how old you want the data to be for dbt to throw a warning or an error. In the example above, if the most recent record was older than 24 hours, dbt would throw an error and your run would be interrupted. If it was older than 12 hours, the run would occur, but with a freshness warning.
There is also a “filter:” parameter, which allows you to filter the rows that dbt will need to scan to determine source freshness. This is useful when a full scan of the table would be costly.
Excluding a table from a freshness snapshot
The freshness config obeys the general rule of precedence in dbt: the more specific the config, the more precedence it will have.
Therefore, if you want to apply freshness to all your sources except one table, you can add the “freshness:” config at source level and add a “freshness: null” to the specific table you want to exclude.
The source freshness command
The “dbt source freshness” command checks the freshness of all your sources. You can also select specific sources using the “source:” method (e.g dbt source freshness --select "source:source_name").
You can also choose to build only the models whose upstream sources are fresh, using the command: dbt build --select source_status:fresher+. Be mindful that models depending on stale sources won’t update if you use this command.
When the source freshness command is completed, a JSON file containing information about the freshness of the sources is saved to “target/sources.json. This is what it could look like:
You can override the destination of this file using the - -output (or -o) flag. Example: dbt source freshness --output target/source_freshness.json