How and when to use Model Versioning
Use cases for model versioning and specific configs.
Part of the “Mastering dbt” series. Access to the full Study Guide. Let’s connect on LinkedIn!
Notes from the Model Versions documentation.
In this post, we will dissect the definition, config, and best practices for Model Versions. This is part of the Model Governance section of the Study Guide, which also includes a post covering the other 3 features: Access Modifiers, Model Contracts, and Namespaces.
Table of contents
What, why, and when: model versioning
Why version a model?
Model versioning is a way to handle situations where you need to modify a model that’s already being used by downstream models or end-user tools. It serves 2 main purposes.
Handling breaking changes
When you run a model and it fails to build due to a breaking change (caused by a deleted column or changed data type, for example), all downstream models and end-user tools will be affected.
In smaller organisations, this may be an easy problem to solve. Fix the model, adjust downstream models and tools. Bob’s your uncle.
However, in larger organisations, a breaking change may affect tons of models and tools across different teams. Instead of making everyone react immediately, you can start working on a new version of the model while keeping the old version running without any breakage. This allows teams to migrate at their own pace without disrupting production workflows.
Controlled change management
Besides handling breaking changes, model versioning is also effective in deploying code changes in a controlled way. Model versioning enables developers to:
Deploy a “pre-release” version in the same environment as the current one.
Gradually migrate downstream users to the new version.
Eventually, deprecate (using the “deprecation_date” config) and remove the old model when it’s no longer necessary.
When should you version a model?
Create a new version of a model when you’re making modifications that intentionally break the model’s contract. If you’re making a non-breaking change, you don’t need a new version.
Remember that the process of deprecating an old model and adopting a new version takes time, resources, and collaboration between teams. When you stick to non-breaking changes, you avoid all of that, but at the cost of models with lots of unused columns. Weigh pros and cons and make a judgment call of what’s best for your use case.
You can also think of a schedule of releases where models get updated to the latest version periodically to clean the unused columns.
Model versioning vs. Version control
Version control is linked to branching strategies. Your entire project goes to a separate branch for reviewing by the relevant teams. You can also leverage state to only rebuild changed resources. Changes can be rolled back by reverting to a previous commit.
Model versions live in the same branch and environment. They exist when people or processes outside of your team depend on the models. They enable you to offer a migration path with clear diffs and deprecation dates.
Model versioning vs. New model
What’s the difference between model versioning and duplicating an existing model and naming it “dim_customers_v2”?
Both have similar effects, but model versioning enables you to:
Reuse model configuration.
You can tag a versioned model as latest, prerelease, or old, and build models based on this designation, using the “version” method.
deliver notification of deprecation or newer versions to consumers.
Creating a model version
The first step is to duplicate the relevant model, adjust the code as needed, and save it with a clear suffix (for instance, “_v2”).
Then, you are going to declare this new model as a version of a model, highlighting the changes made to it. You can also give it a full specification, but dbt recommends adding diffs only.
By adding the “v: 2” config, dbt will link it to a file named “dim_customers_v2”. dbt recommends following this convention, but if you need to override this, you can use the config “defined_in: file_name”.
Besides, dbt will automatically assign the “latest_version” to the greatest version number. In the config above, we assigned latest_version to v1, so dbt will automatically read v2 as a prerelease version. When v2 is ready to be deployed, we could change this config to v2 or remove it altogether.
Configuring a model version
Each version can take standard model configurations (materialized, description, contract, etc) as well as the versioning-specific configs below.
V: denotes the version. It can an integer or a string. dbt sorts versions by ascending numeric or alphabetical order.
defined in (optional): defines the model name related to this version. This is in case you don’t use the naming convention <model_name>_v<v>.
alias (optional): a custom name for the table in the database, in case you don’t want to original file name.
include (optional): lists the columns from the top-level property to include in this version. It can be a list or “all” or “*”.
exclude (optional): columns to be removed from the top-level property. It can only be declared if “include” is declared.
latest_version (optional): denotes the latest version which is the model dbt will refer do in an unpinned ref or in the “version:latest” selection method. If not defined, dbt will interpret the latest version as in the V definition.
Source: versions documentation
In the example above:
v1 includes none of the columns from “customers” and includes an id column.
v2 takes all columns from “customers”, except for customer_country.
v3 takes all columns from “customers” and puts customer_country back in.
v4 takes all columns from “customers”
Setting up an alias for the data warehouse
You may want “dim_customers_v1” to continue populating the “dim_customers” table in the warehouse. For that, you can use the “alias” configuration:
Alternatively, you can create a post-hook with a macro that creates or updates dim_customers with the version you are currently building. That way, if someone is querying the table outside of dbt, they will always get the current version.
Running a model version
By selecting the main model name, in our example “dim_customers”, you run all versions of the model.
You can also run a specific version by selecting the specific file name or use the “.v2” suffix to run the model configured as “v: 2”.
Finally, you can use the “version:” method pointing to latest, prerelease, or old to select a specific method.
Optimising model versions
If the new version only makes simple modifications to the code, like removing a column, you can simply create a view and use Jinja to refer to the previous version and make the necessary changes using a select statement.
