Skip to main content

Introduction

Runners have a version number in the form major.minor.patch. Runners are backwards compatible within a major release.

Each runner image is tagged with the version, e.g. dataopslive/transform-runner:4.17.8. For each major release, there are also two tags which move with releases. These are x-latest and x-stable.

The -stable tag (for example, 4-stable) will point to the latest stable release of the runners. This is the recommended tag to use within projects, as it will track the versions released after they have been through the QA process.

The -latest tag (for example, 4-latest) will point to the latest development release of the runners. This release may have new functionality, but has not yet been through the full QA release cycle.

Current version

The current major release tag is 4-stable

Upgrading between releases

Major releases may not be backwards compatible. The changes required to move a project between major versions are listed below.

Upgrade from 4-stable to 5-stable#

Project changes#

Snowflake is now managed by SOLE. The project should be updated to use a SOLE configuration, and the jobs for "clone, mask and grant" should be removed. The template project should be used as a guide for project changes.

Modelling and Transformation changes#

5-stable introduces dbt version 1.0.1, which is an upgrade from dbt version 0.17 in 4-stable. There are a number of changes required in dataops/modelling/dbt_project.yml

  • 'config-version': 2 should be added to the top of the yml file.

  • The 'source-paths' key should be renamed to 'model-paths'

  • The 'data-paths' key should be renamed to 'seed-paths'

Additionally, the name of the directory that dbt packages into which packages are installed has changed from 'dbt_modules' to 'dbt_packages'. This will only affect projects that are manually installing new dbt packages.

Tests have been redesigned, and now return failing rows. Custom test SQL will need updating.

Last updated on