Introduction

Snowflake Object Lifecycle Engine

The Declarative way to drive your entire Snowflake estate

The Snowflake Object Lifecycle Engine manages the life-cycle of Snowflake Objects defined in a configuration specified by the user. The full list of supported objects are listed at Supported Snowflake Objects

The User can define configuration of Snowflake objects in a YAML file, which is human-readable format. The engine compiles the configuration to work out dependencies and resolves environment and namespace to effectively manage the specified objects.

This operation can be divided into 2 distinct actions:

• Compilation
• Execution

The orchestration of the actions is controlled by the dataops-snowflakeobjectlifecycle-runner.
The runner performs the specified action on the specified resource group.

Compilation

This action handles reading the user-specified configuration describing their Snowflake Infrastructure and generates machine-readable configuration that can be executed to enact those specifications.

This means collecting all configuration-files, namespacing objects as per environment, and resolving dependencies within objects.
Upon successful compilation, resource definition and import files are generated and saved in resource group specific folders.

Execution steps can now be triggered to manage the Snowflake Objects.

Execution

Inspecting Snowflake to get the state of Objects, and applying changes if required is categorized as Execution phase of the Snowflake Object Lifecycle Engine.

Inspection collects the current state of specified Snowflake Object and saves it into local state.
This local state is then used to plan out what changes are required to get the Object to match the user-configuration.

Upon applying, the Snowflake Object Lifecycle Engine enacts the plan in Snowflake Infrastructure.

The following actions are part of Execution phase:

• VALIDATE
• PLAN
• APPLY
• PLAN-DESTROY
• DESTROY

Resource Groups

The Snowflake Object Lifecycle Engine divides Snowflake Objects into 4 resource groups.
The execute action is performed on each resource group in sequential order.

The resource groups are:

• ACCOUNT_LEVEL
• DATABASE
• DATABASE_LEVEL
• GRANT

ACCOUNT_LEVEL

This resource group refers to all Account-Level Snowflake Objects supported by Snowflake Object Lifecycle Engine.
These objects are:

• Resource Monitor
• Role
• Share
• User
• Warehouse

The generated resource definition and import statements files are stored in a directory named account_level inside the cache directory.
This resource group should run at initial stages of pipeline in-parallel with databases.

Setting the parameter LIFECYCLE_MANAGE_OBJECT to ACCOUNT_LEVEL commands the engine to execute actions such as PLAN, APPLY etc for Account-Level objects.

DATABASE

This resource group is used just for management of databases.

The generated resource definition and import statements files are stored in a directory named database inside the cache directory.
This resource group should run at initial stages of pipeline in-parallel with account_level.

Setting the parameter LIFECYCLE_MANAGE_OBJECT to DATABASE commands the engine to execute actions such as PLAN, APPLY etc for Databases.

DATABASE_LEVEL

This resource group refers to all Database-Level Snowflake Objects supported by Snowflake Object Lifecycle Engine.
These objects are:

• External table
• File Format
• Masking Policy
• Pipe
• Schema
• Stage
• Stream
• Table
• Tag
• Task
• View

The generated resource definition and import statements files are stored in a directory named database_level inside the cache directory.
This resource group should run after the execution of APPLY for both databases and account_level objects.
This ensures that all referenced Account-Level objects are already created and Database-Level objects created in-directly(result of cloned database) are managed effectively.

Setting the parameter LIFECYCLE_MANAGE_OBJECT to DATABASE_LEVEL commands the engine to execute actions such as PLAN, APPLY etc for Database-Level objects.

GRANT

This resource group refers to all Privileges Grants(and Revokes) supported by Snowflake Object Lifecycle Engine.
Privilege Grant is supported for following objects:

• Account
• Database
• External Table
• File Format
• Masking Policy
• Resource Monitor
• Role
• Schema
• Stage
• Stream
• Table
• View
• Warehouse

The generated resource definition and import statements files are stored in a directory named grants inside the cache directory.
This resource group should run after execution of APPLY for account-level, database and database-level.

Setting the parameter LIFECYCLE_MANAGE_OBJECT to GRANT commands the engine to execute actions such as PLAN, APPLY etc for Grants.

Supported Parameters

Parameter	Required or Optional (and default)	Description
LIFECYCLE_ACTION	REQUIRED	Must be one of COMPILE, VALIDATE, PLAN, APPLY, PLAN-DESTROY, DESTROY, AGGREGATE, AGGREGATE-DESTROY
DATAOPS_SOLE_ACCOUNT	REQUIRED	Snowflake Account(If region-less or organization name is not used, region should be added as well in this variable).
DATAOPS_SOLE_USERNAME	REQUIRED	Snowflake Username
DATAOPS_SOLE_PASSWORD	REQUIRED	Snowflake Password
DATAOPS_SOLE_ROLE	REQUIRED	Snowflake Role with which to run the queries
DATAOPS_DATABASE	REQUIRED	Name of the default database. Required if default database is specified in configuration in format {{env.DATAOPS_DATABASE}}
DATAOPS_PREFIX	REQUIRED	Prefix to be added to Account-Level and Databases(Except default database)
DATAOPS_ENV_NAME	REQUIRED	Suffix to be added to Account-Level and Databases(Except default database)
CONFIGURATION_DIR	OPTIONAL	Path of directory in which the configuration files are present. required if LIFECYCLE_ACTION is set to COMPILE
DATAOPS_NONDB_ENV_NAME	OPTIONAL	Suffix override value. If this specified, this overrides all branch specific suffix to specified value
DATAOPS_DATABASE_MASTER	OPTIONAL	Name of the PROD or master database. Required if master database is referenced in configuration in format {{env.DATAOPS_DATABASE_MASTER}}
LIFECYCLE_MANAGE_OBJECT	OPTIONAL	Resource Group to be managed. Required if LIFECYCLE_ACTION is one of PLAN, APPLY, PLAN-DESTROY, DESTROY. Must be one of ACCOUNT_LEVEL, DATABASE, DATABASE_LEVEL, GRANT
LIFECYCLE_RESOURCE_FILE	OPTIONAL - Defaults to resources.tf.json	Name of the file which contains resource definitions
LIFECYCLE_IMPORT_FILE	OPTIONAL - Defaults to resource_imports.json	Name of the file which contains resource import statements
PROVIDER_DIR	OPTIONAL - Defaults to /snowflake-provider	Directory path of provider configuration. Must contain file with name provider.tf in the directory which has list of providers
DISABLE_PERSISTENT_CACHE	OPTIONAL	If set, disables usage of persistent cache usage by the runner. The generated files transfer between jobs must be handled by user in such case
ARTIFACT_DIRECTORY	OPTIONAL - Defaults to CI_PROJECT_DIR/snowflake-provider, where CI_PROJECT_DIR is the path of repository	Path to where the artifacts such as resource file, import file and state files are uploaded to. The directory must be set to upload as artifact in CI job as well
LIFECYCLE_STATE_RESET	OPTIONAL	If set, removed the local state file and re-initializes all defined and managed objects
USE_SOLE_GRANT_MANAGEMENT	OPTIONAL	If set, uses SOLE Grant Management for managing the privlige/role grant and revokes rather than Terraform Provider
DATAOPS_DEBUG	OPTIONAL	If set, enables debug logs for the Engine. Note that this will print potentially sensitive information, such as credentials, in the job logs
DATAOPS_SOLE_DEBUG	OPTIONAL	If set, enables debug logs for SOLE Grant Manamegent

User Configuration

The Snowflake Object Lifecycle Engine supports defining multiple files to read for Snowflake Object configuration.
Users can specify their Snowflake configuration in Human Readable format such as YAML in a declarative format, which the runner utilizes.

The runner support the following File Formats for Snowflake Object Definition:

• YAML

The configuration files has the following reserved keywords under which the Snowflake Object definitions must be defined:

Keyword	Description	Type
account	This keyword is used to define the configuration of Account.	Map/Dictionary
databases	This keyword is used to define the configuration of Databases.	Map/Dictionary
resource_monitors	This keyword is used to define the configuration of Resource Monitors.	Map/Dictionary
roles	This keyword is used to define the configuration of Roles.	Map/Dictionary
shares	This keyword is used to define the configuration of Shares.	Map/Dictionary
users	This keyword is used to define the configuration of Users.	Map/Dictionary
warehouses	This keyword is used to define the configuration of Warehouses.	Map/Dictionary
include	This keyword is used to define list of files to be collected by the runner	List

Snowflake Object names specified in configuration are case-insensitive.
warehouse_1 is converted to WAREHOUSE_1.
If such objects are referenced as parameters in another object, the name has to be present in upper-case. Otherwise the engine would assume that as a non-managed object.

Configuration of objects in a file

Snowflake Object Definitions can be defined in a single file.
All Snowflake Object defined in the file is parsed by the runner and used to manage their life-cycle.

Configuration of objects in multiple files

The runner supports splitting the Snowflake Object definitions into multiple files and specifying all the files to be used to the runner.
The runner collects Object definition from all the files and parses it.

info

If multiple files contain definition for same Snowflake Object, then the runner only uses configuration collected from the last file, i.e. the definition is overridden.

Configuration of objects in a nested files

Similar to DataOps CI files, the runner supports including paths to other configuration files by using the keyword include and passing a list of files in it.

info

If multiple files contain definition for same Snowflake Object, then the runner only uses configuration collected from the last file, i.e. the definition is overridden.

Configuration files in a directory or directories

Snowflake Object definition files can be saved in the directory.
The engine requires the path of the directory in which the configuration files are present.

info

If multiple files contain definition for same Snowflake Object, then the runner only uses configuration collected from the last file, i.e. the definition is overridden.

Access Management

Users can define privilege grants to roles in the configuration of the Snowflake Object.
Using this the engine handles the generation of all grants and revoked on the object.

As the Snowflake Object Lifecycle Engine follows a declarative method for defining the Snowflake Objects, the engine updates the Objects to match the configuration.
This means that only the specified privileges are granted to specified roles. Revokes are added implicitly for all un-specified privileges.

info

The implicit revoke only initiates when the keyword grants is used in the configuration of the object.
If the keyword is not specified, then the grants and revokes of privileges is not managed.

Example

In the below example, MODIFY to role role_1 is specified in grants section in the configuration of database_1, while no grants section is defined for database_2.

DATABASE_1:  comment: DATABASE 1  grants:    MODIFY:      - ROLE_3DATABASE_2:  comment: "DATABASE 2"

Using this configuration, the runner would generate revokes on privileges MONITOR, USAGE, CREATE SCHEMA, IMPORTED PRIVILEGES for the database DATABASE_1.
As no grants section is defined in the configuration of DATABASE_2, revokes for this database is not handled.

Adding Revoke to all Privileges

To manage revokes on all the privileges of a Snowflake Object, an empty grants can be specified in the configuration of the said Object.

DATABASE_3:  grants:  comment: "DATABASE 3"

Implicit Grant of USAGE on Parent Object

If a privilege is granted to a role on a Database Object(Schema, Table, View, etc), but USAGE is not granted on parent objects, then the role would not be able to access the Snowflake Object.
In such case scenarios, USAGE on the parent would have to be granted to roles as well.

To reduce the number of configurations to be managed, the runner can implicitly grant USAGE on parent (and grand-parent) objects to such roles.

DATABASE_5:  comment: "test_comment_2"  grants:    MODIFY:      - ROLE_1    monitor:      - ROLE_1      - ROLE_2    schemas:      SCHEMA_1:        comment: Hello        grants:          MODIFY:            - ROLE_1        tables:          TABLE_1:            grants:              select:                - ROLE_1                - ROLE_2                - SYSADMIN              insert:                - ROLE_2            comment: Hello_1            COLUMNS:              column_1:                type: VARCHAR(16777216)

In the above example, on the table TABLE_1, SELECT is granted to roles ROLE_1, ROLE_2 and SYSADMIN, and INSERT is granted to ROLE_2.
As USAGE is not granted to none of the roles in both the schema SCHEMA_1 and the database DATABASE_5, the roles would not be able to access the table TABLE_1.

In this case, the runner would implicitly grant USAGE to roles ROLE_1, ROLE_2 and SYSADMIN on both SCHEMA_1 and DATABASE_5.

Manage Mode

The Snowflake Object Lifecycle Engine supports defining the manage-mode of the specified object.
Configuring manage-mode allows users to specify what properties of the objects are to be managed by the engine and what is to be ignored.

This is essential if only the grants of an object are to be managed, or a child object(Table, Pipe) is managed but its parent(Schema) is not.

Example

databases:  DATABASE_1:    manage_mode: all    comment: "Test Database 1"    grants:      USAGE:        - ROLE_1        - SYSADMIN  DATABASE_2:    manage_mode: grants    comment: "Test Database 2"      grants:      USAGE:        - ROLE_1        - SYSADMIN  DATABASE_3:    comment: "Test Database 3"    manage_mode: none    schemas:      SCHEMA_1:        comment: "Test Schema"

With this configuration,

• for DATABASE_1, all specified properties would be managed. That means both the comment and defined grants.
  
• for DATABASE_2, only the grants would be managed. That means only defined grants. Creation and update to match database parameters is ignored.
  
• for DATABASE_3, none of the parameters would be managed. That means neither the comment nor defined grants. The schema SCHEMA_1 defined inside the database, would be managed as another object.
  

The engine supports the following manage modes:

• all
• grants
• none

All

When all the parameters of the object have to be managed, the setting manage_mode to all would inform the engine that the resource as well as grant definitions has to be generated for the object.

As this is all is the default value for manage_mode, it does not have to be explicitly defined.

Grants

If only the defined grants have to be managed for the object, then setting manage_mode to grants would inform the engine that only grant definitions have to be generated for the object.

None

For database-level objects, such as Schema, Table, Stage, etc, their parent object has to be defined in the configuration.
Schemas have to be defined inside a database.
Tables, Stages, Masking Policies, etc have to be defined inside a schema.

In case that only a child object has to be managed, setting manage_mode to none for the parent object would inform the engine that neither the resource nor the grant definition has to be generated for it.
The definitions would only be generated for child objects.

info

The manage_mode has no option to only manage parameters of the objects and ignore the grants, as this can be done by not defining the grants in the configuration.
See Access Management section for more information.

State Management

Import, Creation and Update save the current state of Snowflake Object into local state.
This local state is saved as a persistent cache by the runner of the host system.

This allows the Engine to use the existing state of the objects for subsequent pipelines.

Each environment/branch has its own set of states.
Furthermore, each resource group has its own version of state.

The state file is present on the host system at the following path:
/<cache_directory>/persistent_cache/<dataops_project_name>/<branch_name>/snowflakelifecycle-runner/<resource_group>/<resource_group>.tfstate

The value in <> refers to:

• cache_directory: Path in the host system where the cache from the runners is saved
  • Default is agent_cache.
  • To get value for your runner see the volume mounts in the srv/<agent_name>/config/config.toml
• dataops_project_name: Name of the project in lower-case
• branch_name: Name of the branch in lower-case
• resource_group: Name of the resource group

State is not maintained for Grants

State Reset

If state file is corrupted or does not match Snowflake, failures can occur in pipelines.

In such scenarios, the local state file can be reset in order to perform fresh inspection of Snowflake.
Doing so deletes the existing local state for the specified resource-group and imports all managed Snowflake Objects.

The reset can be triggered by environment variable LIFECYCLE_STATE_RESET

The state reset can be configured either at Project Level(in pipelines/includes/config/variables.yml file) to reset state of all resource-groups or at individual resource-level(as a variable in the Plan Job for the resource).

info

Individual Resource-Group State reset is not supported in Aggregate Jobs

In-Place Changes

Snowflake uses aliases for data types such as TEXT for VARCHAR(16777216).
Using such aliases in the configuration leads to logging of in-place changes by the runner.
To avoid such in-place changes, the exact data-types should be used in configuration as well.

For Resource Monitor, Snowflake Docs suggest that timestamp can be used for parameter start_timestamp.
But using a timestamp leads to error when creating or updating a resource monitor.
Currently resource monitor only supports date as valid value at the time of creation, but upon successful operation, it is converted to timestamp.
This leads to the engine marking the resource monitor to have an in-place change to replace timestamp with the same date.

Namespace and Environment Management

The Snowflake Object Lifecycle Engine allows control over namespacing and environment for all account-level objects.

Environment Management controls in what environments are objects managed.

Namespace controls the addition of Prefix and Suffix to Account-Level and Databases(except default Database).

Environment Management

Environment Management control can be applied for Account-level objects.
This allows to configure under which environment an object is managed by the engine.

This can be set by adding environment in the properties section of an account-level object and setting it to name of environment name.

Example

roles:  ADMIN:    environment: PROD  QA:    environment: QAdatabases:  DB:    environment: DEV

With this configuration,

• Role ADMIN, is only managed when the environment is PROD(master branch),
• Role QA, is only managed when the environment is QA(qa branch),
• Database DB, is only managed when the environment is DEV(dev branch)

If environment parameter is not specified, the object is created in all environments.

Namespace Management

Specifying namespace controls whether prefix and suffix are added to Account-Level objects and Databases(except default Database).

This can be set by adding namespacing in the properties section of an account-level object and setting it to one of:

• none
  • Neither prefix nor suffix is added to name of Snowflake object
• prefix
  • Only prefix is added to name of Snowflake object
  • Value of environment variable DATAOPS_PREFIX used as prefix
• suffix
  • Only suffix is added to name of Snowflake object
  • Value of environment variable DATAOPS_ENV_NAME used as suffix
  • If environment variable DATAOPS_NONDB_ENV_NAME is set, its value is used as suffix instead
• both
  • Both prefix and suffix is added to name of Snowflake object
  • If namespacing property is not defined, this is used as default

Example

roles:  ADMIN:    namespacing: none  QA:    namespacing: suffixdatabases:  DB_1:    namespacing: both  DB_2:    namespacing: prefix

With DATAOPS_PREFIX set as DATAOPS and DATAOPS_ENV_NAME set as PROD, the generated configuration is,

• Role ADMIN, remains as it is: ADMIN
• Role QA, is converted to QA_PROD,
• Database DB_1, is converted to DATAOPS_DB_1_PROD,
• Database DB_2, is converted to DATAOPS_DB_2

Database-Level Object Reference

As the engine operates on multiple databases and schemas as part of Lifecycle Management, it doesn't set a database or schema as a session when connecting to Snowflake.
Due to this, referencing Database-Level objects requires specifying complete path/namespace to the object.

If the referenced Database is managed and environment controlled, the environment-based namespace is resolved by the engine when referencing the object.

Such Database-Level references could be provided in one of three ways:

• Specifying just the Object Name

  • This can be used when referenced object belongs in the same schema and database

  databases:  DB_1:    schemas:      SCHEMA_1:        pipes:          PIPE_1:            copy_statement:              into:                table: "TABLE_1"              from:                stage: "STAGE_1"

  If Prefix is set to DATAOPS and Suffix is set to PROD:

  • The fully qualified name for table would be DATAOPS_DB_1_PROD.SCHEMA_1.TABLE_1
  • The fully qualified name for stage would be DATAOPS_DB_1_PROD.SCHEMA_1.STAGE_1
    
    

• Specifying Object name as well as Schema name

  • This can be used when a referenced object belongs in a different schema but same database

  databases:  DB_1:    schemas:      SCHEMA_1:        pipes:          PIPE_1:            copy_statement:              into:                schema: SCHEMA_2                table: "TABLE_1"              from:                schema: SCHEMA_3                stage: STAGE_1 

  If Prefix is set to DATAOPS and Suffix is set to PROD:

  • The fully qualified name for table would be DATAOPS_DB_1_PROD.SCHEMA_2.TABLE_1
  • The fully qualified name for stage would be DATAOPS_DB_1_PROD.SCHEMA_3.STAGE_1
    
    

• Specifying Object, Schema and Database name

  • This can be used when a referenced object belongs in the different database.

  databases:  DB_1:    schemas:      SCHEMA_1:        pipes:          PIPE_1:            copy_statement:              into:                database: DB_2                schema: SCHEMA_2                table: "TABLE_1"              from:                database: DB_3                schema: SCHEMA_3                stage: STAGE_1 

  Assuming that DB2 is managed while DB_3 is not. If Prefix is set to _DATAOPS and Suffix is set to PROD:

  • The fully qualified name for table would be DATAOPS_DB_2_PROD.SCHEMA_2.TABLE_1
  • The fully qualified name for stage would be DB_3.SCHEMA_3.STAGE_1

Even if the referenced object belongs to the same database and schema, explicit names can be provided as well.

Connection Parameters

To establish connection to Snowflake account SOLE required the following parameters:

Account

This is the name of the account in which SOLE operates.

This value can be is set in variable DATAOPS_SOLE_ACCOUNT.
If the value is not present in environment variables, SOLE tries to set the value from vault path SNOWFLAKE.SOLE.ACCOUNT.

Value to the variable can also be set using the DATAOPS_VAULT functionality of the runners, if users do not wish to expose credentials in configuration.

Full account value(along with region eg: eu-west-2) has to be defined in DATAOPS_SOLE_ACCOUNT.
Organizational short names and region-less account names can be used in DATAOPS_SOLE_ACCOUNT as well.

To get the region less account name for Snowflake account the following query can be executed:

select system$whitelist();

This returns a JSON result. This JSON has a key SNOWFLAKE_DEPLOYMENT_REGIONLESS which has value in format abcdef-qwerty.snowflakecomputing.com.
In the value the abcdef-qwerty part can be specified DATAOPS_SOLE_ACCOUNT.

Username

This is the username with which SOLE connects to the account specified in DATAOPS_SOLE_ACCOUNT.

This value can be is set in variable DATAOPS_SOLE_USERNAME.
If the value is not present in environment variables, SOLE tries to set the value from vault path SNOWFLAKE.SOLE.USERNAME.

Value to the variable can also be set using the DATAOPS_VAULT functionality of the runners, if users do not wish to expose credentials in configuration.

Password

Password of the username specified in DATAOPS_SOLE_USERNAME.

This value can be is set in variable DATAOPS_SOLE_PASSWORD.
If the value is not present in environment variables, SOLE tries to set the value from vault path SNOWFLAKE.SOLE.PASSWORD.

Value to the variable can also be set using the DATAOPS_VAULT functionality of the runners, if users do not wish to expose credentials in configuration.

Role

The Role of the Snowflake User defined in DATAOPS_SOLE_USERNAME, which runs the SOLE queries.
This value has to be defined even if the role is the default role for the Snowflake User.

This value can be is set in variable DATAOPS_SOLE_ROLE.
If the value is not present in environment variables, SOLE tries to set the value from vault path SNOWFLAKE.SOLE.ROLE.

Value to the variable can also be set using the DATAOPS_VAULT functionality of the runners, if users do not wish to expose credentials in configuration.

info

As SOLE utilizes terraform, the credentials/variables are converted to terraform variables(_TF_VAR__ appended at front of each variables).
The variable DATAOPS_SOLE_ACCOUNT is duplicated with name TF_VAR_DATAOPS_SOLE_ACCOUNT and similarly for other credentials.
If any value already exists to such variables, it would be overridden

SOLE Grant Management

For Privilege/Role Grant Management, SOLE provides an alternative to using Terraform.
The SOLE Grant Management utilizes same configuration as Terraform Snowflake Provider to inspect, compare and execute operations.

The SOLE Grant Management can be used by setting value to environment variable USE_SOLE_GRANT_MANAGEMENT. When value to this variable is set, SOLE would shift to using SOLE Grant Management instead.

info

SOLE Grant Management requires that a Default Warehouse must be assigned to Snowflake User used to connect to Snowflake. To set a default warehouse for the User, the following query can be run in Snowflake: ALTER USER <SOLE USER> SET DEFAULT_WAREHOUSE = <WAREHOUSE NAME>

SOLE Grant Management performs an inspection of Snowflake at the time of running Plan.
It compares grants defined in configuration with Snowflake and works out what Privilege/Role Grant is to be added or revoked.
Due to the run-time inspection, no local state is required.

SOLE Grant Management also supports specifying Roles and Users to operate on.
In the configuration, Roles and Users can be defined with keys specified_roles and specified_users respectively.
When defined, SOLE Grant Management would only operate on Roles and Users that match the values defined.
Either Object name or Regex can be provided in the keys specified_roles and specified_users.

Example

specified_roles:  - DATAOPSADMIN  - DATAOPS_*
specified_users:  - SOLEADMIN  - DATAOPS_*

With the above configuration, SOLE Grant Management would only grant or revoke privilege if the role is either DATAOPSADMIN or matches regex DATAOPS_*.
Similarly, role would only be granted or revoked from if:

• Role is either DATAOPSADMIN or matches regex DATAOPS_*, or
• User is either SOLEADMIN or matches regex DATAOPS_*

Lifecycle Actions

COMPILE

When LIFECYCLE_ACTION is set to COMPILE, the engine generates the configuration required to initialize and manage the Snowflake objects from the user-defined specifications.

Using the specifications defined in files, the engine performs validation, dependency resolution and generates resource configurations and syntax.
During the compile, the engine also resolves the grants and revokes on Snowflake Objects. For more information see Access Management section.

The following parameters can be used to control the execution of COMPILE:

Parameter	Required or Optional (and default)	Description
CONFIGURATION_DIR	REQUIRED	Path of directory in which the configuration files are present
DATAOPS_PREFIX	REQUIRED	Prefix to be added to Account-Level and Databases(Except default database)
DATAOPS_ENV_NAME	REQUIRED	Suffix to be added to Account-Level and Databases(Except default database)
DATAOPS_DEBUG	OPTIONAL	If set, enables debug logs for the Engine

VALIDATE

When LIFECYCLE_ACTION is set to VALIDATE, the engine performs validation of the resource configurations generated during the COMPILE stage.

PLAN

When LIFECYCLE_ACTION is set to PLAN, the engine performs planning of execution steps to create or update the managed Snowflake Objects.

Before generating a plan, the engine tries to import all the specified Snowflake Objects if they exist.
This import helps initialize or update the local state file for existing Snowflake Objects which is used to generate the plan.
An import summary is displayed during and after completion of import.

Due to aliases and multiple format support, a plan might have in-place changes for Snowflake Objects. Refer to In-Place Changes for more information.

In-case the state file is corrupted, if there is a discrepancy in the state file(An Object is managed in state, but deleted from Snowflake externally), the file can be removed and re-initialized with current Snowflake Object configuration.
To do so, any value has to be set in the variable LIFECYCLE_STATE_RESET.
This can be set for the pipeline, to re-initialize the state for all resource groups or individually for any PLAN job to just reset state for that particular resource.

The generated plan is provided to the APPLY action.

PLAN-DESTROY

Similar to PLAN, when LIFECYCLE_ACTION is set to PLAN-DESTROY, the engine performs planning of execution steps to destroy the managed Snowflake Objects.

Using the existing state(if it exists), an execution plan to destroy the managed Snowflake Objects is generated and saved.
The execution plan lists the changes it has detected to perform in order to delete/drop the managed Snowflake objects.

The generated plan is provided to the DESTROY action.

APPLY

When LIFECYCLE_ACTION is set to APPLY, the engine executes the actions specified by the PLAN action.

Using the output from the plan, the engine executes the defined steps.
This ensures that the engine only performs those changes that were logged by the PLAN action.
This implies that the user is aware of all the changes to be made.

DESTROY

When LIFECYCLE_ACTION is set to DESTROY, the engine executes the actions specified by the PLAN-DESTROY action.

Using the output from PLAN-DESTROY, the engine executes the defined deletion steps.
This ensures that the engine only performs those changes that were logged by the PLAN-DESTROY action.
This implies that the user is aware of all the changes to be made.

AGGREGATE

As an alternative to setting up individual jobs for each resource group and action, an aggregate action can be configured to perform COMPILE on configuration and VALIDATE, PLAN and APPLY for all resource groups in sequential manner in a single job.

This cuts down on the number of setup jobs to be setup in a pipeline to single jobs to manage the lifecycle of Snowflake Objects.

This can be executed by setting LIFECYCLE_ACTION to AGGREGATE.

As AGGREGATE acts on all resource groups, LIFECYCLE_MANAGE_OBJECT does not have to be specified, however other variables such as CONFIGURATION_DIR are required.

AGGREGATE-DESTROY

Similar to AGGREGATE, the AGGREGATE-DESTROY performs the PLAN-DESTROY and DESTROY actions in a single job.

This can be executed by setting LIFECYCLE_ACTION to AGGREGATE-DESTROY.

Supported Snowflake Objects

Following is the list of Snowflake Objects whose Lifecycle can be managed with the Snowflake Object Lifecycle Engine.
To get more information regarding configuration, supported parameters and examples, please visit sections for the object.

• Account
• Database
• External Table
• File Format
• Masking Policy
• Pipe
• Resource Monitor
• Role
• Schema
• Share
• Stage
• Stream
• Table
• Tag
• Task
• User
• View
• Warehouse