Skip to main content

Pipe

Configuration can be provided to Snowflake Object Lifecycle Engine for the following operation with Pipe:

  • Manage Lifecycle of new and existing Pipe

Supported Parameters#

The following parameters in a pipe are supported by the runner:

  • COPY_STATEMENT: Specifies the copy statement for the pipe.
    • REQUIRED
    • Configuration Key: copy_statement
    • Data Type: Object . See COPY INTO table for copy_statement parameters.

      Refer Snowflake Docs for COPY INTO table copy options and file format .

  • AUTO_INGEST: Specifies an auto_ingest param for the pipe.
    • Configuration Key: auto_ingest
    • Data Type: Boolean
  • AWS_SNS_TOPIC_ARN: Specifies the Amazon Resource Name (ARN) for the SNS topic for your S3 bucket.
    • Configuration Key: aws_sns_topic_arn
    • Data Type: String
  • COMMENT: Specifies a comment for the Pipe.
    • Configuration Key: comment
    • Data Type: String
  • MANAGE_MODE: Configures what properties to manage for the Pipe.
    • Configuration key: manage_mode
    • Data Type: String
    • Possible Values:
      • none
      • all(Default)

Basic syntax#

databases:  <database-name>:    schemas:      <schema-name>:        pipes:          <pipe-name>:            <configuration-key>: <value>

Supported Pipe Grants to Roles#

Following is the list of Privileges Grant to Roles that can be specified in the pipe definition

  • ALL PRIVILEGES
  • MONITOR
  • OPERATE
  • OWNERSHIP

Examples#

Below are some example for configuring pipes

Pipe configuration with TYPE parameter#

databases:  SALES_RECORD:     comment: "product sales record"    schemas:      comment: "sales records for year 2021"      pipes:        PIPE_1:          comment: A pipe.          auto_ingest: false          copy_statement:            into:              database: "DATABASE_3"              schema: "SCHEMA_1"              table: "TABLE_3"            from:              database: "DUMMY_DATABASE"              schema: "PUBLIC"              stage: "LIFECYCLE_STAGE"            pattern: ".*[.]csv"            file_format:              type: "JSON"              compression: "AUTO"              binary_format: "utf8"              trim_space: false              enable_octal: false              strip_outer_array: false              strip_null_values: false              replace_invalid_characters: false              ignore_utf8_errors: true              skip_byte_order_mark: false            enforce_length: false            truncate_columns: true

Pipe configuration without TYPE parameter#

databases:  SALES_RECORD_FEB:    comment: "product sales record"    schemas:      comment: "sales records for year 2021 of feb month"      PIPE_2:        comment: A pipe.        auto_ingest: false        copy_statement:          into:            database: "DATABASE_3"            schema: "SCHEMA_1"            table: "TABLE_2"          from:            database: "DUMMY_DATABASE"            schema: "PUBLIC"            stage: "LIFECYCLE_STAGE"          pattern: ".*[.]csv"          file_format:            format_database: "DUMMY_DB"            format_schema: "PUBLIC"            format_name: "DMY_FORMAT"
Last updated on