org.apache.beam.sdk.io.snowflake

Class SnowflakeIO.Write<T>

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<T>,org.apache.beam.sdk.values.PDone>

org.apache.beam.sdk.io.snowflake.SnowflakeIO.Write<T>

All Implemented Interfaces:

java.io.Serializable, org.apache.beam.sdk.transforms.display.HasDisplayData

Enclosing class:

SnowflakeIO

public abstract static class SnowflakeIO.Write<T>
extends org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<T>,org.apache.beam.sdk.values.PDone>

Implementation of SnowflakeIO.write().

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

annotations, name, resourceHints

Constructor Summary

Constructors

Constructor and Description
Write()

Method Summary

Modifier and Type	Method and Description
org.apache.beam.sdk.values.PDone	expand(org.apache.beam.sdk.values.PCollection<T> input)
protected org.apache.beam.sdk.transforms.ParDo.SingleOutput<java.util.List<java.lang.String>,java.lang.Void>	streamToTable(SnowflakeServices snowflakeServices, org.apache.beam.sdk.options.ValueProvider<java.lang.String> stagingBucketDir)
SnowflakeIO.Write<T>	to(java.lang.String table) A table name to be written in Snowflake.
SnowflakeIO.Write<T>	to(org.apache.beam.sdk.options.ValueProvider<java.lang.String> table)
SnowflakeIO.Write<T>	withCreateDisposition(CreateDisposition createDisposition) A disposition to be used during table preparation.
SnowflakeIO.Write<T>	withDataSourceConfiguration(SnowflakeIO.DataSourceConfiguration config) Setting information about Snowflake server.
SnowflakeIO.Write<T>	withDataSourceProviderFn(org.apache.beam.sdk.transforms.SerializableFunction<java.lang.Void,javax.sql.DataSource> dataSourceProviderFn) Setting function that will provide SnowflakeIO.DataSourceConfiguration in runtime.
SnowflakeIO.Write<T>	withDebugMode(StreamingLogLevel debugLevel) The option to verbose info (or only errors) of loaded files while streaming.
SnowflakeIO.Write<T>	withFileNameTemplate(java.lang.String fileNameTemplate) A template name for files saved to GCP.
SnowflakeIO.Write<T>	withFlushRowLimit(java.lang.Integer rowsCount) Sets number of row limit that will be saved to the staged file and then loaded to Snowflake.
SnowflakeIO.Write<T>	withFlushTimeLimit(org.joda.time.Duration triggeringFrequency) Sets duration how often staged files will be created and then how often ingested by Snowflake during streaming.
SnowflakeIO.Write<T>	withQueryTransformation(java.lang.String query) A query to be executed in Snowflake.
SnowflakeIO.Write<T>	withQueryTransformation(org.apache.beam.sdk.options.ValueProvider<java.lang.String> query)
SnowflakeIO.Write<T>	withQuotationMark(java.lang.String quotationMark) Sets Snowflake-specific quotations around strings.
SnowflakeIO.Write<T>	withShardsNumber(java.lang.Integer shardsNumber) Number of shards that are created per window.
SnowflakeIO.Write<T>	withSnowflakeServices(SnowflakeServices snowflakeServices) A snowflake service SnowflakeServices implementation which is supposed to be used.
SnowflakeIO.Write<T>	withSnowPipe(java.lang.String snowPipe) Sets name of SnowPipe which can be created in Snowflake dashboard or cli:
SnowflakeIO.Write<T>	withSnowPipe(org.apache.beam.sdk.options.ValueProvider<java.lang.String> snowPipe) Same as withSnowPipe(String, but with a ValueProvider.
SnowflakeIO.Write<T>	withStagingBucketName(java.lang.String stagingBucketName) Name of the cloud bucket (GCS by now) to use as tmp location of CSVs during COPY statement.
SnowflakeIO.Write<T>	withStagingBucketName(org.apache.beam.sdk.options.ValueProvider<java.lang.String> stagingBucketName)
SnowflakeIO.Write<T>	withStorageIntegrationName(java.lang.String integrationName) Name of the Storage Integration in Snowflake to be used.
SnowflakeIO.Write<T>	withStorageIntegrationName(org.apache.beam.sdk.options.ValueProvider<java.lang.String> integrationName)
SnowflakeIO.Write<T>	withTableSchema(SnowflakeTableSchema tableSchema) Table schema to be used during creating table.
SnowflakeIO.Write<T>	withUserDataMapper(SnowflakeIO.UserDataMapper<T> userDataMapper) User-defined function mapping user data into CSV lines.
SnowflakeIO.Write<T>	withWriteDisposition(WriteDisposition writeDisposition) A disposition to be used during writing to table phase.

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

addAnnotation, compose, compose, getAdditionalInputs, getAnnotations, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, populateDisplayData, setResourceHints, toString, validate, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

Write

public Write()

Method Detail

withDataSourceConfiguration

public SnowflakeIO.Write<T> withDataSourceConfiguration(SnowflakeIO.DataSourceConfiguration config)

Setting information about Snowflake server.

Parameters:

config - An instance of SnowflakeIO.DataSourceConfiguration.

withDataSourceProviderFn

public SnowflakeIO.Write<T> withDataSourceProviderFn(org.apache.beam.sdk.transforms.SerializableFunction<java.lang.Void,javax.sql.DataSource> dataSourceProviderFn)

Setting function that will provide SnowflakeIO.DataSourceConfiguration in runtime.

Parameters:

dataSourceProviderFn - a SerializableFunction.

to

public SnowflakeIO.Write<T> to(java.lang.String table)

A table name to be written in Snowflake.

Parameters:

table - String with the name of the table.

to

public SnowflakeIO.Write<T> to(org.apache.beam.sdk.options.ValueProvider<java.lang.String> table)

withStagingBucketName

public SnowflakeIO.Write<T> withStagingBucketName(java.lang.String stagingBucketName)

Name of the cloud bucket (GCS by now) to use as tmp location of CSVs during COPY statement.

Parameters:

stagingBucketName - String with the name of the bucket.

withStagingBucketName

public SnowflakeIO.Write<T> withStagingBucketName(org.apache.beam.sdk.options.ValueProvider<java.lang.String> stagingBucketName)

withStorageIntegrationName

public SnowflakeIO.Write<T> withStorageIntegrationName(java.lang.String integrationName)

Name of the Storage Integration in Snowflake to be used. See https://docs.snowflake.com/en/sql-reference/sql/create-storage-integration.html for reference.

Parameters:

integrationName - String with the name of the Storage Integration.

withStorageIntegrationName

public SnowflakeIO.Write<T> withStorageIntegrationName(org.apache.beam.sdk.options.ValueProvider<java.lang.String> integrationName)

withQueryTransformation

public SnowflakeIO.Write<T> withQueryTransformation(java.lang.String query)

A query to be executed in Snowflake.

Parameters:

query - String with query.

withQueryTransformation

public SnowflakeIO.Write<T> withQueryTransformation(org.apache.beam.sdk.options.ValueProvider<java.lang.String> query)

withFileNameTemplate

public SnowflakeIO.Write<T> withFileNameTemplate(java.lang.String fileNameTemplate)

A template name for files saved to GCP.

Parameters:

fileNameTemplate - String with template name for files.

withUserDataMapper

public SnowflakeIO.Write<T> withUserDataMapper(SnowflakeIO.UserDataMapper<T> userDataMapper)

User-defined function mapping user data into CSV lines.

Parameters:

userDataMapper - an instance of SnowflakeIO.UserDataMapper.

withFlushTimeLimit

public SnowflakeIO.Write<T> withFlushTimeLimit(org.joda.time.Duration triggeringFrequency)

Sets duration how often staged files will be created and then how often ingested by Snowflake during streaming.

Parameters:

triggeringFrequency - time for triggering frequency in Duration type.

Returns:

withSnowPipe

public SnowflakeIO.Write<T> withSnowPipe(java.lang.String snowPipe)

Sets name of SnowPipe which can be created in Snowflake dashboard or cli:

 CREATE snowPipeName AS COPY INTO your_table from @yourstage;
 

The stage in COPY statement should be pointing to the cloud integration with the valid bucket url, ex. for GCS:

 CREATE STAGE yourstage
 URL = 'gcs://yourbucket/path/'
 STORAGE_INTEGRATION = your_integration;
 

 CREATE STORAGE INTEGRATION your_integration
   TYPE = EXTERNAL_STAGE
   STORAGE_PROVIDER = GCS
   ENABLED = TRUE
   STORAGE_ALLOWED_LOCATIONS = ('gcs://yourbucket/path/')
 

Parameters:

snowPipe - name of created SnowPipe in Snowflake dashboard.

Returns:

withSnowPipe

public SnowflakeIO.Write<T> withSnowPipe(org.apache.beam.sdk.options.ValueProvider<java.lang.String> snowPipe)

Same as withSnowPipe(String, but with a ValueProvider.

Parameters:

snowPipe - name of created SnowPipe in Snowflake dashboard.

Returns:

withShardsNumber

public SnowflakeIO.Write<T> withShardsNumber(java.lang.Integer shardsNumber)

Number of shards that are created per window.

Parameters:

shardsNumber - defined number of shards or 1 by default.

Returns:

withFlushRowLimit

public SnowflakeIO.Write<T> withFlushRowLimit(java.lang.Integer rowsCount)

Sets number of row limit that will be saved to the staged file and then loaded to Snowflake. If the number of rows will be lower than the limit it will be loaded with current number of rows after certain time specified by setting withFlushTimeLimit(Duration triggeringFrequency)

Parameters:

rowsCount - Number of rows that will be in one file staged for loading. Default: 10000.

Returns:

withWriteDisposition

public SnowflakeIO.Write<T> withWriteDisposition(WriteDisposition writeDisposition)

A disposition to be used during writing to table phase.

Parameters:

writeDisposition - an instance of WriteDisposition.

withCreateDisposition

public SnowflakeIO.Write<T> withCreateDisposition(CreateDisposition createDisposition)

A disposition to be used during table preparation.

Parameters:

createDisposition - - an instance of CreateDisposition.

withTableSchema

public SnowflakeIO.Write<T> withTableSchema(SnowflakeTableSchema tableSchema)

Table schema to be used during creating table.

Parameters:

tableSchema - - an instance of SnowflakeTableSchema.

withSnowflakeServices

public SnowflakeIO.Write<T> withSnowflakeServices(SnowflakeServices snowflakeServices)

A snowflake service SnowflakeServices implementation which is supposed to be used.

Parameters:

snowflakeServices - an instance of SnowflakeServices.

withQuotationMark

public SnowflakeIO.Write<T> withQuotationMark(java.lang.String quotationMark)

Sets Snowflake-specific quotations around strings.

Parameters:

quotationMark - with possible single quote ', double quote " or nothing. Default value is single quotation '.

Returns:

withDebugMode

public SnowflakeIO.Write<T> withDebugMode(StreamingLogLevel debugLevel)

The option to verbose info (or only errors) of loaded files while streaming. It is not set by default because it may influence performance. For details: insert report REST API.

Parameters:

debugLevel - error or info debug level from enum StreamingLogLevel

Returns:

expand

public org.apache.beam.sdk.values.PDone expand(org.apache.beam.sdk.values.PCollection<T> input)

Specified by:

expand in class org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<T>,org.apache.beam.sdk.values.PDone>

streamToTable

protected org.apache.beam.sdk.transforms.ParDo.SingleOutput<java.util.List<java.lang.String>,java.lang.Void> streamToTable(SnowflakeServices snowflakeServices,
                                                                                                                           org.apache.beam.sdk.options.ValueProvider<java.lang.String> stagingBucketDir)