Package com.snowflake.kafka.connector.internal

Class SnowflakeConnectionServiceV1

java.lang.Object

com.snowflake.kafka.connector.internal.SnowflakeConnectionServiceV1

All Implemented Interfaces:

SnowflakeConnectionService

public class SnowflakeConnectionServiceV1
extends Object
implements SnowflakeConnectionService

Implementation of Snowflake Connection Service interface which includes all handshake between KC and SF through JDBC connection.

Field Summary

Fields

Modifier and Type	Field	Description
static String	USER_AGENT_SUFFIX_FORMAT

Method Summary

Modifier and Type	Method	Description
void	alterNonNullableColumns(String tableName, List<String> columnNames)	Alter table to drop non-nullability of a list of columns
void	appendColumnsToTable(String tableName, Map<String,String> columnToType)	Alter table to add columns according to a map from columnNames to their types
void	appendMetaColIfNotExist(String tableName)	Append a VARIANT type column "RECORD_METADATA" to the table if it is not present.
SnowflakeIngestionService	buildIngestService(String stageName, String pipeName)	build ingest service instance for given stage and pipe
void	close()	Close Connection
void	createPipe(String tableName, String stageName, String pipeName)	create a snowpipe if not exists
void	createPipe(String tableName, String stageName, String pipeName, boolean overwrite)	create a snowpipe
void	createStage(String stageName)	create stage if not exists
void	createStage(String stageName, boolean overwrite)	create a stage
void	createTable(String tableName)	create table is not exists
void	createTable(String tableName, boolean overwrite)	Create a table with two variant columns: RECORD_METADATA and RECORD_CONTENT
void	createTableWithOnlyMetadataColumn(String tableName)	Create a table with only the RECORD_METADATA column.
void	databaseExists(String databaseName)	check if a given database exists
void	dropPipe(String pipeName)	drop snowpipe
void	dropStage(String stageName)	drop stage
boolean	dropStageIfEmpty(String stageName)	drop stage if the given stage is empty
protected ChannelMigrateOffsetTokenResponseDTO	getChannelMigrateOffsetTokenResponseDTO(String migrateOffsetTokenResultFromSysFunc)
Connection	getConnection()
String	getConnectorName()
SnowflakeInternalStage	getInternalStage()
SnowflakeTelemetryService	getTelemetryClient()
boolean	hasSchemaEvolutionPermission(String tableName, String role)	Check whether the user has the role privilege to do schema evolution and whether the schema evolution option is enabled on the table
boolean	isClosed()
boolean	isPipeCompatible(String tableName, String stageName, String pipeName)	check snowpipe definition
boolean	isStageCompatible(String stageName)	Examine all file names matches our pattern
boolean	isTableCompatible(String tableName)	Check the given table has correct schema correct schema: (record_metadata variant, record_content variant)
List<String>	listStage(String stageName, String prefix)	list a non table stage and return a list of file names contained in given subdirectory
List<String>	listStage(String stageName, String prefix, boolean isTableStage)	list a stage and return a list of file names contained in given subdirectory
ChannelMigrateOffsetTokenResponseDTO	migrateStreamingChannelOffsetToken(String tableName, String sourceChannelName, String destinationChannelName)	Migrate Streaming Channel offsetToken from a source Channel to a destination channel.
void	moveToTableStage(String tableName, String stageName, String prefix)	move all files on stage related to given pipe to table stage
void	moveToTableStage(String tableName, String stageName, List<String> files)
boolean	pipeExist(String pipeName)	check pipe existence
void	purgeStage(String stageName, List<String> files)	purge files from given stage
void	put(String stageName, String fileName, String content)	Deprecated.
void	putToTableStage(String tableName, String fileName, byte[] content)	put a file to table stage
void	putWithCache(String stageName, String fileName, String content)	put a file to stage.
void	schemaExists(String schemaName)	check if a given schema exists
boolean	stageExist(String stageName)	check stage existence
boolean	tableExist(String tableName)	check table existence

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

USER_AGENT_SUFFIX_FORMAT

public static final String USER_AGENT_SUFFIX_FORMAT

See Also:

Constant Field Values

Method Detail

createTable

public void createTable(String tableName,
                        boolean overwrite)

Description copied from interface: SnowflakeConnectionService

Create a table with two variant columns: RECORD_METADATA and RECORD_CONTENT

Specified by:

createTable in interface SnowflakeConnectionService

Parameters:

tableName - a string represents table name

overwrite - if true, execute "create or replace table" query; otherwise, run "create table if not exists"

createTable

public void createTable(String tableName)

Description copied from interface: SnowflakeConnectionService

create table is not exists

Specified by:

createTable in interface SnowflakeConnectionService

Parameters:

tableName - table name

createTableWithOnlyMetadataColumn

public void createTableWithOnlyMetadataColumn(String tableName)

Description copied from interface: SnowflakeConnectionService

Create a table with only the RECORD_METADATA column. The rest of the columns might be added through schema evolution

In the beginning of the function we will check if we have the permission to do schema evolution, and we will error out if we don't

Specified by:

createTableWithOnlyMetadataColumn in interface SnowflakeConnectionService

Parameters:

tableName - table name

createPipe

public void createPipe(String tableName,
                       String stageName,
                       String pipeName,
                       boolean overwrite)

Description copied from interface: SnowflakeConnectionService

create a snowpipe

Specified by:

createPipe in interface SnowflakeConnectionService

Parameters:

tableName - table name

stageName - stage name

pipeName - pipe name

overwrite - if true, execute "create or replace pipe" statement, otherwise, run "create pipe if not exists"

createPipe

public void createPipe(String tableName,
                       String stageName,
                       String pipeName)

Description copied from interface: SnowflakeConnectionService

create a snowpipe if not exists

Specified by:

createPipe in interface SnowflakeConnectionService

Parameters:

tableName - table name

stageName - stage name

pipeName - pipe name

createStage

public void createStage(String stageName,
                        boolean overwrite)

Description copied from interface: SnowflakeConnectionService

create a stage

Specified by:

createStage in interface SnowflakeConnectionService

Parameters:

stageName - stage name

overwrite - if true, execute "create or replace stage" statement; otherwise, run "create stage if not exists"

createStage

public void createStage(String stageName)

Description copied from interface: SnowflakeConnectionService

create stage if not exists

Specified by:

createStage in interface SnowflakeConnectionService

Parameters:

stageName - stage name

tableExist

public boolean tableExist(String tableName)

Description copied from interface: SnowflakeConnectionService

check table existence

Specified by:

tableExist in interface SnowflakeConnectionService

Parameters:

tableName - table name

Returns:

true if table exists, false otherwise

stageExist

public boolean stageExist(String stageName)

Description copied from interface: SnowflakeConnectionService

check stage existence

Specified by:

stageExist in interface SnowflakeConnectionService

Parameters:

stageName - stage name

Returns:

true if stage exists, false otherwise

pipeExist

public boolean pipeExist(String pipeName)

Description copied from interface: SnowflakeConnectionService

check pipe existence

Specified by:

pipeExist in interface SnowflakeConnectionService

Parameters:

pipeName - pipe name

Returns:

true if pipe exists, false otherwise

isTableCompatible

public boolean isTableCompatible(String tableName)

Description copied from interface: SnowflakeConnectionService

Check the given table has correct schema correct schema: (record_metadata variant, record_content variant)

Specified by:

isTableCompatible in interface SnowflakeConnectionService

Parameters:

tableName - table name

Returns:

true if schema is correct, false is schema is incorrect or table does not exist

appendMetaColIfNotExist

public void appendMetaColIfNotExist(String tableName)

Description copied from interface: SnowflakeConnectionService

Append a VARIANT type column "RECORD_METADATA" to the table if it is not present.

This method is only called when schematization is enabled

Specified by:

appendMetaColIfNotExist in interface SnowflakeConnectionService

Parameters:

tableName - table name

hasSchemaEvolutionPermission

public boolean hasSchemaEvolutionPermission(String tableName,
                                            String role)

Check whether the user has the role privilege to do schema evolution and whether the schema evolution option is enabled on the table

Specified by:

hasSchemaEvolutionPermission in interface SnowflakeConnectionService

Parameters:

tableName - the name of the table

role - the role of the user

Returns:

whether schema evolution has the required permission to be performed

appendColumnsToTable

public void appendColumnsToTable(String tableName,
                                 Map<String,String> columnToType)

Alter table to add columns according to a map from columnNames to their types

Specified by:

appendColumnsToTable in interface SnowflakeConnectionService

Parameters:

tableName - the name of the table

columnToType - the mapping from the columnNames to their types

alterNonNullableColumns

public void alterNonNullableColumns(String tableName,
                                    List<String> columnNames)

Alter table to drop non-nullability of a list of columns

Specified by:

alterNonNullableColumns in interface SnowflakeConnectionService

Parameters:

tableName - the name of the table

columnNames - the list of columnNames

isStageCompatible

public boolean isStageCompatible(String stageName)

Description copied from interface: SnowflakeConnectionService

Examine all file names matches our pattern

Specified by:

isStageCompatible in interface SnowflakeConnectionService

Parameters:

stageName - stage name

Returns:

true is stage is compatible, false if stage does not exist or file name invalid

isPipeCompatible

public boolean isPipeCompatible(String tableName,
                                String stageName,
                                String pipeName)

Description copied from interface: SnowflakeConnectionService

check snowpipe definition

Specified by:

isPipeCompatible in interface SnowflakeConnectionService

Parameters:

tableName - table name

stageName - stage name

pipeName - pipe name

Returns:

true if definition is correct, false if it is incorrect or pipe does not exists

databaseExists

public void databaseExists(String databaseName)

Description copied from interface: SnowflakeConnectionService

check if a given database exists

Specified by:

databaseExists in interface SnowflakeConnectionService

Parameters:

databaseName - database name

schemaExists

public void schemaExists(String schemaName)

Description copied from interface: SnowflakeConnectionService

check if a given schema exists

Specified by:

schemaExists in interface SnowflakeConnectionService

Parameters:

schemaName - schema name

dropPipe

public void dropPipe(String pipeName)

Description copied from interface: SnowflakeConnectionService

drop snowpipe

Specified by:

dropPipe in interface SnowflakeConnectionService

Parameters:

pipeName - pipe name

dropStageIfEmpty

public boolean dropStageIfEmpty(String stageName)

Description copied from interface: SnowflakeConnectionService

drop stage if the given stage is empty

Specified by:

dropStageIfEmpty in interface SnowflakeConnectionService

Parameters:

stageName - stage name

Returns:

true if stage dropped, otherwise false

dropStage

public void dropStage(String stageName)

Description copied from interface: SnowflakeConnectionService

drop stage

Specified by:

dropStage in interface SnowflakeConnectionService

Parameters:

stageName - stage name

purgeStage

public void purgeStage(String stageName,
                       List<String> files)

Description copied from interface: SnowflakeConnectionService

purge files from given stage

Specified by:

purgeStage in interface SnowflakeConnectionService

Parameters:

stageName - stage name

files - list of file names

moveToTableStage

public void moveToTableStage(String tableName,
                             String stageName,
                             List<String> files)

Specified by:

moveToTableStage in interface SnowflakeConnectionService

moveToTableStage

public void moveToTableStage(String tableName,
                             String stageName,
                             String prefix)

Description copied from interface: SnowflakeConnectionService

move all files on stage related to given pipe to table stage

Specified by:

moveToTableStage in interface SnowflakeConnectionService

Parameters:

tableName - table name

stageName - stage name

prefix - prefix name

listStage

public List<String> listStage(String stageName,
                              String prefix,
                              boolean isTableStage)

Description copied from interface: SnowflakeConnectionService

list a stage and return a list of file names contained in given subdirectory

Specified by:

listStage in interface SnowflakeConnectionService

Parameters:

stageName - stage name

prefix - prefix name

isTableStage - true if it is a table stage

Returns:

a list of file names in given subdirectory, file name = "{prefix}filename"

listStage

public List<String> listStage(String stageName,
                              String prefix)

Description copied from interface: SnowflakeConnectionService

list a non table stage and return a list of file names contained in given subdirectory

Specified by:

listStage in interface SnowflakeConnectionService

Parameters:

stageName - stage name

prefix - prefix name

Returns:

a list of file names in given subdirectory, file name = "{prefix}filename"

put

@Deprecated
public void put(String stageName,
                String fileName,
                String content)

Deprecated.

Description copied from interface: SnowflakeConnectionService

put a file to stage

Specified by:

put in interface SnowflakeConnectionService

Parameters:

stageName - stage name

fileName - file name

content - file content

putWithCache

public void putWithCache(String stageName,
                         String fileName,
                         String content)

Description copied from interface: SnowflakeConnectionService

put a file to stage. Cache credential for AWS, Azure and GCS storage.

Specified by:

putWithCache in interface SnowflakeConnectionService

Parameters:

stageName - stage name

fileName - file name which is of format (stageName/connectorName/tableName/partition/offsetStart_offsetEnd_timestamp)

Check to find out how fileName is determined

content - file content

putToTableStage

public void putToTableStage(String tableName,
                            String fileName,
                            byte[] content)

Description copied from interface: SnowflakeConnectionService

put a file to table stage

Specified by:

putToTableStage in interface SnowflakeConnectionService

Parameters:

tableName - table name

fileName - file name

content - file content

getTelemetryClient

public SnowflakeTelemetryService getTelemetryClient()

Specified by:

getTelemetryClient in interface SnowflakeConnectionService

Returns:

telemetry client

close

public void close()

Description copied from interface: SnowflakeConnectionService

Close Connection

Specified by:

close in interface SnowflakeConnectionService

isClosed

public boolean isClosed()

Specified by:

isClosed in interface SnowflakeConnectionService

Returns:

true is connection is closed

getConnectorName

public String getConnectorName()

Specified by:

getConnectorName in interface SnowflakeConnectionService

Returns:

name of Kafka Connector instance

buildIngestService

public SnowflakeIngestionService buildIngestService(String stageName,
                                                    String pipeName)

Description copied from interface: SnowflakeConnectionService

build ingest service instance for given stage and pipe

Specified by:

buildIngestService in interface SnowflakeConnectionService

Parameters:

stageName - stage name

pipeName - pipe name

Returns:

an instance of SnowflakeIngestService

getConnection

public Connection getConnection()

Specified by:

getConnection in interface SnowflakeConnectionService

Returns:

the raw jdbc connection

getInternalStage

public SnowflakeInternalStage getInternalStage()

migrateStreamingChannelOffsetToken

public ChannelMigrateOffsetTokenResponseDTO migrateStreamingChannelOffsetToken(String tableName,
                                                                               String sourceChannelName,
                                                                               String destinationChannelName)

Description copied from interface: SnowflakeConnectionService

Migrate Streaming Channel offsetToken from a source Channel to a destination channel.

Here, source channel is the new channel format we created here * @see Commit

Destination channel is the original Format containing only topicName and partition number.

We catch SQLException and JsonProcessingException that might happen in this method. The caller should always open the Old Channel format. This old channel format will also be the key to many HashMaps we will create. (For instance SnowflakeSinkServiceV2.partitionsToChannel)

Specified by:

migrateStreamingChannelOffsetToken in interface SnowflakeConnectionService

Parameters:

tableName - Name of the table

sourceChannelName - sourceChannel name from where the offset Token will be fetched. Channel with this name will also be deleted.

destinationChannelName - destinationChannel name to where the offsetToken will be copied over.

Returns:

The DTO serialized from the migration response.

getChannelMigrateOffsetTokenResponseDTO

protected ChannelMigrateOffsetTokenResponseDTO getChannelMigrateOffsetTokenResponseDTO(String migrateOffsetTokenResultFromSysFunc)
                                                                                throws com.fasterxml.jackson.core.JsonProcessingException

Throws:

com.fasterxml.jackson.core.JsonProcessingException