Package com.snowflake.kafka.connector.internal

Interface SnowflakeConnectionService

    • Method Detail

      • createTable

        void createTable​(String tableName,
                 boolean overwrite)
        Create a table with two variant columns: RECORD_METADATA and RECORD_CONTENT
        Parameters:
        tableName - a string represents table name
        overwrite - if true, execute "create or replace table" query; otherwise, run "create table if not exists"

      • createTable

        void createTable​(String tableName)
        create table is not exists
        Parameters:
        tableName - table name

      • createPipe

        void createPipe​(String tableName,
                String stageName,
                String pipeName,
                boolean overwrite)
        create a snowpipe
        Parameters:
        pipeName - pipe name
        tableName - table name
        stageName - stage name
        overwrite - if true, execute "create or replace pipe" statement, otherwise, run "create pipe if not exists"

      • createPipe

        void createPipe​(String tableName,
                String stageName,
                String pipeName)
        create a snowpipe if not exists
        Parameters:
        pipeName - pipe name
        tableName - table name
        stageName - stage name

      • createStage

        void createStage​(String stageName,
                 boolean overwrite)
        create a stage
        Parameters:
        stageName - stage name
        overwrite - if true, execute "create or replace stage" statement; otherwise, run "create stage if not exists"

      • createStage

        void createStage​(String stageName)
        create stage if not exists
        Parameters:
        stageName - stage name

      • tableExist

        boolean tableExist​(String tableName)
        check table existence
        Parameters:
        tableName - table name
        Returns:
        true if table exists, false otherwise

      • stageExist

        boolean stageExist​(String stageName)
        check stage existence
        Parameters:
        stageName - stage name
        Returns:
        true if stage exists, false otherwise

      • pipeExist

        boolean pipeExist​(String pipeName)
        check pipe existence
        Parameters:
        pipeName - pipe name
        Returns:
        true if pipe exists, false otherwise

      • isTableCompatible

        boolean isTableCompatible​(String tableName)
        Check the given table has correct schema correct schema: (record_metadata variant, record_content variant)
        Parameters:
        tableName - table name
        Returns:
        true if schema is correct, false is schema is incorrect or table does not exist

      • hasSchemaEvolutionPermission

        boolean hasSchemaEvolutionPermission​(String tableName,
                                     String role)
        Check whether the user has the role privilege to do schema evolution and whether the schema evolution option is enabled on the table
        Parameters:
        tableName - the name of the table
        role - the role of the user
        Returns:
        whether table and role has the required permission to perform schema evolution

      • appendColumnsToTable

        void appendColumnsToTable​(String tableName,
                          Map<String,​String> columnToType)
        Alter table to add columns according to a map from columnNames to their types
        Parameters:
        tableName - the name of the table
        columnToType - the mapping from the columnNames to their types

      • alterNonNullableColumns

        void alterNonNullableColumns​(String tableName,
                             List<String> columnNames)
        Alter table to drop non-nullability of a list of columns
        Parameters:
        tableName - the name of the table
        columnNames - the list of columnNames

      • isStageCompatible

        boolean isStageCompatible​(String stageName)
        Examine all file names matches our pattern
        Parameters:
        stageName - stage name
        Returns:
        true is stage is compatible, false if stage does not exist or file name invalid

      • isPipeCompatible

        boolean isPipeCompatible​(String tableName,
                         String stageName,
                         String pipeName)
        check snowpipe definition
        Parameters:
        pipeName - pipe name
        tableName - table name
        stageName - stage name
        Returns:
        true if definition is correct, false if it is incorrect or pipe does not exists

      • databaseExists

        void databaseExists​(String databaseName)
        check if a given database exists
        Parameters:
        databaseName - database name

      • schemaExists

        void schemaExists​(String schemaName)
        check if a given schema exists
        Parameters:
        schemaName - schema name

      • dropPipe

        void dropPipe​(String pipeName)
        drop snowpipe
        Parameters:
        pipeName - pipe name

      • dropStageIfEmpty

        boolean dropStageIfEmpty​(String stageName)
        drop stage if the given stage is empty
        Parameters:
        stageName - stage name
        Returns:
        true if stage dropped, otherwise false

      • dropStage

        void dropStage​(String stageName)
        drop stage
        Parameters:
        stageName - stage name

      • purgeStage

        void purgeStage​(String stageName,
                List<String> files)
        purge files from given stage
        Parameters:
        stageName - stage name
        files - list of file names

      • moveToTableStage

        void moveToTableStage​(String tableName,
                      String stageName,
                      String prefix)
        move all files on stage related to given pipe to table stage
        Parameters:
        stageName - stage name
        tableName - table name
        prefix - prefix name

      • listStage

        List<String> listStage​(String stageName,
                       String prefix,
                       boolean isTableStage)
        list a stage and return a list of file names contained in given subdirectory
        Parameters:
        stageName - stage name
        prefix - prefix name
        isTableStage - true if it is a table stage
        Returns:
        a list of file names in given subdirectory, file name = "{prefix}filename"

      • listStage

        List<String> listStage​(String stageName,
                       String prefix)
        list a non table stage and return a list of file names contained in given subdirectory
        Parameters:
        stageName - stage name
        prefix - prefix name
        Returns:
        a list of file names in given subdirectory, file name = "{prefix}filename"

      • put

        @Deprecated
void put​(String stageName,
         String fileName,
         String content)
        Deprecated.
        put a file to stage
        Parameters:
        fileName - file name
        content - file content
        stageName - stage name

      • putWithCache

        void putWithCache​(String stageName,
                  String fileName,
                  String content)
        put a file to stage. Cache credential for AWS, Azure and GCS storage.
        Parameters:
        fileName - file name which is of format (stageName/connectorName/tableName/partition/offsetStart_offsetEnd_timestamp)

        Check to find out how fileName is determined

        content - file content
        stageName - stage name

      • putToTableStage

        void putToTableStage​(String tableName,
                     String fileName,
                     byte[] content)
        put a file to table stage
        Parameters:
        tableName - table name
        fileName - file name
        content - file content

      • close

        void close()
        Close Connection

      • isClosed

        boolean isClosed()
        Returns:
        true is connection is closed

      • getConnectorName

        String getConnectorName()
        Returns:
        name of Kafka Connector instance

      • buildIngestService

        SnowflakeIngestionService buildIngestService​(String stageName,
                                             String pipeName)
        build ingest service instance for given stage and pipe
        Parameters:
        stageName - stage name
        pipeName - pipe name
        Returns:
        an instance of SnowflakeIngestService

      • getConnection

        Connection getConnection()
        Returns:
        the raw jdbc connection

      • appendMetaColIfNotExist

        void appendMetaColIfNotExist​(String tableName)
        Append a VARIANT type column "RECORD_METADATA" to the table if it is not present.

        This method is only called when schematization is enabled

        Parameters:
        tableName - table name

      • createTableWithOnlyMetadataColumn

        void createTableWithOnlyMetadataColumn​(String tableName)
        Create a table with only the RECORD_METADATA column. The rest of the columns might be added through schema evolution

        In the beginning of the function we will check if we have the permission to do schema evolution, and we will error out if we don't

        Parameters:
        tableName - table name

      • migrateStreamingChannelOffsetToken

        ChannelMigrateOffsetTokenResponseDTO migrateStreamingChannelOffsetToken​(String tableName,
                                                                        String sourceChannelName,
                                                                        String destinationChannelName)
        Migrate Streaming Channel offsetToken from a source Channel to a destination channel.

        Here, source channel is the new channel format we created here * @see Commit

        Destination channel is the original Format containing only topicName and partition number.

        We catch SQLException and JsonProcessingException that might happen in this method. The caller should always open the Old Channel format. This old channel format will also be the key to many HashMaps we will create. (For instance SnowflakeSinkServiceV2.partitionsToChannel)

        Parameters:
        tableName - Name of the table
        sourceChannelName - sourceChannel name from where the offset Token will be fetched. Channel with this name will also be deleted.
        destinationChannelName - destinationChannel name to where the offsetToken will be copied over.
        Returns:
        The DTO serialized from the migration response.