Package com.snowflake.kafka.connector.internal

Class SnowflakeInternalStage

java.lang.Object

com.snowflake.kafka.connector.internal.SnowflakeInternalStage

public class SnowflakeInternalStage
extends Object

Implementation of put API through JDBC's API uploadWithoutConnection.

We fetch the credentials and cache it for AWS and Azure. We will refresh if cache hits 30 mins (Cache Eviction)

For GCS, we dont have any cache, we will make a call to GS for every put API since we require presignedURL

Field Summary

Fields

Modifier and Type	Field	Description
static String	dummyPutCommandTemplateAWSAndAzure
static String	dummyPutCommandTemplateGCS
static String	dummyPutCommandToGetStageType

Constructor Summary

Constructors

Constructor	Description
SnowflakeInternalStage(net.snowflake.client.jdbc.SnowflakeConnectionV1 conn, long expirationTimeMillis, Properties proxyProperties)

Method Summary

Modifier and Type	Method	Description
net.snowflake.client.jdbc.cloud.storage.StageInfo.StageType	getStageType(String stageName)	Get the backend stage type, S3, Azure or GCS.
void	putWithCache(String stageName, String fullFilePath, String data, net.snowflake.client.jdbc.cloud.storage.StageInfo.StageType stageType)	Upload file to internal stage with previously cached credentials.
protected void	refreshCredentials(String stageName, net.snowflake.client.jdbc.cloud.storage.StageInfo.StageType stageType, String fullFilePath)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

dummyPutCommandTemplateGCS

public static String dummyPutCommandTemplateGCS

dummyPutCommandTemplateAWSAndAzure

public static String dummyPutCommandTemplateAWSAndAzure

dummyPutCommandToGetStageType

public static String dummyPutCommandToGetStageType

Constructor Detail

SnowflakeInternalStage

public SnowflakeInternalStage(net.snowflake.client.jdbc.SnowflakeConnectionV1 conn,
                              long expirationTimeMillis,
                              Properties proxyProperties)

Method Detail

getStageType

public net.snowflake.client.jdbc.cloud.storage.StageInfo.StageType getStageType(String stageName)

Get the backend stage type, S3, Azure or GCS. Involves one GS only when storageInfoCache has stage name missing.

Parameters:

stageName - name of the stage

Returns:

stage type

putWithCache

public void putWithCache(String stageName,
                         String fullFilePath,
                         String data,
                         net.snowflake.client.jdbc.cloud.storage.StageInfo.StageType stageType)

Upload file to internal stage with previously cached credentials. Refresh credential every 30 minutes for AWS and Azure. We do cache for GCS but we always refresh it for every put(every file upload)

If we pass in expired credentials, we will get expired credentials error from cloud.

JDBC itself should renew the token but looks like that part of the code is not working. https://github.com/snowflakedb/snowflake-jdbc/blob/master/src/main/java/net/snowflake/client/jdbc/cloud/storage/SnowflakeS3Client.java#L738

We are already doing a retry for this failure and renew the credentials from our end by calling tradition put API with connection.

Parameters:

stageName - Stage name

fullFilePath - Full file name to be uploaded

data - Data string to be uploaded

stageType - GCS, Azure or AWS

refreshCredentials

protected void refreshCredentials(String stageName,
                                  net.snowflake.client.jdbc.cloud.storage.StageInfo.StageType stageType,
                                  String fullFilePath)
                           throws net.snowflake.client.jdbc.SnowflakeSQLException

Throws:

net.snowflake.client.jdbc.SnowflakeSQLException