Package net.snowflake.client.jdbc.cloud.storage

Class SnowflakeAzureClient

java.lang.Object

net.snowflake.client.jdbc.cloud.storage.SnowflakeAzureClient

All Implemented Interfaces:

SnowflakeStorageClient

public class SnowflakeAzureClient
extends Object
implements SnowflakeStorageClient

Encapsulates the Azure Storage client and all Azure Storage operations and logic

Author:

lgiakoumakis

Method Summary

Modifier and Type	Method	Description
void	addDigestMetadata(StorageObjectMetadata meta, String digest)	Adds digest metadata to the StorageObjectMetadata object
void	addEncryptionMetadata(StorageObjectMetadata meta, MatDesc matDesc, byte[] ivData, byte[] encKeK, long contentLength)	Adds encryption metadata to the StorageObjectMetadata object
void	addStreamingIngestMetadata(StorageObjectMetadata meta, String clientName, String clientKey)	Adds streaming ingest metadata to the StorageObjectMetadata object, used for streaming ingest per client billing calculation
static SnowflakeAzureClient	createSnowflakeAzureClient(StageInfo stage, net.snowflake.common.core.RemoteStoreFileEncryptionMaterial encMat, SFBaseSession sfSession)
void	download(SFSession session, String command, String localLocation, String destFileName, int parallelism, String remoteStorageLocation, String stageFilePath, String stageRegion, String presignedUrl)	Download a file from remote storage.
InputStream	downloadToStream(SFSession session, String command, int parallelism, String remoteStorageLocation, String stageFilePath, String stageRegion, String presignedUrl)	Download a file from remote storage
String	getDigestMetadata(StorageObjectMetadata meta)	Gets digest metadata to the StorageObjectMetadata object
int	getEncryptionKeySize()
String	getMatdescKey()	Returns the material descriptor key
int	getMaxRetries()
StorageObjectMetadata	getObjectMetadata(String remoteStorageLocation, String prefix)	Returns the metadata properties for a remote storage object
int	getRetryBackoffMaxExponent()	Returns the max exponent for multiplying backoff with the power of 2, the value of 4 will give us 16secs as the max number of time to sleep before retry
int	getRetryBackoffMin()
String	getStreamingIngestClientKey(StorageObjectMetadata meta)	Gets streaming ingest client key to the StorageObjectMetadata object
String	getStreamingIngestClientName(StorageObjectMetadata meta)	Gets streaming ingest client name to the StorageObjectMetadata object
void	handleStorageException(Exception ex, int retryCount, String operation, SFSession session, String command)	Handles exceptions thrown by Azure Storage
boolean	isEncrypting()
StorageObjectSummaryCollection	listObjects(String remoteStorageLocation, String prefix)	For a set of remote storage objects under a remote location and a given prefix/path returns their properties wrapped in ObjectSummary objects
void	renew(Map<?,?> stageCredentials)	Re-creates the encapsulated storage client with a fresh access token
void	shutdown()	shuts down the client
void	upload(SFSession session, String command, int parallelism, boolean uploadFromStream, String remoteStorageLocation, File srcFile, String destFileName, InputStream inputStream, FileBackedOutputStream fileBackedOutputStream, StorageObjectMetadata meta, String stageRegion, String presignedUrl)	Upload a file/stream to remote storage

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface net.snowflake.client.jdbc.cloud.storage.SnowflakeStorageClient

requirePresignedUrl, uploadWithPresignedUrlWithoutConnection

Method Detail

createSnowflakeAzureClient

public static SnowflakeAzureClient createSnowflakeAzureClient(StageInfo stage,
                                                              net.snowflake.common.core.RemoteStoreFileEncryptionMaterial encMat,
                                                              SFBaseSession sfSession)
                                                       throws SnowflakeSQLException

Throws:

SnowflakeSQLException

getMaxRetries

public int getMaxRetries()

Specified by:

getMaxRetries in interface SnowflakeStorageClient

Returns:

Returns the Max number of retry attempts

getRetryBackoffMaxExponent

public int getRetryBackoffMaxExponent()

Description copied from interface: SnowflakeStorageClient

Returns the max exponent for multiplying backoff with the power of 2, the value of 4 will give us 16secs as the max number of time to sleep before retry

Specified by:

getRetryBackoffMaxExponent in interface SnowflakeStorageClient

Returns:

Returns the exponent

getRetryBackoffMin

public int getRetryBackoffMin()

Specified by:

getRetryBackoffMin in interface SnowflakeStorageClient

Returns:

Returns the min number of milliseconds to sleep before retry

isEncrypting

public boolean isEncrypting()

Specified by:

isEncrypting in interface SnowflakeStorageClient

Returns:

Returns true if encryption is enabled

getEncryptionKeySize

public int getEncryptionKeySize()

Specified by:

getEncryptionKeySize in interface SnowflakeStorageClient

Returns:

Returns the size of the encryption key

renew

public void renew(Map<?,?> stageCredentials)
           throws SnowflakeSQLException

Re-creates the encapsulated storage client with a fresh access token

Specified by:

renew in interface SnowflakeStorageClient

Parameters:

stageCredentials - a Map (as returned by GS) which contains the new credential properties

Throws:

SnowflakeSQLException - failure to renew the client

shutdown

public void shutdown()

shuts down the client

Specified by:

shutdown in interface SnowflakeStorageClient

listObjects

public StorageObjectSummaryCollection listObjects(String remoteStorageLocation,
                                                  String prefix)
                                           throws StorageProviderException

For a set of remote storage objects under a remote location and a given prefix/path returns their properties wrapped in ObjectSummary objects

Specified by:

listObjects in interface SnowflakeStorageClient

Parameters:

remoteStorageLocation - location, i.e. container for Azure

prefix - the prefix/path to list under

Returns:

a collection of storage summary objects

Throws:

StorageProviderException - Azure storage exception

getObjectMetadata

public StorageObjectMetadata getObjectMetadata(String remoteStorageLocation,
                                               String prefix)
                                        throws StorageProviderException

Returns the metadata properties for a remote storage object

Specified by:

getObjectMetadata in interface SnowflakeStorageClient

Parameters:

remoteStorageLocation - location, i.e. bucket for S3

prefix - the prefix/path of the object to retrieve

Returns:

storage metadata object

Throws:

StorageProviderException - azure storage exception

download

public void download(SFSession session,
                     String command,
                     String localLocation,
                     String destFileName,
                     int parallelism,
                     String remoteStorageLocation,
                     String stageFilePath,
                     String stageRegion,
                     String presignedUrl)
              throws SnowflakeSQLException

Download a file from remote storage.

Specified by:

download in interface SnowflakeStorageClient

Parameters:

session - session object

command - command to download file

localLocation - local file path

destFileName - destination file name

parallelism - number of threads for parallel downloading

remoteStorageLocation - remote storage location, i.e. bucket for S3

stageFilePath - stage file path

stageRegion - region name where the stage persists

presignedUrl - Unused in Azure

Throws:

SnowflakeSQLException - download failure

downloadToStream

public InputStream downloadToStream(SFSession session,
                                    String command,
                                    int parallelism,
                                    String remoteStorageLocation,
                                    String stageFilePath,
                                    String stageRegion,
                                    String presignedUrl)
                             throws SnowflakeSQLException

Download a file from remote storage

Specified by:

downloadToStream in interface SnowflakeStorageClient

Parameters:

session - session object

command - command to download file

parallelism - number of threads for parallel downloading

remoteStorageLocation - remote storage location, i.e. bucket for s3

stageFilePath - stage file path

stageRegion - region name where the stage persists

presignedUrl - Unused in Azure

Returns:

input file stream

Throws:

SnowflakeSQLException - when download failure

upload

public void upload(SFSession session,
                   String command,
                   int parallelism,
                   boolean uploadFromStream,
                   String remoteStorageLocation,
                   File srcFile,
                   String destFileName,
                   InputStream inputStream,
                   FileBackedOutputStream fileBackedOutputStream,
                   StorageObjectMetadata meta,
                   String stageRegion,
                   String presignedUrl)
            throws SnowflakeSQLException

Upload a file/stream to remote storage

Specified by:

upload in interface SnowflakeStorageClient

Parameters:

session - session object

command - upload command

parallelism - number of threads for parallel uploading

uploadFromStream - true if upload source is stream

remoteStorageLocation - storage container name

srcFile - source file if not uploading from a stream

destFileName - file name on remote storage after upload

inputStream - stream used for uploading if fileBackedOutputStream is null

fileBackedOutputStream - stream used for uploading if not null

meta - object meta data

stageRegion - region name where the stage persists

presignedUrl - Unused in Azure

Throws:

SnowflakeSQLException - if upload failed even after retry

handleStorageException

public void handleStorageException(Exception ex,
                                   int retryCount,
                                   String operation,
                                   SFSession session,
                                   String command)
                            throws SnowflakeSQLException

Handles exceptions thrown by Azure Storage

Specified by:

handleStorageException in interface SnowflakeStorageClient

Parameters:

ex - the exception to handle

retryCount - current number of retries, incremented by the caller before each call

operation - string that indicates the function/operation that was taking place, when the exception was raised, for example "upload"

session - the current SFSession object used by the client

command - the command attempted at the time of the exception

Throws:

SnowflakeSQLException - exceptions not handled

getMatdescKey

public String getMatdescKey()

Returns the material descriptor key

Specified by:

getMatdescKey in interface SnowflakeStorageClient

Returns:

the material descriptor key

addEncryptionMetadata

public void addEncryptionMetadata(StorageObjectMetadata meta,
                                  MatDesc matDesc,
                                  byte[] ivData,
                                  byte[] encKeK,
                                  long contentLength)

Adds encryption metadata to the StorageObjectMetadata object

Specified by:

addEncryptionMetadata in interface SnowflakeStorageClient

Parameters:

meta - the storage metadata object to add the encryption info to

matDesc - the material descriptor

ivData - the initialization vector

encKeK - the key encryption key

contentLength - the length of the encrypted content

addDigestMetadata

public void addDigestMetadata(StorageObjectMetadata meta,
                              String digest)

Adds digest metadata to the StorageObjectMetadata object

Specified by:

addDigestMetadata in interface SnowflakeStorageClient

Parameters:

meta - the storage metadata object to add the digest to

digest - the digest metadata to add

getDigestMetadata

public String getDigestMetadata(StorageObjectMetadata meta)

Gets digest metadata to the StorageObjectMetadata object

Specified by:

getDigestMetadata in interface SnowflakeStorageClient

Parameters:

meta - the metadata object to extract the digest metadata from

Returns:

the digest metadata value

addStreamingIngestMetadata

public void addStreamingIngestMetadata(StorageObjectMetadata meta,
                                       String clientName,
                                       String clientKey)

Adds streaming ingest metadata to the StorageObjectMetadata object, used for streaming ingest per client billing calculation

Specified by:

addStreamingIngestMetadata in interface SnowflakeStorageClient

Parameters:

meta - the storage metadata object to add the digest to

clientName - streaming ingest client name

clientKey - streaming ingest client key, provided by Snowflake

getStreamingIngestClientName

public String getStreamingIngestClientName(StorageObjectMetadata meta)

Gets streaming ingest client name to the StorageObjectMetadata object

Specified by:

getStreamingIngestClientName in interface SnowflakeStorageClient

getStreamingIngestClientKey

public String getStreamingIngestClientKey(StorageObjectMetadata meta)

Gets streaming ingest client key to the StorageObjectMetadata object

Specified by:

getStreamingIngestClientKey in interface SnowflakeStorageClient