net.snowflake.client.jdbc

Class SnowflakeFileTransferAgent

java.lang.Object

net.snowflake.client.jdbc.SnowflakeFileTransferAgent

All Implemented Interfaces:

SnowflakeFixedView

public class SnowflakeFileTransferAgent
extends Object
implements SnowflakeFixedView

Class for uploading/downloading files

Author:

jhuang

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SnowflakeFileTransferAgent.CommandType
class	SnowflakeFileTransferAgent.DownloadCommandEncryptionFacade
class	SnowflakeFileTransferAgent.DownloadCommandFacade A class for encapsulating the columns to return for the download command
static class	SnowflakeFileTransferAgent.ResultStatus Result status enum
static class	SnowflakeFileTransferAgent.UploadColumns A class for encapsulating the columns to return for the upload command
class	SnowflakeFileTransferAgent.UploadCommandEncryptionFacade
class	SnowflakeFileTransferAgent.UploadCommandFacade

Field Summary

Fields

Modifier and Type	Field and Description
static String	SRC_FILE_NAME_FOR_STREAM

Constructor Summary

Constructors

Constructor and Description
SnowflakeFileTransferAgent(String command, SFSession session, SFStatement statement)

Method Summary

Modifier and Type	Method and Description
void	cancel()
List<SnowflakeColumnMetadata>	describeColumns(SFSession session) Describe the metadata of a fixed view.
boolean	execute()
static net.snowflake.client.jdbc.SnowflakeFileTransferAgent.remoteLocation	extractLocationAndPath(String stageLocationPath) A small helper for extracting location name and path from full location path
SnowflakeFileTransferAgent.CommandType	getCommandType()
static Callable<Void>	getDownloadFileCallable(StageInfo stage, String srcFilePath, String localLocation, Map<String,net.snowflake.client.jdbc.SnowflakeFileTransferAgent.FileMetadata> fileMetadataMap, SnowflakeStorageClient client, SFSession session, String command, int parallel, net.snowflake.common.core.RemoteStoreFileEncryptionMaterial encMat, String presignedUrl) A callable that can be executed in a separate thread using executor service.
List<net.snowflake.common.core.RemoteStoreFileEncryptionMaterial>	getEncryptionMaterial()
List<SnowflakeFileTransferMetadata>	getFileTransferMetadatas() This is API function to retrieve the File Transfer Metadatas.
List<Object>	getNextRow()
Object	getResultSet()
Map<String,net.snowflake.common.core.RemoteStoreFileEncryptionMaterial>	getSrcToMaterialsMap()
Map<String,String>	getSrcToPresignedUrlMap()
Map<?,?>	getStageCredentials()
StageInfo	getStageInfo()
String	getStageLocation()
int	getTotalRows()
static Callable<Void>	getUploadFileCallable(StageInfo stage, String srcFilePath, net.snowflake.client.jdbc.SnowflakeFileTransferAgent.FileMetadata metadata, SnowflakeStorageClient client, SFSession session, String command, InputStream inputStream, boolean sourceFromStream, int parallel, File srcFile, net.snowflake.common.core.RemoteStoreFileEncryptionMaterial encMat) A callable that can be executed in a separate thread using exeuctor service.
static void	renewExpiredToken(SFSession session, String command, SnowflakeStorageClient client) This static method is called when we are handling an expired token exception It retrieves a fresh token from GS and then calls .renew() on the storage client to refresh itself with the new token
void	setCompressSourceFromStream(boolean compressSourceFromStream)
void	setDestFileNameForStreamSource(String destFileNameForStreamSource)
void	setSourceStream(InputStream sourceStream)
static void	throwJCEMissingError(String operation, Exception ex)
static void	uploadWithoutConnection(SnowflakeFileTransferConfig config) Static API function to upload data without JDBC session.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SRC_FILE_NAME_FOR_STREAM

public static final String SRC_FILE_NAME_FOR_STREAM

See Also:

Constant Field Values

Constructor Detail

SnowflakeFileTransferAgent

public SnowflakeFileTransferAgent(String command,
                                  SFSession session,
                                  SFStatement statement)
                           throws SnowflakeSQLException

Throws:

SnowflakeSQLException

Method Detail

getStageInfo

public StageInfo getStageInfo()

getStageCredentials

public Map<?,?> getStageCredentials()

getEncryptionMaterial

public List<net.snowflake.common.core.RemoteStoreFileEncryptionMaterial> getEncryptionMaterial()

getSrcToMaterialsMap

public Map<String,net.snowflake.common.core.RemoteStoreFileEncryptionMaterial> getSrcToMaterialsMap()

getSrcToPresignedUrlMap

public Map<String,String> getSrcToPresignedUrlMap()

getStageLocation

public String getStageLocation()

getUploadFileCallable

public static Callable<Void> getUploadFileCallable(StageInfo stage,
                                                   String srcFilePath,
                                                   net.snowflake.client.jdbc.SnowflakeFileTransferAgent.FileMetadata metadata,
                                                   SnowflakeStorageClient client,
                                                   SFSession session,
                                                   String command,
                                                   InputStream inputStream,
                                                   boolean sourceFromStream,
                                                   int parallel,
                                                   File srcFile,
                                                   net.snowflake.common.core.RemoteStoreFileEncryptionMaterial encMat)

A callable that can be executed in a separate thread using exeuctor service.

The callable does compression if needed and upload the result to the table's staging area.

Parameters:

stage - information about the stage

srcFilePath - source file path

metadata - file metadata

client - client object used to communicate with c3

session - session object

command - command string

inputStream - null if upload source is file

sourceFromStream - whether upload source is file or stream

parallel - number of threads for parallel uploading

srcFile - source file name

encMat - not null if encryption is required

Returns:

a callable that uploading file to the remote store

getDownloadFileCallable

public static Callable<Void> getDownloadFileCallable(StageInfo stage,
                                                     String srcFilePath,
                                                     String localLocation,
                                                     Map<String,net.snowflake.client.jdbc.SnowflakeFileTransferAgent.FileMetadata> fileMetadataMap,
                                                     SnowflakeStorageClient client,
                                                     SFSession session,
                                                     String command,
                                                     int parallel,
                                                     net.snowflake.common.core.RemoteStoreFileEncryptionMaterial encMat,
                                                     String presignedUrl)

A callable that can be executed in a separate thread using executor service.

The callable download files from a stage location to a local location

Parameters:

stage - stage information

srcFilePath - path that stores the downloaded file

localLocation - local location

fileMetadataMap - file metadata map

client - remote store client

session - session object

command - command string

encMat - remote store encryption material

parallel - number of parallel threads for downloading

presignedUrl - Presigned URL for file download

Returns:

a callable responsible for downloading files

getFileTransferMetadatas

public List<SnowflakeFileTransferMetadata> getFileTransferMetadatas()
                                                             throws SnowflakeSQLException

This is API function to retrieve the File Transfer Metadatas.

NOTE: It only supports PUT on S3/AZURE/GCS

Returns:

The file transfer metadatas for to-be-transferred files.

Throws:

SnowflakeSQLException - if any error occurs

execute

public boolean execute()
                throws SQLException

Throws:

SQLException

cancel

public void cancel()

uploadWithoutConnection

public static void uploadWithoutConnection(SnowflakeFileTransferConfig config)
                                    throws Exception

Static API function to upload data without JDBC session.

NOTE: This function is developed based on getUploadFileCallable().

Parameters:

config - Configuration to upload a file to cloud storage

Throws:

Exception - if error occurs while data upload.

renewExpiredToken

public static void renewExpiredToken(SFSession session,
                                     String command,
                                     SnowflakeStorageClient client)
                              throws SnowflakeSQLException

This static method is called when we are handling an expired token exception It retrieves a fresh token from GS and then calls .renew() on the storage client to refresh itself with the new token

Parameters:

session - a session object

command - a command to be retried

client - a Snowflake Storage client object

Throws:

SnowflakeSQLException - if any error occurs

extractLocationAndPath

public static net.snowflake.client.jdbc.SnowflakeFileTransferAgent.remoteLocation extractLocationAndPath(String stageLocationPath)

A small helper for extracting location name and path from full location path

Parameters:

stageLocationPath - stage location

Returns:

remoteLocation object

describeColumns

public List<SnowflakeColumnMetadata> describeColumns(SFSession session)
                                              throws Exception

Describe the metadata of a fixed view.

Specified by:

describeColumns in interface SnowflakeFixedView

Returns:

list of column meta data

Throws:

Exception - failed to construct list

getNextRow

public List<Object> getNextRow()
                        throws Exception

Specified by:

getNextRow in interface SnowflakeFixedView

Throws:

Exception

getResultSet

public Object getResultSet()
                    throws SnowflakeSQLException

Throws:

SnowflakeSQLException

getCommandType

public SnowflakeFileTransferAgent.CommandType getCommandType()

setSourceStream

public void setSourceStream(InputStream sourceStream)

setDestFileNameForStreamSource

public void setDestFileNameForStreamSource(String destFileNameForStreamSource)

setCompressSourceFromStream

public void setCompressSourceFromStream(boolean compressSourceFromStream)

throwJCEMissingError

public static void throwJCEMissingError(String operation,
                                        Exception ex)
                                 throws SnowflakeSQLException

Throws:

SnowflakeSQLException

getTotalRows

public int getTotalRows()

Specified by:

getTotalRows in interface SnowflakeFixedView