net.snowflake.ingest.utils

Class ParameterProvider

java.lang.Object

net.snowflake.ingest.utils.ParameterProvider

public class ParameterProvider
extends Object

Utility class to provide configurable constants

Field Summary

Fields

Modifier and Type	Field and Description
static String	BLOB_FORMAT_VERSION
static Constants.BdecVersion	BLOB_FORMAT_VERSION_DEFAULT
static String	BLOB_UPLOAD_MAX_RETRY_COUNT
static int	BLOB_UPLOAD_MAX_RETRY_COUNT_DEFAULT
static String	BUFFER_FLUSH_CHECK_INTERVAL_IN_MILLIS
static long	BUFFER_FLUSH_CHECK_INTERVAL_IN_MILLIS_DEFAULT
static String	BUFFER_FLUSH_INTERVAL_IN_MILLIS
static long	BUFFER_FLUSH_INTERVAL_IN_MILLIS_DEFAULT
static String	ENABLE_PARQUET_INTERNAL_BUFFERING
static boolean	ENABLE_PARQUET_INTERNAL_BUFFERING_DEFAULT
static String	ENABLE_SNOWPIPE_STREAMING_METRICS
static String	INSERT_THROTTLE_INTERVAL_IN_MILLIS
static long	INSERT_THROTTLE_INTERVAL_IN_MILLIS_DEFAULT
static String	INSERT_THROTTLE_THRESHOLD_IN_BYTES
static int	INSERT_THROTTLE_THRESHOLD_IN_BYTES_DEFAULT
static String	INSERT_THROTTLE_THRESHOLD_IN_PERCENTAGE
static int	INSERT_THROTTLE_THRESHOLD_IN_PERCENTAGE_DEFAULT
static String	IO_TIME_CPU_RATIO
static int	IO_TIME_CPU_RATIO_DEFAULT
static String	MAX_MEMORY_LIMIT_IN_BYTES
static long	MAX_MEMORY_LIMIT_IN_BYTES_DEFAULT
static boolean	SNOWPIPE_STREAMING_METRICS_DEFAULT

Constructor Summary

Constructors

Constructor and Description
ParameterProvider() Empty constructor for tests
ParameterProvider(Map<String,Object> parameterOverrides, Properties props) Constructor.

Method Summary

Modifier and Type	Method and Description
Constants.BdecVersion	getBlobFormatVersion()
int	getBlobUploadMaxRetryCount()
long	getBufferFlushCheckIntervalInMs()
long	getBufferFlushIntervalInMs()
boolean	getEnableParquetInternalBuffering()
long	getInsertThrottleIntervalInMs()
int	getInsertThrottleThresholdInBytes()
int	getInsertThrottleThresholdInPercentage()
int	getIOTimeCpuRatio()
long	getMaxMemoryLimitInBytes()
boolean	hasEnabledSnowpipeStreamingMetrics()
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

BUFFER_FLUSH_INTERVAL_IN_MILLIS

public static final String BUFFER_FLUSH_INTERVAL_IN_MILLIS

BUFFER_FLUSH_CHECK_INTERVAL_IN_MILLIS

public static final String BUFFER_FLUSH_CHECK_INTERVAL_IN_MILLIS

INSERT_THROTTLE_INTERVAL_IN_MILLIS

public static final String INSERT_THROTTLE_INTERVAL_IN_MILLIS

INSERT_THROTTLE_THRESHOLD_IN_PERCENTAGE

public static final String INSERT_THROTTLE_THRESHOLD_IN_PERCENTAGE

INSERT_THROTTLE_THRESHOLD_IN_BYTES

public static final String INSERT_THROTTLE_THRESHOLD_IN_BYTES

ENABLE_SNOWPIPE_STREAMING_METRICS

public static final String ENABLE_SNOWPIPE_STREAMING_METRICS

BLOB_FORMAT_VERSION

public static final String BLOB_FORMAT_VERSION

IO_TIME_CPU_RATIO

public static final String IO_TIME_CPU_RATIO

BLOB_UPLOAD_MAX_RETRY_COUNT

public static final String BLOB_UPLOAD_MAX_RETRY_COUNT

MAX_MEMORY_LIMIT_IN_BYTES

public static final String MAX_MEMORY_LIMIT_IN_BYTES

ENABLE_PARQUET_INTERNAL_BUFFERING

public static final String ENABLE_PARQUET_INTERNAL_BUFFERING

BUFFER_FLUSH_INTERVAL_IN_MILLIS_DEFAULT

public static final long BUFFER_FLUSH_INTERVAL_IN_MILLIS_DEFAULT

See Also:

Constant Field Values

BUFFER_FLUSH_CHECK_INTERVAL_IN_MILLIS_DEFAULT

public static final long BUFFER_FLUSH_CHECK_INTERVAL_IN_MILLIS_DEFAULT

See Also:

Constant Field Values

INSERT_THROTTLE_INTERVAL_IN_MILLIS_DEFAULT

public static final long INSERT_THROTTLE_INTERVAL_IN_MILLIS_DEFAULT

See Also:

Constant Field Values

INSERT_THROTTLE_THRESHOLD_IN_PERCENTAGE_DEFAULT

public static final int INSERT_THROTTLE_THRESHOLD_IN_PERCENTAGE_DEFAULT

See Also:

Constant Field Values

INSERT_THROTTLE_THRESHOLD_IN_BYTES_DEFAULT

public static final int INSERT_THROTTLE_THRESHOLD_IN_BYTES_DEFAULT

See Also:

Constant Field Values

SNOWPIPE_STREAMING_METRICS_DEFAULT

public static final boolean SNOWPIPE_STREAMING_METRICS_DEFAULT

See Also:

Constant Field Values

BLOB_FORMAT_VERSION_DEFAULT

public static final Constants.BdecVersion BLOB_FORMAT_VERSION_DEFAULT

IO_TIME_CPU_RATIO_DEFAULT

public static final int IO_TIME_CPU_RATIO_DEFAULT

See Also:

Constant Field Values

BLOB_UPLOAD_MAX_RETRY_COUNT_DEFAULT

public static final int BLOB_UPLOAD_MAX_RETRY_COUNT_DEFAULT

See Also:

Constant Field Values

MAX_MEMORY_LIMIT_IN_BYTES_DEFAULT

public static final long MAX_MEMORY_LIMIT_IN_BYTES_DEFAULT

See Also:

Constant Field Values

ENABLE_PARQUET_INTERNAL_BUFFERING_DEFAULT

public static final boolean ENABLE_PARQUET_INTERNAL_BUFFERING_DEFAULT

See Also:

Constant Field Values

Constructor Detail

ParameterProvider

public ParameterProvider(Map<String,Object> parameterOverrides,
                         Properties props)

Constructor. Takes properties from profile file and properties from client constructor and resolves final parameter value

Parameters:

parameterOverrides - Map of parameter name to value

props - Properties from profile file

ParameterProvider

public ParameterProvider()

Empty constructor for tests

Method Detail

getBufferFlushIntervalInMs

public long getBufferFlushIntervalInMs()

Returns:

Longest interval in milliseconds between buffer flushes

getBufferFlushCheckIntervalInMs

public long getBufferFlushCheckIntervalInMs()

Returns:

Time in milliseconds between checks to see if the buffer should be flushed

getInsertThrottleIntervalInMs

public long getInsertThrottleIntervalInMs()

Returns:

Duration in milliseconds to delay data insertion to the buffer when throttled

getInsertThrottleThresholdInPercentage

public int getInsertThrottleThresholdInPercentage()

Returns:

Percent of free total memory at which we throttle row inserts

getInsertThrottleThresholdInBytes

public int getInsertThrottleThresholdInBytes()

Returns:

Absolute size in bytes of free total memory at which we throttle row inserts

hasEnabledSnowpipeStreamingMetrics

public boolean hasEnabledSnowpipeStreamingMetrics()

Returns:

true if jmx metrics are enabled for a client

getBlobFormatVersion

public Constants.BdecVersion getBlobFormatVersion()

Returns:

Blob format version: 1 (arrow stream write mode), 2 (arrow file write mode) etc

getIOTimeCpuRatio

public int getIOTimeCpuRatio()

Returns:

the IO_TIME/CPU ratio that we will use to determine the number of buildAndUpload threads

getBlobUploadMaxRetryCount

public int getBlobUploadMaxRetryCount()

Returns:

the max retry count when waiting for a blob upload task to finish

getMaxMemoryLimitInBytes

public long getMaxMemoryLimitInBytes()

Returns:

The max memory limit in bytes

getEnableParquetInternalBuffering

public boolean getEnableParquetInternalBuffering()

Returns:

Return whether memory optimization for Parquet is enabled.

toString

public String toString()

Overrides:

toString in class Object