Package com.snowflake.kafka.connector.internal.streaming

Class StreamingUtils

java.lang.Object

com.snowflake.kafka.connector.internal.streaming.StreamingUtils

public class StreamingUtils
extends Object

Field Summary

Fields

Modifier and Type	Field	Description
protected static Duration	DURATION_BETWEEN_GET_OFFSET_TOKEN_RETRY
protected static int	MAX_GET_OFFSET_TOKEN_RETRIES
static int	MAX_RECORD_OVERHEAD_BYTES
protected static long	STREAMING_BUFFER_BYTES_DEFAULT	Keeping this default as ~ 20MB.
protected static long	STREAMING_BUFFER_COUNT_RECORDS_DEFAULT
static long	STREAMING_BUFFER_FLUSH_TIME_DEFAULT_SEC
static long	STREAMING_BUFFER_FLUSH_TIME_MINIMUM_SEC
static String	STREAMING_CONSTANT_AUTHORIZATION_TYPE
static String	STREAMING_CONSTANT_JWT
static String	STREAMING_CONSTANT_OAUTH
static String	STREAMING_CONSTANT_OAUTH_CLIENT_ID
static String	STREAMING_CONSTANT_OAUTH_CLIENT_SECRET
static String	STREAMING_CONSTANT_OAUTH_REFRESH_TOKEN

Constructor Summary

Constructors

Constructor	Description
StreamingUtils()

Method Summary

Modifier and Type	Method	Description
static Map<String,String>	convertConfigForStreamingClient(Map<String,String> connectorConfig)
static String	getDlqTopicName(Map<String,String> sfConnectorConfig)
static boolean	logErrors(Map<String,String> sfConnectorConfig)
static boolean	tolerateErrors(Map<String,String> sfConnectorConfig)
static com.google.common.collect.ImmutableMap<String,String>	validateStreamingSnowpipeConfig(Map<String,String> inputConfig)	Validate Streaming snowpipe related config provided by config(customer's config)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DURATION_BETWEEN_GET_OFFSET_TOKEN_RETRY

protected static final Duration DURATION_BETWEEN_GET_OFFSET_TOKEN_RETRY

MAX_GET_OFFSET_TOKEN_RETRIES

protected static final int MAX_GET_OFFSET_TOKEN_RETRIES

See Also:

Constant Field Values

STREAMING_BUFFER_FLUSH_TIME_MINIMUM_SEC

public static final long STREAMING_BUFFER_FLUSH_TIME_MINIMUM_SEC

STREAMING_BUFFER_FLUSH_TIME_DEFAULT_SEC

public static final long STREAMING_BUFFER_FLUSH_TIME_DEFAULT_SEC

STREAMING_BUFFER_COUNT_RECORDS_DEFAULT

protected static final long STREAMING_BUFFER_COUNT_RECORDS_DEFAULT

See Also:

Constant Field Values

STREAMING_BUFFER_BYTES_DEFAULT

protected static final long STREAMING_BUFFER_BYTES_DEFAULT

Keeping this default as ~ 20MB.

Logic behind this optimium value is we will do gzip compression and json to UTF conversion which will account to almost 95% compression.

1 MB is an ideal size for streaming ingestion so 95% if 20MB = 1MB

See Also:

Constant Field Values

MAX_RECORD_OVERHEAD_BYTES

public static final int MAX_RECORD_OVERHEAD_BYTES

See Also:

Constant Field Values

STREAMING_CONSTANT_AUTHORIZATION_TYPE

public static final String STREAMING_CONSTANT_AUTHORIZATION_TYPE

See Also:

Constant Field Values

STREAMING_CONSTANT_JWT

public static final String STREAMING_CONSTANT_JWT

See Also:

Constant Field Values

STREAMING_CONSTANT_OAUTH

public static final String STREAMING_CONSTANT_OAUTH

See Also:

Constant Field Values

STREAMING_CONSTANT_OAUTH_CLIENT_ID

public static final String STREAMING_CONSTANT_OAUTH_CLIENT_ID

See Also:

Constant Field Values

STREAMING_CONSTANT_OAUTH_CLIENT_SECRET

public static final String STREAMING_CONSTANT_OAUTH_CLIENT_SECRET

See Also:

Constant Field Values

STREAMING_CONSTANT_OAUTH_REFRESH_TOKEN

public static final String STREAMING_CONSTANT_OAUTH_REFRESH_TOKEN

See Also:

Constant Field Values

Constructor Detail

StreamingUtils

public StreamingUtils()

Method Detail

convertConfigForStreamingClient

public static Map<String,String> convertConfigForStreamingClient(Map<String,String> connectorConfig)

tolerateErrors

public static boolean tolerateErrors(Map<String,String> sfConnectorConfig)

logErrors

public static boolean logErrors(Map<String,String> sfConnectorConfig)

getDlqTopicName

public static String getDlqTopicName(Map<String,String> sfConnectorConfig)

validateStreamingSnowpipeConfig

public static com.google.common.collect.ImmutableMap<String,String> validateStreamingSnowpipeConfig(Map<String,String> inputConfig)

Validate Streaming snowpipe related config provided by config(customer's config)

Parameters:

inputConfig - given in connector json file

Returns:

map of invalid parameters