Package com.snowflake.kafka.connector.internal

Class BufferThreshold

java.lang.Object

com.snowflake.kafka.connector.internal.BufferThreshold

Direct Known Subclasses:

StreamingBufferThreshold

public abstract class BufferThreshold
extends Object

Helper class associated to runtime of Kafka Connect which can help to identify if there is a need to flush the buffered records.

Constructor Summary

Constructors

Constructor	Description
BufferThreshold(IngestionMethodConfig ingestionMethodConfig, long bufferFlushTimeThreshold, long bufferByteSizeThreshold, long bufferRecordCountThreshold)	Public constructor

Method Summary

Modifier and Type	Method	Description
long	getFlushTimeThresholdSeconds()
boolean	shouldFlushOnBufferByteSize(long currBufferByteSize)	Returns true the buffer should flush based on the current buffer byte size
boolean	shouldFlushOnBufferRecordCount(long currentBufferedRecordCount)	Returns true the buffer should flush based on the current buffer record count
boolean	shouldFlushOnBufferTime(long previousFlushTimeStampMs)	Returns true the buffer should flush based on the last flush time
String	toString()
static com.google.common.collect.ImmutableMap<String,String>	validateBufferThreshold(Map<String,String> providedSFConnectorConfig, IngestionMethodConfig ingestionMethodConfig)	Check if provided snowflake kafka connector buffer properties are within permissible values.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

BufferThreshold

public BufferThreshold(IngestionMethodConfig ingestionMethodConfig,
                       long bufferFlushTimeThreshold,
                       long bufferByteSizeThreshold,
                       long bufferRecordCountThreshold)

Public constructor

Parameters:

ingestionMethodConfig - enum accepting ingestion method (selected in config json)

bufferFlushTimeThreshold - flush time threshold in seconds given in connector config

bufferByteSizeThreshold - buffer size threshold in bytes given in connector config

bufferRecordCountThreshold - record count threshold in number of kafka records given in connector config

Method Detail

shouldFlushOnBufferByteSize

public boolean shouldFlushOnBufferByteSize(long currBufferByteSize)

Returns true the buffer should flush based on the current buffer byte size

Threshold is config parameter: SnowflakeSinkConnectorConfig.BUFFER_SIZE_BYTES

Parameters:

currBufferByteSize - current size of buffer in bytes

Returns:

true if the currByteSize > configByteSizeThreshold

shouldFlushOnBufferRecordCount

public boolean shouldFlushOnBufferRecordCount(long currentBufferedRecordCount)

Returns true the buffer should flush based on the current buffer record count

Threshold is config parameter: SnowflakeSinkConnectorConfig.BUFFER_COUNT_RECORDS

Parameters:

currentBufferedRecordCount - current size of buffer in number of kafka records

Returns:

true if the currRecordCount > configRecordCountThreshold

shouldFlushOnBufferTime

public boolean shouldFlushOnBufferTime(long previousFlushTimeStampMs)

Returns true the buffer should flush based on the last flush time

Threshold is config parameter: SnowflakeSinkConnectorConfig.BUFFER_FLUSH_TIME_SEC

Parameters:

previousFlushTimeStampMs - when the previous buffered records flushed

Returns:

true if currentTime - previousTime > configTimeThreshold

getFlushTimeThresholdSeconds

public long getFlushTimeThresholdSeconds()

Returns:

Get flush time threshold in seconds

validateBufferThreshold

public static com.google.common.collect.ImmutableMap<String,String> validateBufferThreshold(Map<String,String> providedSFConnectorConfig,
                                                                                                  IngestionMethodConfig ingestionMethodConfig)

Check if provided snowflake kafka connector buffer properties are within permissible values.

This method invokes three verifiers - Time based threshold, buffer size and buffer count threshold.

Parameters:

providedSFConnectorConfig - provided by customer

ingestionMethodConfig - ingestion method used. Check IngestionMethodConfig

Returns:

invalid config parameters, if exists

toString

public String toString()

Overrides:

toString in class Object