Package com.snowflake.kafka.connector.internal

Class PartitionBuffer<T>

  • Type Parameters:
    T - Return type of getData()
    Direct Known Subclasses:
    TopicPartitionChannel.StreamingBuffer
    public abstract class PartitionBuffer<T>
extends Object
    Abstract class which holds buffered data per partition including its start offset, end offset, size.

    The getData() method returns the data specific to the implementation.

    Buffer stores the converted records to Json format.

    Long lived buffer would mean the data in partition would stay across two put APIs since the buffer thresholds were not met.

    Please check respective implementation class for more details.

    • Constructor Detail

      • PartitionBuffer

        public PartitionBuffer()
        Public constructor.

    • Method Detail

      • getNumOfRecords

        public int getNumOfRecords()
        Returns:
        Number of records in this buffer

      • getBufferSizeBytes

        public long getBufferSizeBytes()
        Returns:
        Buffer size in bytes

      • getFirstOffset

        public long getFirstOffset()
        Returns:
        First offset number in this buffer

      • getLastOffset

        public long getLastOffset()
        Returns:
        Last offset number in this buffer

      • setNumOfRecords

        public void setNumOfRecords​(int numOfRecords)
        Parameters:
        numOfRecords - Updates number of records (Usually by 1)

      • setBufferSizeBytes

        public void setBufferSizeBytes​(long bufferSizeBytes)
        Parameters:
        bufferSizeBytes - Updates sum of size of records present in this buffer (Bytes)

      • setFirstOffset

        public void setFirstOffset​(long firstOffset)
        Parameters:
        firstOffset - First offset no to set in this buffer

      • setLastOffset

        public void setLastOffset​(long lastOffset)
        Parameters:
        lastOffset - Last offset no to set in this buffer

      • isEmpty

        public boolean isEmpty()
        Returns:
        true if buffer is empty

      • insert

        public abstract void insert​(org.apache.kafka.connect.sink.SinkRecord record)
        Inserts the row into Buffer.
        Parameters:
        record - the record which kafka sends to KC. We also convert the record to a format which is understood by Snowflake Table

      • getData

        public abstract T getData()
        Return the data that was buffered because buffer threshold might have been reached
        Returns:
        respective data type implemented by the class.

      • getSinkRecords

        public abstract List<org.apache.kafka.connect.sink.SinkRecord> getSinkRecords()
        TODO:SNOW-552576 Avoid extra memory in buffer.
        Returns:
        the sinkrecords corresponding to this buffer