Package org.apache.pinot.spi.stream

Interface PartitionGroupConsumer

    • Method Summary

      Modifier and Type Method Description
      default StreamPartitionMsgOffset checkpoint​(StreamPartitionMsgOffset lastOffset)
      Checkpoints the consumption state of the stream partition group in the source This is useful in streaming systems that require preserving consumption state on the source in order to resume or replay consumption of data.
      MessageBatch fetchMessages​(StreamPartitionMsgOffset startOffset, StreamPartitionMsgOffset endOffset, int timeoutMs)
      Return messages from the stream partition group within the specified timeout The message may be fetched by actively polling the source or by retrieving from a pre-fetched buffer.
      default void start​(StreamPartitionMsgOffset startOffset)
      Starts a stream consumer This is useful in cases where starting the consumer involves preparing / initializing the source.

    • Method Detail

      • start

        default void start​(StreamPartitionMsgOffset startOffset)
        Starts a stream consumer This is useful in cases where starting the consumer involves preparing / initializing the source. A typical example is that of an asynchronous / non-poll based consumption model where this method will be used to setup or initialize the consumer to fetch messages from the source stream. Poll-based consumers can optionally use this to prefetch metadata from the source. This method should be invoked by the caller before trying to invoke fetchMessages(StreamPartitionMsgOffset, StreamPartitionMsgOffset, int).
        Parameters:
        startOffset - Offset (inclusive) at which the consumption should begin

      • fetchMessages

        MessageBatch fetchMessages​(StreamPartitionMsgOffset startOffset,
                           StreamPartitionMsgOffset endOffset,
                           int timeoutMs)
                    throws TimeoutException
        Return messages from the stream partition group within the specified timeout The message may be fetched by actively polling the source or by retrieving from a pre-fetched buffer. This depends on the implementation.
        Parameters:
        startOffset - The offset of the first message desired, inclusive
        endOffset - The offset of the last message desired, exclusive, or null
        timeoutMs - Timeout in milliseconds
        Returns:
        An iterable containing messages fetched from the stream partition and their offsets
        Throws:
        TimeoutException - If the operation could not be completed within timeoutMillis milliseconds

      • checkpoint

        default StreamPartitionMsgOffset checkpoint​(StreamPartitionMsgOffset lastOffset)
        Checkpoints the consumption state of the stream partition group in the source This is useful in streaming systems that require preserving consumption state on the source in order to resume or replay consumption of data. The consumer implementation is responsible for managing this state. The offset returned will be used for offset comparisons within the local server (say, for catching up) and also, persisted to the ZK segment metadata. Hence, the returned value should be same or equivalent to the lastOffset provided as input (that is, compareTo of the input and returned offset should be 0).
        Parameters:
        lastOffset - checkpoint the stream at this offset (exclusive)
        Returns:
        Returns the offset that should be used as the next upcoming offset for the stream partition group