Package org.apache.pinot.spi.stream

Interface StreamMetadataProvider

All Superinterfaces:

AutoCloseable, Closeable

@Public
@Stable
public interface StreamMetadataProvider
extends Closeable

Interface for provider of stream metadata such as partition count, partition offsets

Nested Class Summary

Nested Classes

Modifier and Type	Interface	Description
static class	StreamMetadataProvider.UnknownLagState

Method Summary

Modifier and Type	Method	Description
default List<PartitionGroupMetadata>	computePartitionGroupMetadata(String clientId, StreamConfig streamConfig, List<PartitionGroupConsumptionStatus> partitionGroupConsumptionStatuses, int timeoutMillis)	Computes the list of PartitionGroupMetadata for the latest state of the stream, using the current PartitionGroupConsumptionStatus Default behavior is the one for the Kafka stream, where each partition group contains only one partition
int	fetchPartitionCount(long timeoutMillis)	Fetches the number of partitions for a topic given the stream configs
StreamPartitionMsgOffset	fetchStreamPartitionOffset(OffsetCriteria offsetCriteria, long timeoutMillis)	Fetches the offset for a given partition and offset criteria
default Map<String,PartitionLagState>	getCurrentPartitionLagState(Map<String,ConsumerPartitionState> currentPartitionStateMap)

Methods inherited from interface java.io.Closeable

close

Method Detail

fetchPartitionCount

int fetchPartitionCount(long timeoutMillis)

Fetches the number of partitions for a topic given the stream configs

Parameters:

timeoutMillis - Fetch timeout

Returns:

number of partitions

fetchStreamPartitionOffset

StreamPartitionMsgOffset fetchStreamPartitionOffset(OffsetCriteria offsetCriteria,
                                                    long timeoutMillis)
                                             throws TimeoutException

Fetches the offset for a given partition and offset criteria

Parameters:

offsetCriteria - offset criteria to fetchStreamPartitionMsgOffset. Depends on the semantics of the stream e.g. smallest, largest for Kafka

timeoutMillis - fetch timeout

Returns:

StreamPartitionMsgOffset based on the offset criteria provided

Throws:

TimeoutException - if timed out trying to connect and fetch from stream

computePartitionGroupMetadata

default List<PartitionGroupMetadata> computePartitionGroupMetadata(String clientId,
                                                                   StreamConfig streamConfig,
                                                                   List<PartitionGroupConsumptionStatus> partitionGroupConsumptionStatuses,
                                                                   int timeoutMillis)
                                                            throws IOException,
                                                                   TimeoutException

Computes the list of PartitionGroupMetadata for the latest state of the stream, using the current PartitionGroupConsumptionStatus Default behavior is the one for the Kafka stream, where each partition group contains only one partition

Parameters:

partitionGroupConsumptionStatuses - list of PartitionGroupConsumptionStatus for current partition groups

Throws:

IOException

TimeoutException

getCurrentPartitionLagState

default Map<String,PartitionLagState> getCurrentPartitionLagState(Map<String,ConsumerPartitionState> currentPartitionStateMap)