org.apache.hudi.common.util.queue

Class BoundedInMemoryQueue<I,O>

java.lang.Object

org.apache.hudi.common.util.queue.BoundedInMemoryQueue<I,O>

Type Parameters:

I - input payload data type

O - output payload data type

All Implemented Interfaces:

Iterable<O>

public class BoundedInMemoryQueue<I,O>
extends Object
implements Iterable<O>

Used for enqueueing input records. Queue limit is controlled by memoryLimit. Unlike standard bounded queue implementations, this queue bounds the size by memory bytes occupied by its tenants. The standard implementation bounds by the number of entries in the queue. It internally samples every RECORD_SAMPLING_RATEth record and adjusts number of records in queue accordingly. This is done to ensure that we don't OOM. This queue supports multiple producer single consumer pattern.

Field Summary

Fields

Modifier and Type	Field and Description
long	avgRecordSizeInBytes Indicates avg record size in bytes.
int	currentRateLimit indicates rate limit (number of records to cache).
Semaphore	rateLimiter It indicates number of records to cache.
static int	RECORD_POLL_INTERVAL_SEC Interval used for polling records in the queue.
static int	RECORD_SAMPLING_RATE Rate used for sampling records to determine avg record size in bytes.
AtomicLong	samplingRecordCounter Used for sampling records with "RECORD_SAMPLING_RATE" frequency.

Constructor Summary

Constructors

Constructor and Description
BoundedInMemoryQueue(long memoryLimit, Function<I,O> transformFunction) Construct BoundedInMemoryQueue with default SizeEstimator.
BoundedInMemoryQueue(long memoryLimit, Function<I,O> transformFunction, SizeEstimator<O> payloadSizeEstimator) Construct BoundedInMemoryQueue with passed in size estimator.

Method Summary

Modifier and Type	Method and Description
void	close() Puts an empty entry to queue to denote termination.
void	insertRecord(I t) Inserts record into queue after applying transformation.
Iterator<O>	iterator()
void	markAsFailed(Throwable e) API to allow producers and consumer to communicate termination due to failure.
int	size()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Field Detail

RECORD_POLL_INTERVAL_SEC

public static final int RECORD_POLL_INTERVAL_SEC

Interval used for polling records in the queue.

See Also:

Constant Field Values

RECORD_SAMPLING_RATE

public static final int RECORD_SAMPLING_RATE

Rate used for sampling records to determine avg record size in bytes.

See Also:

Constant Field Values

rateLimiter

public final Semaphore rateLimiter

It indicates number of records to cache. We will be using sampled record's average size to determine how many records we should cache and will change (increase/decrease) permits accordingly.

samplingRecordCounter

public final AtomicLong samplingRecordCounter

Used for sampling records with "RECORD_SAMPLING_RATE" frequency.

currentRateLimit

public int currentRateLimit

indicates rate limit (number of records to cache). it is updated whenever there is a change in avg record size.

avgRecordSizeInBytes

public long avgRecordSizeInBytes

Indicates avg record size in bytes. It is updated whenever a new record is sampled.

Constructor Detail

BoundedInMemoryQueue

public BoundedInMemoryQueue(long memoryLimit,
                            Function<I,O> transformFunction)

Construct BoundedInMemoryQueue with default SizeEstimator.

Parameters:

memoryLimit - MemoryLimit in bytes

transformFunction - Transformer Function to convert input payload type to stored payload type

BoundedInMemoryQueue

public BoundedInMemoryQueue(long memoryLimit,
                            Function<I,O> transformFunction,
                            SizeEstimator<O> payloadSizeEstimator)

Construct BoundedInMemoryQueue with passed in size estimator.

Parameters:

memoryLimit - MemoryLimit in bytes

transformFunction - Transformer Function to convert input payload type to stored payload type

payloadSizeEstimator - Payload Size Estimator

Method Detail

size

public int size()

insertRecord

public void insertRecord(I t)
                  throws Exception

Inserts record into queue after applying transformation.

Parameters:

t - Item to be queued

Throws:

Exception

close

public void close()

Puts an empty entry to queue to denote termination.

markAsFailed

public void markAsFailed(Throwable e)

API to allow producers and consumer to communicate termination due to failure.

iterator

public Iterator<O> iterator()

Specified by:

iterator in interface Iterable<O>