org.apache.gobblin.source.workunit

Interface WorkUnitStream

All Known Implementing Classes:

BasicWorkUnitStream

public interface WorkUnitStream

A stream of WorkUnits, allows for working with large numbers of work units in a memory-efficient way, as well as processing infinite streams of work units.

Method Summary

Modifier and Type	Method and Description
WorkUnitStream	filter(com.google.common.base.Predicate<WorkUnit> predicate) Apply a filtering function to this stream.
Collection<WorkUnit>	getMaterializedWorkUnitCollection() Get a materialized collection of the WorkUnits in this stream.
Iterator<WorkUnit>	getWorkUnits()
boolean	isFiniteStream()
boolean	isSafeToMaterialize()
WorkUnitStream	transform(com.google.common.base.Function<WorkUnit,WorkUnit> function) Apply a transformation function to this stream.

Method Detail

getWorkUnits

Iterator<WorkUnit> getWorkUnits()

Returns:

Iterator of WorkUnits.

isFiniteStream

boolean isFiniteStream()

Returns:

true if this WorkUnitStream is finite.

transform

WorkUnitStream transform(com.google.common.base.Function<WorkUnit,WorkUnit> function)

Apply a transformation function to this stream.

filter

WorkUnitStream filter(com.google.common.base.Predicate<WorkUnit> predicate)

Apply a filtering function to this stream.

isSafeToMaterialize

boolean isSafeToMaterialize()

Returns:

true if it is safe to call getMaterializedWorkUnitCollection() to get a collection of WorkUnits.

getMaterializedWorkUnitCollection

Collection<WorkUnit> getMaterializedWorkUnitCollection()

Get a materialized collection of the WorkUnits in this stream. Note this call will fail if isSafeToMaterialize() is false. This method should be avoided unless strictly necessary, as it introduces a synchronization point for all work units, removing the speed and memory-efficiency benefits.