org.apache.gobblin.dataset

Interface PartitionableDataset<T extends PartitionableDataset.DatasetPartition>

Type Parameters:

T - the type of partitions returned by the dataset.

All Superinterfaces:

Dataset, URNIdentified

All Known Implementing Classes:

SimplePartitionableDatasetForTesting

public interface PartitionableDataset<T extends PartitionableDataset.DatasetPartition>
extends Dataset

A Dataset that can be partitioned into disjoint subsets of the dataset.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static interface	PartitionableDataset.DatasetPartition A partition of a PartitionableDataset.

Method Summary

Modifier and Type	Method and Description
java.util.stream.Stream<T>	getPartitions(int desiredCharacteristics, Comparator<T> suggestedOrder) Get a stream of partitions.

Methods inherited from interface org.apache.gobblin.dataset.Dataset

datasetURN, getUrn

Method Detail

getPartitions

java.util.stream.Stream<T> getPartitions(int desiredCharacteristics,
                                         Comparator<T> suggestedOrder)
                                  throws IOException

Get a stream of partitions.

Parameters:

desiredCharacteristics - desired Spliterator characteristics of this stream. The returned stream need not satisfy these characteristics, this argument merely implies that the caller will run optimally when those characteristics are present, allowing pushdown of those characteristics. For example Spliterator.SORTED can sometimes be pushed down at a cost, so the Dataset would only push it down if it is valuable for the caller.

suggestedOrder - suggested order of the partitions in the stream. Implementation may or may not return the entries in that order. If the entries are in that order, implementation should ensure the spliterator is annotated as such.

Returns:

a Stream over PartitionableDataset.DatasetPartitions in this dataset.

Throws:

IOException