org.apache.tez.runtime.library.common.sort.impl

Class PipelinedSorter

java.lang.Object

org.apache.tez.runtime.library.common.sort.impl.ExternalSorter

org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter

public class PipelinedSorter
extends ExternalSorter

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.tez.runtime.library.common.sort.impl.ExternalSorter

ExternalSorter.MapBufferTooSmallException

Field Summary

Fields

Modifier and Type	Field and Description
static int	MAP_OUTPUT_INDEX_RECORD_LENGTH The size of each record in the index file for the map-outputs.

Fields inherited from class org.apache.tez.runtime.library.common.sort.impl.ExternalSorter

additionalSpillBytesRead, additionalSpillBytesWritten, availableMemoryMb, cleanup, codec, combiner, comparator, conf, fileOutputByteCounter, finalIndexFile, finalMergeEnabled, finalOutputFile, ifileBufferSize, ifileReadAhead, ifileReadAheadLength, keyClass, keySerializer, mapOutputByteCounter, mapOutputFile, mapOutputRecordCounter, numAdditionalSpills, numShuffleChunks, numSpills, outputBytesWithOverheadCounter, outputContext, partitioner, partitions, partitionStats, progressable, rfs, sendEmptyPartitionDetails, serializationFactory, sorter, spilledRecordsCounter, spillFileIndexPaths, spillFilePaths, statsReporter, valClass, valSerializer

Constructor Summary

Constructors

Constructor and Description
PipelinedSorter(org.apache.tez.runtime.api.OutputContext outputContext, org.apache.hadoop.conf.Configuration conf, int numOutputs, long initialMemoryAvailable)

Method Summary

Modifier and Type	Method and Description
void	flush()
boolean	needsRLE()
void	sort()
boolean	spill(boolean ignoreEmptySpills)
void	write(Object key, Object value)

Methods inherited from class org.apache.tez.runtime.library.common.sort.impl.ExternalSorter

cleanup, cleanup, cleanup, close, getFinalIndexFile, getFinalOutputFile, getInitialMemoryRequirement, getMapOutput, getNumSpills, getPartitionStats, getShuffleHeader, getSortedStream, isFinalMergeEnabled, reportDetailedPartitionStats, reportPartitionStats, reportStatistics, runCombineProcessor, sameVolRename, write

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAP_OUTPUT_INDEX_RECORD_LENGTH

public static final int MAP_OUTPUT_INDEX_RECORD_LENGTH

The size of each record in the index file for the map-outputs.

See Also:

Constant Field Values

Constructor Detail

PipelinedSorter

public PipelinedSorter(org.apache.tez.runtime.api.OutputContext outputContext,
                       org.apache.hadoop.conf.Configuration conf,
                       int numOutputs,
                       long initialMemoryAvailable)
                throws IOException

Throws:

IOException

Method Detail

sort

public void sort()
          throws IOException

Throws:

IOException

write

public void write(Object key,
                  Object value)
           throws IOException

Specified by:

write in class ExternalSorter

Throws:

IOException

spill

public boolean spill(boolean ignoreEmptySpills)
              throws IOException

Throws:

IOException

flush

public void flush()
           throws IOException

Specified by:

flush in class ExternalSorter

Throws:

IOException

needsRLE

@InterfaceAudience.Private
public boolean needsRLE()