org.apache.tez.runtime.library.common.task.local.output

Class TezTaskOutputFiles

java.lang.Object

org.apache.tez.runtime.library.common.task.local.output.TezTaskOutput

org.apache.tez.runtime.library.common.task.local.output.TezTaskOutputFiles

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public class TezTaskOutputFiles
extends TezTaskOutput

Manipulate the working area for the transient store for components in tez-runtime-library This class is used by Inputs and Outputs in tez-runtime-library to identify the directories that they need to write to / read from for intermediate files.

Field Summary

Fields inherited from class org.apache.tez.runtime.library.common.task.local.output.TezTaskOutput

conf, dagId, uniqueId

Constructor Summary

Constructors

Constructor and Description
TezTaskOutputFiles(org.apache.hadoop.conf.Configuration conf, String uniqueId, int dagID)

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.fs.Path	getInputFileForWrite(int srcIdentifier, int spillNum, long size) Create a local input file name.
org.apache.hadoop.fs.Path	getOutputFileForWrite() Create a local output file name.
org.apache.hadoop.fs.Path	getOutputFileForWrite(long size) Create a local output file name.
org.apache.hadoop.fs.Path	getOutputFileForWriteInVolume(org.apache.hadoop.fs.Path existing) Create a local output file name on the same volume.
org.apache.hadoop.fs.Path	getOutputIndexFileForWrite(long size) Create a local output index file name.
org.apache.hadoop.fs.Path	getOutputIndexFileForWriteInVolume(org.apache.hadoop.fs.Path existing) Create a local output index file name on the same volume.
org.apache.hadoop.fs.Path	getSpillFileForWrite(int spillNumber, long size) Create a local spill file name.
String	getSpillFileName(int srcId, int spillNum) Construct a spill file name, given a spill number and src id ${uniqueId}_src_${srcId}_spill_${spillNumber}.out e.g.
org.apache.hadoop.fs.Path	getSpillIndexFileForWrite(int spillNumber, long size) Create a local output spill index file name.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TezTaskOutputFiles

public TezTaskOutputFiles(org.apache.hadoop.conf.Configuration conf,
                          String uniqueId,
                          int dagID)

Method Detail

getOutputFileForWrite

public org.apache.hadoop.fs.Path getOutputFileForWrite(long size)
                                                throws IOException

Create a local output file name. ${appDir}/output/${uniqueId}/file.out e.g. application_1418684642047_0006/output/attempt_1418684642047_0006_1_00_000000_0_10003/file.out The structure of this file name is critical, to be served by the MapReduce ShuffleHandler.

Specified by:

getOutputFileForWrite in class TezTaskOutput

Parameters:

size - the size of the file

Returns:

path the path to write to

Throws:

IOException

getOutputFileForWrite

public org.apache.hadoop.fs.Path getOutputFileForWrite()
                                                throws IOException

Create a local output file name. This should *only* be used if the size of the file is not known. Otherwise use the equivalent which accepts a size parameter. ${appDir}/output/${uniqueId}/file.out e.g. application_1418684642047_0006/output/attempt_1418684642047_0006_1_00_000000_0_10003/file.out The structure of this file name is critical, to be served by the MapReduce ShuffleHandler.

Specified by:

getOutputFileForWrite in class TezTaskOutput

Returns:

path the path to write to

Throws:

IOException

getOutputFileForWriteInVolume

public org.apache.hadoop.fs.Path getOutputFileForWriteInVolume(org.apache.hadoop.fs.Path existing)

Create a local output file name on the same volume. This is only meant to be used to rename temporary files to their final destination within the same volume. ${appDir}/output/${uniqueId}/file.out e.g. existing: application_1424502260528_0119/output/attempt_1424502260528_0119_1_07_000058_0_10012_0/file.out returnValue: application_1424502260528_0119/output/attempt_1424502260528_0119_1_07_000058_0_10012/file.out The structure of this file name is critical, to be served by the MapReduce ShuffleHandler.

Specified by:

getOutputFileForWriteInVolume in class TezTaskOutput

Returns:

path the path of the output file within the same volume

getOutputIndexFileForWrite

public org.apache.hadoop.fs.Path getOutputIndexFileForWrite(long size)
                                                     throws IOException

Create a local output index file name. ${appDir}/output/${uniqueId}/file.out.index e.g. application_1418684642047_0006/output/attempt_1418684642047_0006_1_00_000000_0_10003/file.out.index The structure of this file name is critical, to be served by the MapReduce ShuffleHandler.

Specified by:

getOutputIndexFileForWrite in class TezTaskOutput

Parameters:

size - the size of the file

Returns:

path the path to write the index file to

Throws:

IOException

getOutputIndexFileForWriteInVolume

public org.apache.hadoop.fs.Path getOutputIndexFileForWriteInVolume(org.apache.hadoop.fs.Path existing)

Create a local output index file name on the same volume. The intended usage of this method is to write the index file on the same volume as the associated data file. ${appDir}/output/${uniqueId}/file.out.index e.g. existing: application_1424502260528_0119/output/attempt_1424502260528_0119_1_07_000058_0_10012_0/file .out.index returnValue: application_1424502260528_0119/output/attempt_1424502260528_0119_1_07_000058_0_10012/file .out.index The structure of this file name is critical, to be served by the MapReduce ShuffleHandler.

Specified by:

getOutputIndexFileForWriteInVolume in class TezTaskOutput

Returns:

path the path of the index file within the same volume

getSpillFileForWrite

public org.apache.hadoop.fs.Path getSpillFileForWrite(int spillNumber,
                                                      long size)
                                               throws IOException

Create a local spill file name. ${appDir}/output/${uniqueId}_${spillNumber}/file.out e.g. application_1422270854961_0027/output/attempt_1422270854961_0027_1_00_000001_0_10003_1/file.out

Specified by:

getSpillFileForWrite in class TezTaskOutput

Parameters:

spillNumber - the spill number

size - the size of the spill file

Returns:

path the path to write the spill file for the specific spillNumber

Throws:

IOException

getSpillIndexFileForWrite

public org.apache.hadoop.fs.Path getSpillIndexFileForWrite(int spillNumber,
                                                           long size)
                                                    throws IOException

Create a local output spill index file name. ${appDir}/output/${uniqueId}_${spillNumber}/file.out.index e.g. application_1422270854961_0027/output/attempt_1422270854961_0027_1_00_000001_0_10003_1/file.out.index

Specified by:

getSpillIndexFileForWrite in class TezTaskOutput

Parameters:

spillNumber - the spill number

size - the size of the file

Returns:

path the path to write the spill index file for the specific spillNumber

Throws:

IOException

getInputFileForWrite

public org.apache.hadoop.fs.Path getInputFileForWrite(int srcIdentifier,
                                                      int spillNum,
                                                      long size)
                                               throws IOException

Create a local input file name. ${appDir}/${uniqueId}_src_{$srcId}_spill_${spillNumber}.out e.g. application_1418684642047_0006/attempt_1418684642047_0006_1_00_000000_0_10004_src_10_spill_0.out Files are not clobbered due to the uniqueId along with spillId being different for Outputs / Inputs within the same task (and across tasks)

Specified by:

getInputFileForWrite in class TezTaskOutput

Parameters:

srcIdentifier - The identifier for the source

spillNum -

size - the size of the file @return path the path to the input file.

Throws:

IOException

getSpillFileName

public String getSpillFileName(int srcId,
                               int spillNum)

Construct a spill file name, given a spill number and src id ${uniqueId}_src_${srcId}_spill_${spillNumber}.out e.g. attempt_1418684642047_0006_1_00_000000_0_10004_src_10_spill_0.out

Specified by:

getSpillFileName in class TezTaskOutput

Parameters:

srcId -

spillNum -

Returns:

a spill file name independent of the unique identifier and local directories