org.apache.tez.runtime.library.common.sort.impl

Class IFile.Reader

java.lang.Object

org.apache.tez.runtime.library.common.sort.impl.IFile.Reader

Direct Known Subclasses:

InMemoryReader

Enclosing class:

IFile

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public static class IFile.Reader
extends Object

IFile.Reader to read intermediate map-outputs.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	IFile.Reader.KeyState

Field Summary

Fields

Modifier and Type	Field and Description
protected byte[]	buffer
protected int	bufferSize
long	bytesRead
protected int	currentKeyLength
protected int	currentValueLength
protected DataInputStream	dataIn
protected boolean	eof
protected static int	MAX_BUFFER_SIZE
protected int	originalKeyLength
protected int	prevKeyLength
protected int	recNo

Constructor Summary

Constructors

Constructor and Description
Reader(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path file, org.apache.hadoop.io.compress.CompressionCodec codec, org.apache.tez.common.counters.TezCounter readsCounter, org.apache.tez.common.counters.TezCounter bytesReadCounter, boolean ifileReadAhead, int ifileReadAheadLength, int bufferSize) Construct an IFile Reader.
Reader(InputStream in, long length, org.apache.hadoop.io.compress.CompressionCodec codec, org.apache.tez.common.counters.TezCounter readsCounter, org.apache.tez.common.counters.TezCounter bytesReadCounter, boolean readAhead, int readAheadLength, int bufferSize) Construct an IFile Reader.
Reader(InputStream in, long length, org.apache.hadoop.io.compress.CompressionCodec codec, org.apache.tez.common.counters.TezCounter readsCounter, org.apache.tez.common.counters.TezCounter bytesReadCounter, boolean readAhead, int readAheadLength, int bufferSize, boolean isCompressed) Construct an IFile Reader.

Method Summary

Modifier and Type	Method and Description
void	close()
void	disableChecksumValidation()
long	getLength()
long	getPosition()
static boolean	isCompressedFlagEnabled(InputStream in)
boolean	nextRawKey(org.apache.hadoop.io.DataInputBuffer key)
void	nextRawValue(org.apache.hadoop.io.DataInputBuffer value)
protected boolean	positionToNextRecord(DataInput dIn) Reset key length and value length for next record in the file
protected void	readKeyValueLength(DataInput dIn)
IFile.Reader.KeyState	readRawKey(org.apache.hadoop.io.DataInputBuffer key)
static long	readToDisk(OutputStream out, InputStream in, long length, boolean ifileReadAhead, int ifileReadAheadLength) Read entire IFile content to disk.
static void	readToMemory(byte[] buffer, InputStream in, int compressedLength, org.apache.hadoop.io.compress.CompressionCodec codec, boolean ifileReadAhead, int ifileReadAheadLength) Read entire ifile content to memory.
protected void	readValueLength(DataInput dIn)
void	reset(int offset)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAX_BUFFER_SIZE

protected static int MAX_BUFFER_SIZE

bytesRead

public long bytesRead

eof

protected boolean eof

buffer

protected byte[] buffer

bufferSize

protected int bufferSize

dataIn

protected DataInputStream dataIn

recNo

protected int recNo

originalKeyLength

protected int originalKeyLength

prevKeyLength

protected int prevKeyLength

currentKeyLength

protected int currentKeyLength

currentValueLength

protected int currentValueLength

Constructor Detail

Reader

public Reader(org.apache.hadoop.fs.FileSystem fs,
              org.apache.hadoop.fs.Path file,
              org.apache.hadoop.io.compress.CompressionCodec codec,
              org.apache.tez.common.counters.TezCounter readsCounter,
              org.apache.tez.common.counters.TezCounter bytesReadCounter,
              boolean ifileReadAhead,
              int ifileReadAheadLength,
              int bufferSize)
       throws IOException

Construct an IFile Reader.

Parameters:

fs - FileSystem

file - Path of the file to be opened. This file should have checksum bytes for the data at the end of the file.

codec - codec

readsCounter - Counter for records read from disk

Throws:

IOException

Reader

public Reader(InputStream in,
              long length,
              org.apache.hadoop.io.compress.CompressionCodec codec,
              org.apache.tez.common.counters.TezCounter readsCounter,
              org.apache.tez.common.counters.TezCounter bytesReadCounter,
              boolean readAhead,
              int readAheadLength,
              int bufferSize)
       throws IOException

Construct an IFile Reader.

Parameters:

in - The input stream

length - Length of the data in the stream, including the checksum bytes.

codec - codec

readsCounter - Counter for records read from disk

Throws:

IOException

Reader

public Reader(InputStream in,
              long length,
              org.apache.hadoop.io.compress.CompressionCodec codec,
              org.apache.tez.common.counters.TezCounter readsCounter,
              org.apache.tez.common.counters.TezCounter bytesReadCounter,
              boolean readAhead,
              int readAheadLength,
              int bufferSize,
              boolean isCompressed)
       throws IOException

Construct an IFile Reader.

Parameters:

in - The input stream

length - Length of the data in the stream, including the checksum bytes.

codec - codec

readsCounter - Counter for records read from disk

Throws:

IOException

Method Detail

readToMemory

public static void readToMemory(byte[] buffer,
                                InputStream in,
                                int compressedLength,
                                org.apache.hadoop.io.compress.CompressionCodec codec,
                                boolean ifileReadAhead,
                                int ifileReadAheadLength)
                         throws IOException

Read entire ifile content to memory.

Parameters:

buffer -

in -

compressedLength -

codec -

ifileReadAhead -

ifileReadAheadLength -

Throws:

IOException

readToDisk

public static long readToDisk(OutputStream out,
                              InputStream in,
                              long length,
                              boolean ifileReadAhead,
                              int ifileReadAheadLength)
                       throws IOException

Read entire IFile content to disk.

Parameters:

out - the output stream that will receive the data

in - the input stream containing the IFile data

length - the amount of data to read from the input

Returns:

the number of bytes copied

Throws:

IOException

getLength

public long getLength()

getPosition

public long getPosition()
                 throws IOException

Throws:

IOException

readValueLength

protected void readValueLength(DataInput dIn)
                        throws IOException

Throws:

IOException

readKeyValueLength

protected void readKeyValueLength(DataInput dIn)
                           throws IOException

Throws:

IOException

positionToNextRecord

protected boolean positionToNextRecord(DataInput dIn)
                                throws IOException

Reset key length and value length for next record in the file

Parameters:

dIn -

Returns:

true if key length and value length were set to the next false if end of file (EOF) marker was reached

Throws:

IOException

nextRawKey

public final boolean nextRawKey(org.apache.hadoop.io.DataInputBuffer key)
                         throws IOException

Throws:

IOException

readRawKey

public IFile.Reader.KeyState readRawKey(org.apache.hadoop.io.DataInputBuffer key)
                                 throws IOException

Throws:

IOException

nextRawValue

public void nextRawValue(org.apache.hadoop.io.DataInputBuffer value)
                  throws IOException

Throws:

IOException

isCompressedFlagEnabled

public static boolean isCompressedFlagEnabled(InputStream in)
                                       throws IOException

Throws:

IOException

close

public void close()
           throws IOException

Throws:

IOException

reset

public void reset(int offset)

disableChecksumValidation

public void disableChecksumValidation()