org.apache.tez.runtime.library.common.sort.impl

Class IFile.Writer

java.lang.Object

org.apache.tez.runtime.library.common.sort.impl.IFile.Writer

Direct Known Subclasses:

InMemoryWriter

Enclosing class:

IFile

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public static class IFile.Writer
extends Object

IFile.Writer to write out intermediate map-outputs.

Field Summary

Fields

Modifier and Type	Field and Description
protected DataOutputStream	out
protected boolean	rle

Constructor Summary

Constructors

Modifier	Constructor and Description
	Writer(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path file)
	Writer(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path file, Class keyClass, Class valueClass, org.apache.hadoop.io.compress.CompressionCodec codec, org.apache.tez.common.counters.TezCounter writesCounter, org.apache.tez.common.counters.TezCounter serializedBytesCounter)
	Writer(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FSDataOutputStream outputStream, Class keyClass, Class valueClass, org.apache.hadoop.io.compress.CompressionCodec codec, org.apache.tez.common.counters.TezCounter writesCounter, org.apache.tez.common.counters.TezCounter serializedBytesCounter)
	Writer(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FSDataOutputStream outputStream, Class keyClass, Class valueClass, org.apache.hadoop.io.compress.CompressionCodec codec, org.apache.tez.common.counters.TezCounter writesCounter, org.apache.tez.common.counters.TezCounter serializedBytesCounter, boolean rle)
protected	Writer(org.apache.tez.common.counters.TezCounter writesCounter, org.apache.tez.common.counters.TezCounter serializedBytesCounter, boolean rle)

Method Summary

Modifier and Type	Method and Description
void	append(org.apache.hadoop.io.DataInputBuffer key, org.apache.hadoop.io.DataInputBuffer value) Send key/value to be appended to IFile.
void	append(Object key, Object value) Send key/value to be appended to IFile.
<K,V> void	appendKeyValues(K key, Iterator<V> valuesItr) Append key and its associated set of values.
void	appendValue(org.apache.hadoop.io.DataInputBuffer value) Appends the value to previous key.
void	appendValue(Object value) Appends the value to previous key.
<V> void	appendValues(Iterator<V> valuesItr) Appends the value to previous key.
void	close()
long	getCompressedLength()
DataOutputStream	getOutputStream()
long	getRawLength()
void	updateCountersForExternalAppend(long length)
protected void	writeHeader(OutputStream outputStream)
protected void	writeKVPair(byte[] keyData, int keyPos, int keyLength, byte[] valueData, int valPos, int valueLength)
protected void	writeRLE(DataOutputStream out)
protected void	writeValue(byte[] data, int offset, int length)
protected void	writeValueMarker(DataOutputStream out)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

out

protected DataOutputStream out

rle

protected final boolean rle

Constructor Detail

Writer

public Writer(org.apache.hadoop.conf.Configuration conf,
              org.apache.hadoop.fs.FileSystem fs,
              org.apache.hadoop.fs.Path file,
              Class keyClass,
              Class valueClass,
              org.apache.hadoop.io.compress.CompressionCodec codec,
              org.apache.tez.common.counters.TezCounter writesCounter,
              org.apache.tez.common.counters.TezCounter serializedBytesCounter)
       throws IOException

Throws:

IOException

Writer

protected Writer(org.apache.tez.common.counters.TezCounter writesCounter,
                 org.apache.tez.common.counters.TezCounter serializedBytesCounter,
                 boolean rle)

Writer

public Writer(org.apache.hadoop.conf.Configuration conf,
              org.apache.hadoop.fs.FSDataOutputStream outputStream,
              Class keyClass,
              Class valueClass,
              org.apache.hadoop.io.compress.CompressionCodec codec,
              org.apache.tez.common.counters.TezCounter writesCounter,
              org.apache.tez.common.counters.TezCounter serializedBytesCounter)
       throws IOException

Throws:

IOException

Writer

public Writer(org.apache.hadoop.conf.Configuration conf,
              org.apache.hadoop.fs.FSDataOutputStream outputStream,
              Class keyClass,
              Class valueClass,
              org.apache.hadoop.io.compress.CompressionCodec codec,
              org.apache.tez.common.counters.TezCounter writesCounter,
              org.apache.tez.common.counters.TezCounter serializedBytesCounter,
              boolean rle)
       throws IOException

Throws:

IOException

Writer

public Writer(org.apache.hadoop.conf.Configuration conf,
              org.apache.hadoop.fs.FileSystem fs,
              org.apache.hadoop.fs.Path file)
       throws IOException

Throws:

IOException

Method Detail

writeHeader

protected void writeHeader(OutputStream outputStream)
                    throws IOException

Throws:

IOException

close

public void close()
           throws IOException

Throws:

IOException

append

public void append(Object key,
                   Object value)
            throws IOException

Send key/value to be appended to IFile. To represent same key as previous one, send IFile.REPEAT_KEY as key parameter. Should not call this method with IFile.REPEAT_KEY as the first key. It is caller's responsibility to ensure that correct key/value type checks and key/value length (non-negative) checks are done properly.

Parameters:

key -

value -

Throws:

IOException

appendValue

public void appendValue(Object value)
                 throws IOException

Appends the value to previous key. Assumes that the caller has already done relevant checks for identical keys. Also, no validations are done in this method

Parameters:

value -

Throws:

IOException

appendValue

public void appendValue(org.apache.hadoop.io.DataInputBuffer value)
                 throws IOException

Appends the value to previous key. Assumes that the caller has already done relevant checks for identical keys. Also, no validations are done in this method. It is caller's responsibility to pass non-negative key/value lengths. Otherwise,IndexOutOfBoundsException could be thrown at runtime.

Parameters:

value -

Throws:

IOException

appendValues

public <V> void appendValues(Iterator<V> valuesItr)
                      throws IOException

Appends the value to previous key. Assumes that the caller has already done relevant checks for identical keys. Also, no validations are done in this method

Parameters:

valuesItr -

Throws:

IOException

appendKeyValues

public <K,V> void appendKeyValues(K key,
                                  Iterator<V> valuesItr)
                           throws IOException

Append key and its associated set of values.

Type Parameters:

K -

V -

Parameters:

key -

valuesItr -

Throws:

IOException

append

public void append(org.apache.hadoop.io.DataInputBuffer key,
                   org.apache.hadoop.io.DataInputBuffer value)
            throws IOException

Send key/value to be appended to IFile. To represent same key as previous one, send IFile.REPEAT_KEY as key parameter. Should not call this method with IFile.REPEAT_KEY as the first key. It is caller's responsibility to pass non-negative key/value lengths. Otherwise,IndexOutOfBoundsException could be thrown at runtime.

Parameters:

key -

value -

Throws:

IOException

writeValue

protected void writeValue(byte[] data,
                          int offset,
                          int length)
                   throws IOException

Throws:

IOException

writeKVPair

protected void writeKVPair(byte[] keyData,
                           int keyPos,
                           int keyLength,
                           byte[] valueData,
                           int valPos,
                           int valueLength)
                    throws IOException

Throws:

IOException

writeRLE

protected void writeRLE(DataOutputStream out)
                 throws IOException

Throws:

IOException

writeValueMarker

protected void writeValueMarker(DataOutputStream out)
                         throws IOException

Throws:

IOException

getOutputStream

public DataOutputStream getOutputStream()

updateCountersForExternalAppend

public void updateCountersForExternalAppend(long length)

getRawLength

public long getRawLength()

getCompressedLength

public long getCompressedLength()