org.apache.hadoop.streaming.mapreduce

Class StreamBaseRecordReader

java.lang.Object

org.apache.hadoop.mapreduce.RecordReader<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text>

org.apache.hadoop.streaming.mapreduce.StreamBaseRecordReader

All Implemented Interfaces:

Closeable, AutoCloseable

Direct Known Subclasses:

StreamXmlRecordReader

public abstract class StreamBaseRecordReader
extends org.apache.hadoop.mapreduce.RecordReader<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text>

Shared functionality for hadoopStreaming formats. A custom reader can be defined to be a RecordReader with the constructor below and is selected with the option bin/hadoopStreaming -inputreader ...

See Also:

StreamXmlRecordReader

Field Summary

Fields

Modifier and Type	Field and Description
protected static org.slf4j.Logger	LOG

Constructor Summary

Constructors

Constructor and Description
StreamBaseRecordReader(org.apache.hadoop.fs.FSDataInputStream in, org.apache.hadoop.mapreduce.lib.input.FileSplit split, org.apache.hadoop.mapreduce.TaskAttemptContext context, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs)

Method Summary

Modifier and Type	Method and Description
void	close() Close this to future operations.
org.apache.hadoop.io.Text	createKey()
org.apache.hadoop.io.Text	createValue()
long	getPos() Returns the current position in the input.
float	getProgress()
abstract boolean	next(org.apache.hadoop.io.Text key, org.apache.hadoop.io.Text value) Read a record.
abstract void	seekNextRecordBoundary() Implementation should seek forward in_ to the first byte of the next record.

Methods inherited from class org.apache.hadoop.mapreduce.RecordReader

getCurrentKey, getCurrentValue, initialize, nextKeyValue

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

protected static final org.slf4j.Logger LOG

Constructor Detail

StreamBaseRecordReader

public StreamBaseRecordReader(org.apache.hadoop.fs.FSDataInputStream in,
                              org.apache.hadoop.mapreduce.lib.input.FileSplit split,
                              org.apache.hadoop.mapreduce.TaskAttemptContext context,
                              org.apache.hadoop.conf.Configuration conf,
                              org.apache.hadoop.fs.FileSystem fs)
                       throws IOException

Throws:

IOException

Method Detail

next

public abstract boolean next(org.apache.hadoop.io.Text key,
                             org.apache.hadoop.io.Text value)
                      throws IOException

Read a record. Implementation should call numRecStats at the end

Throws:

IOException

getPos

public long getPos()
            throws IOException

Returns the current position in the input.

Throws:

IOException

close

public void close()
           throws IOException

Close this to future operations.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Specified by:

close in class org.apache.hadoop.mapreduce.RecordReader<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text>

Throws:

IOException

getProgress

public float getProgress()
                  throws IOException

Specified by:

getProgress in class org.apache.hadoop.mapreduce.RecordReader<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text>

Throws:

IOException

createKey

public org.apache.hadoop.io.Text createKey()

createValue

public org.apache.hadoop.io.Text createValue()

seekNextRecordBoundary

public abstract void seekNextRecordBoundary()
                                     throws IOException

Implementation should seek forward in_ to the first byte of the next record. The initial byte offset in the stream is arbitrary.

Throws:

IOException