edu.umd.cloud9.collection.line

Class TextDocumentInputFormat

java.lang.Object

org.apache.hadoop.mapred.FileInputFormat<org.apache.hadoop.io.LongWritable,TextDocument>

edu.umd.cloud9.collection.line.TextDocumentInputFormat

All Implemented Interfaces:

org.apache.hadoop.mapred.InputFormat<org.apache.hadoop.io.LongWritable,TextDocument>, org.apache.hadoop.mapred.JobConfigurable

public class TextDocumentInputFormat
extends org.apache.hadoop.mapred.FileInputFormat<org.apache.hadoop.io.LongWritable,TextDocument>
implements org.apache.hadoop.mapred.JobConfigurable

Hadoop InputFormat for processing a simple collection. Each document of the collection consists of a single line of text: the docid, followed by a tab, followed by the document contents. Note that the document content cannot contain embedded tabs or newlines.

Author:

Jimmy Lin

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	TextDocumentInputFormat.TextDocumentLineRecordReader

Field Summary

Fields inherited from class org.apache.hadoop.mapred.FileInputFormat

LOG

Constructor Summary

Constructors

Constructor and Description
TextDocumentInputFormat()

Method Summary

Methods

Modifier and Type	Method and Description
void	configure(org.apache.hadoop.mapred.JobConf conf)
org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.LongWritable,TextDocument>	getRecordReader(org.apache.hadoop.mapred.InputSplit genericSplit, org.apache.hadoop.mapred.JobConf job, org.apache.hadoop.mapred.Reporter reporter)
protected boolean	isSplitable(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path file)

Methods inherited from class org.apache.hadoop.mapred.FileInputFormat

addInputPath, addInputPaths, computeSplitSize, getBlockIndex, getInputPathFilter, getInputPaths, getSplitHosts, getSplits, listStatus, setInputPathFilter, setInputPaths, setInputPaths, setMinSplitSize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TextDocumentInputFormat

public TextDocumentInputFormat()

Method Detail

configure

public void configure(org.apache.hadoop.mapred.JobConf conf)

Specified by:

configure in interface org.apache.hadoop.mapred.JobConfigurable

isSplitable

protected boolean isSplitable(org.apache.hadoop.fs.FileSystem fs,
                  org.apache.hadoop.fs.Path file)

Overrides:

isSplitable in class org.apache.hadoop.mapred.FileInputFormat<org.apache.hadoop.io.LongWritable,TextDocument>

getRecordReader

public org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.LongWritable,TextDocument> getRecordReader(org.apache.hadoop.mapred.InputSplit genericSplit,
                                                                                                    org.apache.hadoop.mapred.JobConf job,
                                                                                                    org.apache.hadoop.mapred.Reporter reporter)
                                                                                                      throws IOException

Specified by:

getRecordReader in interface org.apache.hadoop.mapred.InputFormat<org.apache.hadoop.io.LongWritable,TextDocument>

Specified by:

getRecordReader in class org.apache.hadoop.mapred.FileInputFormat<org.apache.hadoop.io.LongWritable,TextDocument>

Throws:

IOException