org.apache.hadoop.hive.ql.io

Class SkippingTextInputFormat

java.lang.Object

org.apache.hadoop.mapred.FileInputFormat<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text>

org.apache.hadoop.mapred.TextInputFormat

org.apache.hadoop.hive.ql.io.SkippingTextInputFormat

All Implemented Interfaces:

org.apache.hadoop.mapred.InputFormat<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text>, org.apache.hadoop.mapred.JobConfigurable

public class SkippingTextInputFormat
extends org.apache.hadoop.mapred.TextInputFormat

SkippingInputFormat is a header/footer aware input format. It truncates splits identified by TextInputFormat. Header and footers are removed from the splits. This InputFormat does NOT support Compressed Files!

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.mapred.FileInputFormat

org.apache.hadoop.mapred.FileInputFormat.Counter

Field Summary

Fields inherited from class org.apache.hadoop.mapred.FileInputFormat

INPUT_DIR_NONRECURSIVE_IGNORE_SUBDIRS, INPUT_DIR_RECURSIVE, LOG, NUM_INPUT_FILES

Constructor Summary

Constructors

Constructor and Description
SkippingTextInputFormat()

Method Summary

Modifier and Type	Method and Description
void	configure(org.apache.hadoop.mapred.JobConf conf)
void	configure(org.apache.hadoop.mapred.JobConf conf, int headerCount, int footerCount)
protected org.apache.hadoop.mapred.FileSplit	makeSplit(org.apache.hadoop.fs.Path file, long start, long length, String[] hosts)
protected org.apache.hadoop.mapred.FileSplit	makeSplit(org.apache.hadoop.fs.Path file, long start, long length, String[] hosts, String[] inMemoryHosts)

Methods inherited from class org.apache.hadoop.mapred.TextInputFormat

getRecordReader, isSplitable

Methods inherited from class org.apache.hadoop.mapred.FileInputFormat

addInputPath, addInputPathRecursively, addInputPaths, computeSplitSize, getBlockIndex, getInputPathFilter, getInputPaths, getSplitHosts, getSplits, listStatus, setInputPathFilter, setInputPaths, setInputPaths, setMinSplitSize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SkippingTextInputFormat

public SkippingTextInputFormat()

Method Detail

configure

public void configure(org.apache.hadoop.mapred.JobConf conf)

Specified by:

configure in interface org.apache.hadoop.mapred.JobConfigurable

Overrides:

configure in class org.apache.hadoop.mapred.TextInputFormat

configure

public void configure(org.apache.hadoop.mapred.JobConf conf,
                      int headerCount,
                      int footerCount)

makeSplit

protected org.apache.hadoop.mapred.FileSplit makeSplit(org.apache.hadoop.fs.Path file,
                                                       long start,
                                                       long length,
                                                       String[] hosts)

Overrides:

makeSplit in class org.apache.hadoop.mapred.FileInputFormat<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text>

makeSplit

protected org.apache.hadoop.mapred.FileSplit makeSplit(org.apache.hadoop.fs.Path file,
                                                       long start,
                                                       long length,
                                                       String[] hosts,
                                                       String[] inMemoryHosts)

Overrides:

makeSplit in class org.apache.hadoop.mapred.FileInputFormat<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text>