org.apache.hadoop.hive.ql.io

Class TeradataBinaryFileInputFormat

java.lang.Object

org.apache.hadoop.mapred.FileInputFormat<org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.BytesWritable>

org.apache.hadoop.hive.ql.io.TeradataBinaryFileInputFormat

All Implemented Interfaces:

org.apache.hadoop.mapred.InputFormat<org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.BytesWritable>

public class TeradataBinaryFileInputFormat
extends org.apache.hadoop.mapred.FileInputFormat<org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.BytesWritable>

https://cwiki.apache.org/confluence/display/Hive/TeradataBinarySerde. FileInputFormat for Teradata binary files. In the Teradata Binary File, each record constructs as below: The first 2 bytes represents the length of the bytes next for this record. Then the null bitmap whose length is depended on the number of fields is followed. Then each field of the record is serialized into bytes - the serialization strategy is decided by the type of field. At last, there is one byte (0x0a) in the end of the record. This InputFormat currently doesn't support the split of the file. Teradata binary files are using little endian.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.mapred.FileInputFormat

org.apache.hadoop.mapred.FileInputFormat.Counter

Field Summary

Fields inherited from class org.apache.hadoop.mapred.FileInputFormat

INPUT_DIR_NONRECURSIVE_IGNORE_SUBDIRS, INPUT_DIR_RECURSIVE, LOG, NUM_INPUT_FILES

Constructor Summary

Constructors

Constructor and Description
TeradataBinaryFileInputFormat()

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.BytesWritable>	getRecordReader(org.apache.hadoop.mapred.InputSplit split, org.apache.hadoop.mapred.JobConf job, org.apache.hadoop.mapred.Reporter reporter)
protected boolean	isSplitable(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path filename) the TeradataBinaryFileInputFormat is not splittable right now.

Methods inherited from class org.apache.hadoop.mapred.FileInputFormat

addInputPath, addInputPathRecursively, addInputPaths, computeSplitSize, getBlockIndex, getInputPathFilter, getInputPaths, getSplitHosts, getSplits, listStatus, makeSplit, makeSplit, setInputPathFilter, setInputPaths, setInputPaths, setMinSplitSize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TeradataBinaryFileInputFormat

public TeradataBinaryFileInputFormat()

Method Detail

getRecordReader

public org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.BytesWritable> getRecordReader(org.apache.hadoop.mapred.InputSplit split,
                                                                                                                                   org.apache.hadoop.mapred.JobConf job,
                                                                                                                                   org.apache.hadoop.mapred.Reporter reporter)
                                                                                                                            throws IOException

Specified by:

getRecordReader in interface org.apache.hadoop.mapred.InputFormat<org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.BytesWritable>

Specified by:

getRecordReader in class org.apache.hadoop.mapred.FileInputFormat<org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.BytesWritable>

Throws:

IOException

isSplitable

protected boolean isSplitable(org.apache.hadoop.fs.FileSystem fs,
                              org.apache.hadoop.fs.Path filename)

the TeradataBinaryFileInputFormat is not splittable right now. Override the isSplitable function.

Overrides:

isSplitable in class org.apache.hadoop.mapred.FileInputFormat<org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.BytesWritable>

Parameters:

fs - the file system that the file is on

filename - the file name to check

Returns:

is this file splitable?