org.apache.hudi.hadoop.hive

Class HoodieCombineHiveInputFormat.HoodieCombineFileInputFormatShim<K,V>

java.lang.Object

org.apache.hadoop.mapreduce.InputFormat<K,V>

org.apache.hadoop.mapreduce.lib.input.FileInputFormat<K,V>

org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat<K,V>

org.apache.hadoop.mapred.lib.CombineFileInputFormat<K,V>

org.apache.hudi.hadoop.hive.HoodieCombineHiveInputFormat.HoodieCombineFileInputFormatShim<K,V>

All Implemented Interfaces:

org.apache.hadoop.hive.shims.HadoopShims.CombineFileInputFormatShim<K,V>, org.apache.hadoop.mapred.InputFormat<K,V>

Enclosing class:

HoodieCombineHiveInputFormat<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>

public static class HoodieCombineHiveInputFormat.HoodieCombineFileInputFormatShim<K,V>
extends org.apache.hadoop.mapred.lib.CombineFileInputFormat<K,V>
implements org.apache.hadoop.hive.shims.HadoopShims.CombineFileInputFormatShim<K,V>

**MOD** This is the implementation of CombineFileInputFormat which is a copy of org.apache.hadoop.hive.shims.HadoopShimsSecure.CombineFileInputFormatShim with changes in listStatus.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat

org.apache.hadoop.mapreduce.lib.input.FileInputFormat.Counter

Field Summary

Fields inherited from class org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat

SPLIT_MINSIZE_PERNODE, SPLIT_MINSIZE_PERRACK

Fields inherited from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat

DEFAULT_LIST_STATUS_NUM_THREADS, INPUT_DIR, INPUT_DIR_RECURSIVE, LIST_STATUS_NUM_THREADS, NUM_INPUT_FILES, PATHFILTER_CLASS, SPLIT_MAXSIZE, SPLIT_MINSIZE

Constructor Summary

Constructors

Constructor and Description
HoodieCombineFileInputFormatShim()

Method Summary

Modifier and Type	Method and Description
protected HoodieParquetInputFormat	createParquetInputFormat()
protected HoodieParquetRealtimeInputFormat	createParquetRealtimeInputFormat()
void	createPool(org.apache.hadoop.mapred.JobConf conf, org.apache.hadoop.fs.PathFilter... filters)
org.apache.hadoop.fs.Path[]	getInputPathsShim(org.apache.hadoop.mapred.JobConf conf)
org.apache.hadoop.hive.shims.HadoopShimsSecure.InputSplitShim	getInputSplitShim()
org.apache.hadoop.mapred.RecordReader<K,V>	getRecordReader(org.apache.hadoop.mapred.InputSplit split, org.apache.hadoop.mapred.JobConf job, org.apache.hadoop.mapred.Reporter reporter)
org.apache.hadoop.mapred.RecordReader	getRecordReader(org.apache.hadoop.mapred.JobConf job, org.apache.hadoop.mapred.lib.CombineFileSplit split, org.apache.hadoop.mapred.Reporter reporter, Class<org.apache.hadoop.mapred.RecordReader<K,V>> rrClass)
org.apache.hadoop.mapred.lib.CombineFileSplit[]	getSplits(org.apache.hadoop.mapred.JobConf job, int numSplits)
protected List<org.apache.hadoop.fs.FileStatus>	listStatus(org.apache.hadoop.mapreduce.JobContext job)
void	setHoodieFilter(boolean hoodieFilter)
void	setRealTime(boolean realTime)

Methods inherited from class org.apache.hadoop.mapred.lib.CombineFileInputFormat

createPool, createRecordReader, isSplitable, isSplitable, listStatus

Methods inherited from class org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat

createPool, createPool, getFileBlockLocations, getSplits, setMaxSplitSize, setMinSplitSizeNode, setMinSplitSizeRack

Methods inherited from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat

addInputPath, addInputPathRecursively, addInputPaths, computeSplitSize, getBlockIndex, getFormatMinSplitSize, getInputDirRecursive, getInputPathFilter, getInputPaths, getMaxSplitSize, getMinSplitSize, makeSplit, makeSplit, setInputDirRecursive, setInputPathFilter, setInputPaths, setInputPaths, setMaxInputSplitSize, setMinInputSplitSize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HoodieCombineFileInputFormatShim

public HoodieCombineFileInputFormatShim()

Method Detail

createParquetInputFormat

protected HoodieParquetInputFormat createParquetInputFormat()

createParquetRealtimeInputFormat

protected HoodieParquetRealtimeInputFormat createParquetRealtimeInputFormat()

getInputPathsShim

public org.apache.hadoop.fs.Path[] getInputPathsShim(org.apache.hadoop.mapred.JobConf conf)

Specified by:

getInputPathsShim in interface org.apache.hadoop.hive.shims.HadoopShims.CombineFileInputFormatShim<K,V>

createPool

public void createPool(org.apache.hadoop.mapred.JobConf conf,
                       org.apache.hadoop.fs.PathFilter... filters)

Specified by:

createPool in interface org.apache.hadoop.hive.shims.HadoopShims.CombineFileInputFormatShim<K,V>

Overrides:

createPool in class org.apache.hadoop.mapred.lib.CombineFileInputFormat<K,V>

getRecordReader

public org.apache.hadoop.mapred.RecordReader<K,V> getRecordReader(org.apache.hadoop.mapred.InputSplit split,
                                                                  org.apache.hadoop.mapred.JobConf job,
                                                                  org.apache.hadoop.mapred.Reporter reporter)
                                                           throws IOException

Specified by:

getRecordReader in interface org.apache.hadoop.mapred.InputFormat<K,V>

Specified by:

getRecordReader in class org.apache.hadoop.mapred.lib.CombineFileInputFormat<K,V>

Throws:

IOException

listStatus

protected List<org.apache.hadoop.fs.FileStatus> listStatus(org.apache.hadoop.mapreduce.JobContext job)
                                                    throws IOException

Overrides:

listStatus in class org.apache.hadoop.mapreduce.lib.input.FileInputFormat<K,V>

Throws:

IOException

getSplits

public org.apache.hadoop.mapred.lib.CombineFileSplit[] getSplits(org.apache.hadoop.mapred.JobConf job,
                                                                 int numSplits)
                                                          throws IOException

Specified by:

getSplits in interface org.apache.hadoop.hive.shims.HadoopShims.CombineFileInputFormatShim<K,V>

Specified by:

getSplits in interface org.apache.hadoop.mapred.InputFormat<K,V>

Overrides:

getSplits in class org.apache.hadoop.mapred.lib.CombineFileInputFormat<K,V>

Throws:

IOException

getInputSplitShim

public org.apache.hadoop.hive.shims.HadoopShimsSecure.InputSplitShim getInputSplitShim()

Specified by:

getInputSplitShim in interface org.apache.hadoop.hive.shims.HadoopShims.CombineFileInputFormatShim<K,V>

getRecordReader

public org.apache.hadoop.mapred.RecordReader getRecordReader(org.apache.hadoop.mapred.JobConf job,
                                                             org.apache.hadoop.mapred.lib.CombineFileSplit split,
                                                             org.apache.hadoop.mapred.Reporter reporter,
                                                             Class<org.apache.hadoop.mapred.RecordReader<K,V>> rrClass)
                                                      throws IOException

Specified by:

getRecordReader in interface org.apache.hadoop.hive.shims.HadoopShims.CombineFileInputFormatShim<K,V>

Throws:

IOException

setHoodieFilter

public void setHoodieFilter(boolean hoodieFilter)

setRealTime

public void setRealTime(boolean realTime)