org.apache.hudi.hadoop.hive

Class HoodieCombineHiveInputFormat<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>

java.lang.Object

org.apache.hadoop.hive.ql.io.HiveInputFormat<K,V>

org.apache.hudi.hadoop.hive.HoodieCombineHiveInputFormat<K,V>

All Implemented Interfaces:

org.apache.hadoop.mapred.InputFormat<K,V>, org.apache.hadoop.mapred.JobConfigurable

public class HoodieCombineHiveInputFormat<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>
extends org.apache.hadoop.hive.ql.io.HiveInputFormat<K,V>

This is just a copy of the org.apache.hadoop.hive.ql.io.CombineHiveInputFormat from Hive 2.x Search for **MOD** to see minor modifications to support custom inputformat in CombineHiveInputFormat. See https://issues.apache.org/jira/browse/HIVE-9771

CombineHiveInputFormat is a parameterized InputFormat which looks at the path name and determine the correct InputFormat for that path name from mapredPlan.pathToPartitionInfo(). It can be used to read files with different input format in the same map-reduce job. NOTE : This class is implemented to work with Hive 2.x +

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	HoodieCombineHiveInputFormat.AvoidSplitCombination This is a marker interface that is used to identify the formats where combine split generation is not applicable.
static class	HoodieCombineHiveInputFormat.CombineHiveInputSplit CombineHiveInputSplit encapsulates an InputSplit with its corresponding inputFormatClassName.
static class	HoodieCombineHiveInputFormat.HoodieCombineFileInputFormatShim<K,V> **MOD** This is the implementation of CombineFileInputFormat which is a copy of org.apache.hadoop.hive.shims.HadoopShimsSecure.CombineFileInputFormatShim with changes in listStatus.

Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.io.HiveInputFormat

org.apache.hadoop.hive.ql.io.HiveInputFormat.HiveInputSplit

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.log4j.Logger	LOG

Fields inherited from class org.apache.hadoop.hive.ql.io.HiveInputFormat

mrwork, pathToPartitionInfo

Constructor Summary

Constructors

Constructor and Description
HoodieCombineHiveInputFormat()

Method Summary

Modifier and Type	Method and Description
protected HoodieCombineHiveInputFormat.HoodieCombineFileInputFormatShim	createInputFormatShim()
Set<Integer>	getNonCombinablePathIndices(org.apache.hadoop.mapred.JobConf job, org.apache.hadoop.fs.Path[] paths, int numThreads) Gets all the path indices that should not be combined.
protected String	getParquetInputFormatClassName()
protected String	getParquetRealtimeInputFormatClassName()
org.apache.hadoop.mapred.RecordReader	getRecordReader(org.apache.hadoop.mapred.InputSplit split, org.apache.hadoop.mapred.JobConf job, org.apache.hadoop.mapred.Reporter reporter) Create a generic Hive RecordReader than can iterate over all chunks in a CombinedFileSplit.
org.apache.hadoop.mapred.InputSplit[]	getSplits(org.apache.hadoop.mapred.JobConf job, int numSplits) Create Hive splits based on CombineFileSplit.

Methods inherited from class org.apache.hadoop.hive.ql.io.HiveInputFormat

canWrapForLlap, configure, getInputFormatFromCache, getPartitionDescFromPath, init, pushFilters, pushProjectionsAndFilters, pushProjectionsAndFilters, wrapForLlap

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.apache.log4j.Logger LOG

Constructor Detail

HoodieCombineHiveInputFormat

public HoodieCombineHiveInputFormat()

Method Detail

getParquetInputFormatClassName

protected String getParquetInputFormatClassName()

getParquetRealtimeInputFormatClassName

protected String getParquetRealtimeInputFormatClassName()

createInputFormatShim

protected HoodieCombineHiveInputFormat.HoodieCombineFileInputFormatShim createInputFormatShim()

getNonCombinablePathIndices

public Set<Integer> getNonCombinablePathIndices(org.apache.hadoop.mapred.JobConf job,
                                                org.apache.hadoop.fs.Path[] paths,
                                                int numThreads)
                                         throws ExecutionException,
                                                InterruptedException

Gets all the path indices that should not be combined.

Throws:

ExecutionException

InterruptedException

getSplits

public org.apache.hadoop.mapred.InputSplit[] getSplits(org.apache.hadoop.mapred.JobConf job,
                                                       int numSplits)
                                                throws IOException

Create Hive splits based on CombineFileSplit.

Specified by:

getSplits in interface org.apache.hadoop.mapred.InputFormat<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>

Overrides:

getSplits in class org.apache.hadoop.hive.ql.io.HiveInputFormat<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>

Throws:

IOException

getRecordReader

public org.apache.hadoop.mapred.RecordReader getRecordReader(org.apache.hadoop.mapred.InputSplit split,
                                                             org.apache.hadoop.mapred.JobConf job,
                                                             org.apache.hadoop.mapred.Reporter reporter)
                                                      throws IOException

Create a generic Hive RecordReader than can iterate over all chunks in a CombinedFileSplit.

Specified by:

getRecordReader in interface org.apache.hadoop.mapred.InputFormat<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>

Overrides:

getRecordReader in class org.apache.hadoop.hive.ql.io.HiveInputFormat<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>

Throws:

IOException