org.apache.hudi.hadoop

Class HoodieParquetInputFormatBase

java.lang.Object

org.apache.hadoop.mapred.FileInputFormat<org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.ArrayWritable>

org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat

org.apache.hudi.hadoop.HoodieParquetInputFormatBase

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, org.apache.hadoop.hive.ql.exec.vector.VectorizedInputFormatInterface, org.apache.hadoop.mapred.InputFormat<org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.ArrayWritable>

Direct Known Subclasses:

HoodieParquetInputFormat

public abstract class HoodieParquetInputFormatBase
extends org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat
implements org.apache.hadoop.conf.Configurable

!!! PLEASE READ CAREFULLY !!! NOTE: Hive bears optimizations which are based upon validating whether FileInputFormat implementation inherits from MapredParquetInputFormat. To make sure that Hudi implementations are leveraging these optimizations to the fullest, this class serves as a base-class for every FileInputFormat implementations working with Parquet file-format. However, this class serves as a simple delegate to the actual implementation hierarchy: it expects either HoodieCopyOnWriteTableInputFormat or HoodieMergeOnReadTableInputFormat to be supplied to which it delegates all of its necessary methods.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.mapred.FileInputFormat

org.apache.hadoop.mapred.FileInputFormat.Counter

Field Summary

Fields inherited from class org.apache.hadoop.mapred.FileInputFormat

INPUT_DIR_RECURSIVE, NUM_INPUT_FILES

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	HoodieParquetInputFormatBase(HoodieCopyOnWriteTableInputFormat inputFormatDelegate)

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.conf.Configuration	getConf()
org.apache.hadoop.mapred.InputSplit[]	getSplits(org.apache.hadoop.mapred.JobConf job, int numSplits)
protected boolean	isSplitable(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path filename)
org.apache.hadoop.fs.FileStatus[]	listStatus(org.apache.hadoop.mapred.JobConf job)
protected org.apache.hadoop.mapred.FileSplit	makeSplit(org.apache.hadoop.fs.Path file, long start, long length, String[] hosts)
protected org.apache.hadoop.mapred.FileSplit	makeSplit(org.apache.hadoop.fs.Path file, long start, long length, String[] hosts, String[] inMemoryHosts)
void	setConf(org.apache.hadoop.conf.Configuration conf)

Methods inherited from class org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat

getRecordReader

Methods inherited from class org.apache.hadoop.mapred.FileInputFormat

addInputPath, addInputPathRecursively, addInputPaths, computeSplitSize, getBlockIndex, getInputPathFilter, getInputPaths, getSplitHosts, setInputPathFilter, setInputPaths, setInputPaths, setMinSplitSize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HoodieParquetInputFormatBase

protected HoodieParquetInputFormatBase(HoodieCopyOnWriteTableInputFormat inputFormatDelegate)

Method Detail

setConf

public final void setConf(org.apache.hadoop.conf.Configuration conf)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

getConf

public final org.apache.hadoop.conf.Configuration getConf()

Specified by:

getConf in interface org.apache.hadoop.conf.Configurable

getSplits

public final org.apache.hadoop.mapred.InputSplit[] getSplits(org.apache.hadoop.mapred.JobConf job,
                                                             int numSplits)
                                                      throws IOException

Specified by:

getSplits in interface org.apache.hadoop.mapred.InputFormat<org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.ArrayWritable>

Overrides:

getSplits in class org.apache.hadoop.mapred.FileInputFormat<org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.ArrayWritable>

Throws:

IOException

isSplitable

protected final boolean isSplitable(org.apache.hadoop.fs.FileSystem fs,
                                    org.apache.hadoop.fs.Path filename)

Overrides:

isSplitable in class org.apache.hadoop.mapred.FileInputFormat<org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.ArrayWritable>

makeSplit

protected final org.apache.hadoop.mapred.FileSplit makeSplit(org.apache.hadoop.fs.Path file,
                                                             long start,
                                                             long length,
                                                             String[] hosts)

Overrides:

makeSplit in class org.apache.hadoop.mapred.FileInputFormat<org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.ArrayWritable>

makeSplit

protected final org.apache.hadoop.mapred.FileSplit makeSplit(org.apache.hadoop.fs.Path file,
                                                             long start,
                                                             long length,
                                                             String[] hosts,
                                                             String[] inMemoryHosts)

Overrides:

makeSplit in class org.apache.hadoop.mapred.FileInputFormat<org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.ArrayWritable>

listStatus

public final org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.mapred.JobConf job)
                                                   throws IOException

Overrides:

listStatus in class org.apache.hadoop.mapred.FileInputFormat<org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.ArrayWritable>

Throws:

IOException