org.apache.hudi.hadoop

Class HoodieCopyOnWriteTableInputFormat

java.lang.Object

org.apache.hadoop.mapred.FileInputFormat<org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.ArrayWritable>

org.apache.hudi.hadoop.HoodieTableInputFormat

org.apache.hudi.hadoop.HoodieCopyOnWriteTableInputFormat

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, org.apache.hadoop.mapred.InputFormat<org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.ArrayWritable>

Direct Known Subclasses:

HoodieHFileInputFormat, HoodieMergeOnReadTableInputFormat

public class HoodieCopyOnWriteTableInputFormat
extends HoodieTableInputFormat

Base implementation of the Hive's FileInputFormat allowing for reading of Hudi's Copy-on-Write (COW) tables in various configurations:

• Snapshot mode: reading table's state as of particular timestamp (or instant, in Hudi's terms)
• Incremental mode: reading table's state as of particular timestamp (or instant, in Hudi's terms)
• External mode: reading non-Hudi partitions

NOTE: This class is invariant of the underlying file-format of the files being read

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.mapred.FileInputFormat

org.apache.hadoop.mapred.FileInputFormat.Counter

Field Summary

Fields inherited from class org.apache.hudi.hadoop.HoodieTableInputFormat

conf

Fields inherited from class org.apache.hadoop.mapred.FileInputFormat

INPUT_DIR_RECURSIVE, NUM_INPUT_FILES

Constructor Summary

Constructors

Constructor and Description
HoodieCopyOnWriteTableInputFormat()

Method Summary

Modifier and Type	Method and Description
protected boolean	checkIfValidFileSlice(FileSlice fileSlice)
protected org.apache.hadoop.fs.FileStatus	createFileStatusUnchecked(FileSlice fileSlice, Option<HoodieInstant> instantOpt, String basePath, Option<HoodieVirtualKeyInfo> virtualKeyInfoOpt)
protected org.apache.hadoop.fs.FileStatus[]	doListStatus(org.apache.hadoop.mapred.JobConf job) Abstracts and exposes FileInputFormat.listStatus(JobConf) operation to subclasses that lists files (returning an array of FileStatus) corresponding to the input paths specified as part of provided JobConf
protected static org.apache.hadoop.fs.FileStatus	getFileStatusUnchecked(HoodieBaseFile baseFile)
protected static Option<HoodieVirtualKeyInfo>	getHoodieVirtualKeyInfo(HoodieTableMetaClient metaClient)
org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.ArrayWritable>	getRecordReader(org.apache.hadoop.mapred.InputSplit split, org.apache.hadoop.mapred.JobConf job, org.apache.hadoop.mapred.Reporter reporter)
protected boolean	isSplitable(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path filename)
org.apache.hadoop.fs.FileStatus[]	listStatus(org.apache.hadoop.mapred.JobConf job)
protected List<org.apache.hadoop.fs.FileStatus>	listStatusForIncrementalMode(org.apache.hadoop.mapred.JobConf job, HoodieTableMetaClient tableMetaClient, List<org.apache.hadoop.fs.Path> inputPaths, String incrementalTable) Achieves listStatus functionality for an incrementally queried table.
protected org.apache.hadoop.mapred.FileSplit	makeSplit(org.apache.hadoop.fs.Path file, long start, long length, String[] hosts)
protected org.apache.hadoop.mapred.FileSplit	makeSplit(org.apache.hadoop.fs.Path file, long start, long length, String[] hosts, String[] inMemoryHosts)

Methods inherited from class org.apache.hudi.hadoop.HoodieTableInputFormat

getConf, setConf

Methods inherited from class org.apache.hadoop.mapred.FileInputFormat

addInputPath, addInputPathRecursively, addInputPaths, computeSplitSize, getBlockIndex, getInputPathFilter, getInputPaths, getSplitHosts, getSplits, setInputPathFilter, setInputPaths, setInputPaths, setMinSplitSize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HoodieCopyOnWriteTableInputFormat

public HoodieCopyOnWriteTableInputFormat()

Method Detail

isSplitable

protected boolean isSplitable(org.apache.hadoop.fs.FileSystem fs,
                              org.apache.hadoop.fs.Path filename)

Overrides:

isSplitable in class HoodieTableInputFormat

makeSplit

protected org.apache.hadoop.mapred.FileSplit makeSplit(org.apache.hadoop.fs.Path file,
                                                       long start,
                                                       long length,
                                                       String[] hosts)

Overrides:

makeSplit in class HoodieTableInputFormat

makeSplit

protected org.apache.hadoop.mapred.FileSplit makeSplit(org.apache.hadoop.fs.Path file,
                                                       long start,
                                                       long length,
                                                       String[] hosts,
                                                       String[] inMemoryHosts)

Overrides:

makeSplit in class HoodieTableInputFormat

listStatus

public org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.mapred.JobConf job)
                                             throws IOException

Overrides:

listStatus in class HoodieTableInputFormat

Throws:

IOException

getRecordReader

public org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.ArrayWritable> getRecordReader(org.apache.hadoop.mapred.InputSplit split,
                                                                                                                                   org.apache.hadoop.mapred.JobConf job,
                                                                                                                                   org.apache.hadoop.mapred.Reporter reporter)
                                                                                                                            throws IOException

Specified by:

getRecordReader in interface org.apache.hadoop.mapred.InputFormat<org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.ArrayWritable>

Specified by:

getRecordReader in class org.apache.hadoop.mapred.FileInputFormat<org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.ArrayWritable>

Throws:

IOException

doListStatus

protected final org.apache.hadoop.fs.FileStatus[] doListStatus(org.apache.hadoop.mapred.JobConf job)
                                                        throws IOException

Abstracts and exposes FileInputFormat.listStatus(JobConf) operation to subclasses that lists files (returning an array of FileStatus) corresponding to the input paths specified as part of provided JobConf

Throws:

IOException

listStatusForIncrementalMode

protected List<org.apache.hadoop.fs.FileStatus> listStatusForIncrementalMode(org.apache.hadoop.mapred.JobConf job,
                                                                             HoodieTableMetaClient tableMetaClient,
                                                                             List<org.apache.hadoop.fs.Path> inputPaths,
                                                                             String incrementalTable)
                                                                      throws IOException

Achieves listStatus functionality for an incrementally queried table. Instead of listing all partitions and then filtering based on the commits of interest, this logic first extracts the partitions touched by the desired commits and then lists only those partitions.

Throws:

IOException

createFileStatusUnchecked

protected org.apache.hadoop.fs.FileStatus createFileStatusUnchecked(FileSlice fileSlice,
                                                                    Option<HoodieInstant> instantOpt,
                                                                    String basePath,
                                                                    Option<HoodieVirtualKeyInfo> virtualKeyInfoOpt)

checkIfValidFileSlice

protected boolean checkIfValidFileSlice(FileSlice fileSlice)

getFileStatusUnchecked

@Nonnull
protected static org.apache.hadoop.fs.FileStatus getFileStatusUnchecked(HoodieBaseFile baseFile)

getHoodieVirtualKeyInfo

protected static Option<HoodieVirtualKeyInfo> getHoodieVirtualKeyInfo(HoodieTableMetaClient metaClient)