org.apache.hudi.hadoop.realtime

Class HoodieMergeOnReadTableInputFormat

java.lang.Object

org.apache.hadoop.mapred.FileInputFormat<org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.ArrayWritable>

org.apache.hudi.hadoop.HoodieTableInputFormat

org.apache.hudi.hadoop.HoodieCopyOnWriteTableInputFormat

org.apache.hudi.hadoop.realtime.HoodieMergeOnReadTableInputFormat

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, org.apache.hadoop.mapred.InputFormat<org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.ArrayWritable>

Direct Known Subclasses:

HoodieHFileRealtimeInputFormat

public class HoodieMergeOnReadTableInputFormat
extends HoodieCopyOnWriteTableInputFormat
implements org.apache.hadoop.conf.Configurable

Base implementation of the Hive's FileInputFormat allowing for reading of Hudi's Merge-on-Read (COW) tables in various configurations:

• Snapshot mode: reading table's state as of particular timestamp (or instant, in Hudi's terms)
• Incremental mode: reading table's state as of particular timestamp (or instant, in Hudi's terms)
• External mode: reading non-Hudi partitions

NOTE: This class is invariant of the underlying file-format of the files being read

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.mapred.FileInputFormat

org.apache.hadoop.mapred.FileInputFormat.Counter

Field Summary

Fields inherited from class org.apache.hudi.hadoop.HoodieTableInputFormat

conf

Fields inherited from class org.apache.hadoop.mapred.FileInputFormat

INPUT_DIR_RECURSIVE, NUM_INPUT_FILES

Constructor Summary

Constructors

Constructor and Description
HoodieMergeOnReadTableInputFormat()

Method Summary

Modifier and Type	Method and Description
protected boolean	checkIfValidFileSlice(FileSlice fileSlice)
protected org.apache.hadoop.fs.FileStatus	createFileStatusUnchecked(FileSlice fileSlice, Option<HoodieInstant> latestCompletedInstantOpt, String tableBasePath, Option<HoodieVirtualKeyInfo> virtualKeyInfoOpt)
org.apache.hadoop.mapred.InputSplit[]	getSplits(org.apache.hadoop.mapred.JobConf job, int numSplits)
protected boolean	isSplitable(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path filename)
protected List<org.apache.hadoop.fs.FileStatus>	listStatusForIncrementalMode(org.apache.hadoop.mapred.JobConf job, HoodieTableMetaClient tableMetaClient, List<org.apache.hadoop.fs.Path> inputPaths, String incrementalTableName) Keep the logic of mor_incr_view as same as spark datasource.
protected org.apache.hadoop.mapred.FileSplit	makeSplit(org.apache.hadoop.fs.Path file, long start, long length, String[] hosts)
protected org.apache.hadoop.mapred.FileSplit	makeSplit(org.apache.hadoop.fs.Path file, long start, long length, String[] hosts, String[] inMemoryHosts)

Methods inherited from class org.apache.hudi.hadoop.HoodieCopyOnWriteTableInputFormat

doListStatus, getFileStatusUnchecked, getHoodieVirtualKeyInfo, getRecordReader, listStatus

Methods inherited from class org.apache.hudi.hadoop.HoodieTableInputFormat

getConf, setConf

Methods inherited from class org.apache.hadoop.mapred.FileInputFormat

addInputPath, addInputPathRecursively, addInputPaths, computeSplitSize, getBlockIndex, getInputPathFilter, getInputPaths, getSplitHosts, setInputPathFilter, setInputPaths, setInputPaths, setMinSplitSize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Constructor Detail

HoodieMergeOnReadTableInputFormat

public HoodieMergeOnReadTableInputFormat()

Method Detail

getSplits

public org.apache.hadoop.mapred.InputSplit[] getSplits(org.apache.hadoop.mapred.JobConf job,
                                                       int numSplits)
                                                throws IOException

Specified by:

getSplits in interface org.apache.hadoop.mapred.InputFormat<org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.ArrayWritable>

Overrides:

getSplits in class org.apache.hadoop.mapred.FileInputFormat<org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.ArrayWritable>

Throws:

IOException

createFileStatusUnchecked

protected org.apache.hadoop.fs.FileStatus createFileStatusUnchecked(FileSlice fileSlice,
                                                                    Option<HoodieInstant> latestCompletedInstantOpt,
                                                                    String tableBasePath,
                                                                    Option<HoodieVirtualKeyInfo> virtualKeyInfoOpt)

Overrides:

createFileStatusUnchecked in class HoodieCopyOnWriteTableInputFormat

checkIfValidFileSlice

protected boolean checkIfValidFileSlice(FileSlice fileSlice)

Overrides:

checkIfValidFileSlice in class HoodieCopyOnWriteTableInputFormat

listStatusForIncrementalMode

protected List<org.apache.hadoop.fs.FileStatus> listStatusForIncrementalMode(org.apache.hadoop.mapred.JobConf job,
                                                                             HoodieTableMetaClient tableMetaClient,
                                                                             List<org.apache.hadoop.fs.Path> inputPaths,
                                                                             String incrementalTableName)
                                                                      throws IOException

Keep the logic of mor_incr_view as same as spark datasource. Step1: Get list of commits to be fetched based on start commit and max commits(for snapshot max commits is -1). Step2: Get list of affected files status for these affected file status. Step3: Construct HoodieTableFileSystemView based on those affected file status. a. Filter affected partitions based on inputPaths. b. Get list of fileGroups based on affected partitions by fsView.getAllFileGroups. Step4: Set input paths based on filtered affected partition paths. changes that amony original input paths passed to this method. some partitions did not have commits as part of the trimmed down list of commits and hence we need this step. Step5: Find candidate fileStatus, since when we get baseFileStatus from HoodieTableFileSystemView, the BaseFileStatus will missing file size information. We should use candidate fileStatus to update the size information for BaseFileStatus. Step6: For every file group from step3(b) Get 1st available base file from all file slices. then we use candidate file status to update the baseFileStatus, and construct RealTimeFileStatus and add it to result along with log files. If file group just has log files, construct RealTimeFileStatus and add it to result. TODO: unify the incremental view code between hive/spark-sql and spark datasource

Overrides:

listStatusForIncrementalMode in class HoodieCopyOnWriteTableInputFormat

Throws:

IOException

isSplitable

protected boolean isSplitable(org.apache.hadoop.fs.FileSystem fs,
                              org.apache.hadoop.fs.Path filename)

Overrides:

isSplitable in class HoodieCopyOnWriteTableInputFormat

makeSplit

protected org.apache.hadoop.mapred.FileSplit makeSplit(org.apache.hadoop.fs.Path file,
                                                       long start,
                                                       long length,
                                                       String[] hosts)

Overrides:

makeSplit in class HoodieCopyOnWriteTableInputFormat

makeSplit

protected org.apache.hadoop.mapred.FileSplit makeSplit(org.apache.hadoop.fs.Path file,
                                                       long start,
                                                       long length,
                                                       String[] hosts,
                                                       String[] inMemoryHosts)

Overrides:

makeSplit in class HoodieCopyOnWriteTableInputFormat