org.apache.hudi.hadoop.realtime

Class HoodieRealtimeFileSplit

java.lang.Object

org.apache.hadoop.mapreduce.InputSplit

org.apache.hadoop.mapred.FileSplit

org.apache.hudi.hadoop.realtime.HoodieRealtimeFileSplit

All Implemented Interfaces:

org.apache.hadoop.io.Writable, org.apache.hadoop.mapred.InputSplit, org.apache.hadoop.mapred.InputSplitWithLocationInfo, RealtimeSplit

public class HoodieRealtimeFileSplit
extends org.apache.hadoop.mapred.FileSplit
implements RealtimeSplit

FileSplit implementation that holds

1. Split corresponding to the base file
2. List of HoodieLogFile that holds the delta to be merged (upon reading)

This split is correspondent to a single file-slice in the Hudi terminology. NOTE: If you're adding fields here you need to make sure that you appropriately de-/serialize them in RealtimeSplit.readFromInput(DataInput) and RealtimeSplit.writeToOutput(DataOutput)

Constructor Summary

Constructors

Constructor and Description
HoodieRealtimeFileSplit()
HoodieRealtimeFileSplit(org.apache.hadoop.mapred.FileSplit baseSplit, HoodieRealtimePath path)
HoodieRealtimeFileSplit(org.apache.hadoop.mapred.FileSplit baseSplit, String basePath, List<HoodieLogFile> deltaLogFiles, String maxCommitTime, boolean belongsToIncrementalQuery, Option<HoodieVirtualKeyInfo> virtualKeyInfo)

Method Summary

Modifier and Type	Method and Description
String	getBasePath() Return Base Path of the dataset.
boolean	getBelongsToIncrementalQuery() Returns the flag whether this split belongs to an Incremental Query
List<HoodieLogFile>	getDeltaLogFiles()
String	getMaxCommitTime() Return Max Instant Time.
Option<HoodieVirtualKeyInfo>	getVirtualKeyInfo() Returns Virtual key info if meta fields are disabled.
void	readFields(DataInput in)
void	setBasePath(String basePath) Set Base Path.
void	setBelongsToIncrementalQuery(boolean belongsToIncrementalPath) Sets the flag whether this split belongs to an Incremental Query
void	setDeltaLogFiles(List<HoodieLogFile> deltaLogFiles)
void	setMaxCommitTime(String maxCommitTime) Update Maximum valid instant time.
void	setVirtualKeyInfo(Option<HoodieVirtualKeyInfo> virtualKeyInfo)
String	toString()
void	write(DataOutput out)

Methods inherited from class org.apache.hadoop.mapred.FileSplit

getLength, getLocationInfo, getLocations, getPath, getStart

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hudi.hadoop.realtime.RealtimeSplit

getDeltaLogPaths, getLength, getPath, getStart, readFromInput, writeToOutput

Methods inherited from interface org.apache.hadoop.mapred.InputSplitWithLocationInfo

getLocationInfo

Methods inherited from interface org.apache.hadoop.mapred.InputSplit

getLocations

Constructor Detail

HoodieRealtimeFileSplit

public HoodieRealtimeFileSplit()

HoodieRealtimeFileSplit

public HoodieRealtimeFileSplit(org.apache.hadoop.mapred.FileSplit baseSplit,
                               HoodieRealtimePath path)
                        throws IOException

Throws:

IOException

HoodieRealtimeFileSplit

public HoodieRealtimeFileSplit(org.apache.hadoop.mapred.FileSplit baseSplit,
                               String basePath,
                               List<HoodieLogFile> deltaLogFiles,
                               String maxCommitTime,
                               boolean belongsToIncrementalQuery,
                               Option<HoodieVirtualKeyInfo> virtualKeyInfo)
                        throws IOException

Throws:

IOException

Method Detail

getDeltaLogFiles

public List<HoodieLogFile> getDeltaLogFiles()

Specified by:

getDeltaLogFiles in interface RealtimeSplit

setDeltaLogFiles

public void setDeltaLogFiles(List<HoodieLogFile> deltaLogFiles)

Specified by:

setDeltaLogFiles in interface RealtimeSplit

getMaxCommitTime

public String getMaxCommitTime()

Description copied from interface: RealtimeSplit

Return Max Instant Time.

Specified by:

getMaxCommitTime in interface RealtimeSplit

Returns:

setMaxCommitTime

public void setMaxCommitTime(String maxCommitTime)

Description copied from interface: RealtimeSplit

Update Maximum valid instant time.

Specified by:

setMaxCommitTime in interface RealtimeSplit

getBasePath

public String getBasePath()

Description copied from interface: RealtimeSplit

Return Base Path of the dataset.

Specified by:

getBasePath in interface RealtimeSplit

Returns:

setBasePath

public void setBasePath(String basePath)

Description copied from interface: RealtimeSplit

Set Base Path.

Specified by:

setBasePath in interface RealtimeSplit

setVirtualKeyInfo

public void setVirtualKeyInfo(Option<HoodieVirtualKeyInfo> virtualKeyInfo)

Specified by:

setVirtualKeyInfo in interface RealtimeSplit

getVirtualKeyInfo

public Option<HoodieVirtualKeyInfo> getVirtualKeyInfo()

Description copied from interface: RealtimeSplit

Returns Virtual key info if meta fields are disabled.

Specified by:

getVirtualKeyInfo in interface RealtimeSplit

Returns:

getBelongsToIncrementalQuery

public boolean getBelongsToIncrementalQuery()

Description copied from interface: RealtimeSplit

Returns the flag whether this split belongs to an Incremental Query

Specified by:

getBelongsToIncrementalQuery in interface RealtimeSplit

setBelongsToIncrementalQuery

public void setBelongsToIncrementalQuery(boolean belongsToIncrementalPath)

Description copied from interface: RealtimeSplit

Sets the flag whether this split belongs to an Incremental Query

Specified by:

setBelongsToIncrementalQuery in interface RealtimeSplit

write

public void write(DataOutput out)
           throws IOException

Specified by:

write in interface org.apache.hadoop.io.Writable

Overrides:

write in class org.apache.hadoop.mapred.FileSplit

Throws:

IOException

readFields

public void readFields(DataInput in)
                throws IOException

Specified by:

readFields in interface org.apache.hadoop.io.Writable

Overrides:

readFields in class org.apache.hadoop.mapred.FileSplit

Throws:

IOException

toString

public String toString()

Overrides:

toString in class org.apache.hadoop.mapred.FileSplit