org.apache.hudi.hadoop.utils

Class HoodieInputFormatUtils

java.lang.Object

org.apache.hudi.hadoop.utils.HoodieInputFormatUtils

Direct Known Subclasses:

HoodieRealtimeInputFormatUtils

public class HoodieInputFormatUtils
extends Object

Field Summary

Fields

Modifier and Type	Field and Description
static int	HOODIE_COMMIT_TIME_COL_POS
static int	HOODIE_PARTITION_PATH_COL_POS
static String	HOODIE_READ_COLUMNS_PROP
static int	HOODIE_RECORD_KEY_COL_POS

Constructor Summary

Constructors

Constructor and Description
HoodieInputFormatUtils()

Method Summary

Modifier and Type	Method and Description
static HoodieMetadataConfig	buildMetadataConfig(org.apache.hadoop.conf.Configuration conf)
static List<org.apache.hadoop.fs.FileStatus>	filterIncrementalFileStatus(org.apache.hadoop.mapreduce.Job job, HoodieTableMetaClient tableMetaClient, HoodieTimeline timeline, org.apache.hadoop.fs.FileStatus[] fileStatuses, List<HoodieInstant> commitsToCheck) Filter a list of FileStatus based on commitsToCheck for incremental view.
static HoodieDefaultTimeline	filterInstantsTimeline(HoodieDefaultTimeline timeline) Filter any specific instants that we do not want to process.
static Option<String>	getAffectedPartitions(List<HoodieInstant> commitsToCheck, HoodieTableMetaClient tableMetaClient, HoodieTimeline timeline, List<org.apache.hadoop.fs.Path> inputPaths) Extract partitions touched by the commitsToCheck.
static HoodieCommitMetadata	getCommitMetadata(HoodieInstant instant, HoodieTimeline timeline) Returns the commit metadata of the given instant.
static Option<List<HoodieInstant>>	getCommitsForIncrementalQuery(org.apache.hadoop.mapreduce.Job job, String tableName, HoodieTimeline timeline) Get commits for incremental query from Hive map reduce configuration.
static org.apache.hadoop.fs.FileStatus	getFileStatus(HoodieBaseFile baseFile)
static Option<HoodieTimeline>	getFilteredCommitsTimeline(org.apache.hadoop.mapreduce.JobContext job, HoodieTableMetaClient tableMetaClient) Extract HoodieTimeline based on HoodieTableMetaClient.
static HoodieTimeline	getHoodieTimelineForIncrementalQuery(org.apache.hadoop.mapreduce.JobContext job, String tableName, HoodieTimeline timeline) Get HoodieTimeline for incremental query from Hive map reduce configuration.
static org.apache.hadoop.mapred.FileInputFormat	getInputFormat(HoodieFileFormat baseFileFormat, boolean realtime, org.apache.hadoop.conf.Configuration conf)
static org.apache.hadoop.mapred.FileInputFormat	getInputFormat(String path, boolean realtime, org.apache.hadoop.conf.Configuration conf)
static String	getInputFormatClassName(HoodieFileFormat baseFileFormat, boolean realtime)
static String	getOutputFormatClassName(HoodieFileFormat baseFileFormat)
static String	getSerDeClassName(HoodieFileFormat baseFileFormat)
static Map<org.apache.hadoop.fs.Path,HoodieTableMetaClient>	getTableMetaClientByPartitionPath(org.apache.hadoop.conf.Configuration conf, Set<org.apache.hadoop.fs.Path> partitions) Extract HoodieTableMetaClient by partition path.
static HoodieTableMetaClient	getTableMetaClientForBasePathUnchecked(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path partitionPath) Extract HoodieTableMetaClient from a partition path (not base path)
static Set<String>	getWritePartitionPaths(List<HoodieCommitMetadata> metadataList) Returns all the incremental write partition paths as a set with the given commits metadata.
static Map<HoodieTableMetaClient,List<org.apache.hadoop.fs.FileStatus>>	groupFileStatusForSnapshotPaths(org.apache.hadoop.fs.FileStatus[] fileStatuses, String fileExtension, Collection<HoodieTableMetaClient> metaClientList) Takes in a list of filesStatus and a list of table metadatas.
static Map<HoodieTableMetaClient,List<org.apache.hadoop.fs.Path>>	groupSnapshotPathsByMetaClient(Collection<HoodieTableMetaClient> metaClientList, List<org.apache.hadoop.fs.Path> snapshotPaths)
static org.apache.hadoop.fs.FileStatus[]	listAffectedFilesForCommits(org.apache.hadoop.conf.Configuration hadoopConf, org.apache.hadoop.fs.Path basePath, List<HoodieCommitMetadata> metadataList) Iterate through a list of commit metadata in natural order, and extract the file status of all affected files from the commits metadata grouping by file full path.
static HoodieBaseFile	refreshFileStatus(org.apache.hadoop.conf.Configuration conf, HoodieBaseFile dataFile) Checks the file status for a race condition which can set the file size to 0.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

HOODIE_COMMIT_TIME_COL_POS

public static final int HOODIE_COMMIT_TIME_COL_POS

See Also:

Constant Field Values

HOODIE_RECORD_KEY_COL_POS

public static final int HOODIE_RECORD_KEY_COL_POS

See Also:

Constant Field Values

HOODIE_PARTITION_PATH_COL_POS

public static final int HOODIE_PARTITION_PATH_COL_POS

See Also:

Constant Field Values

HOODIE_READ_COLUMNS_PROP

public static final String HOODIE_READ_COLUMNS_PROP

See Also:

Constant Field Values

Constructor Detail

HoodieInputFormatUtils

public HoodieInputFormatUtils()

Method Detail

getInputFormat

public static org.apache.hadoop.mapred.FileInputFormat getInputFormat(HoodieFileFormat baseFileFormat,
                                                                      boolean realtime,
                                                                      org.apache.hadoop.conf.Configuration conf)

getInputFormatClassName

public static String getInputFormatClassName(HoodieFileFormat baseFileFormat,
                                             boolean realtime)

getOutputFormatClassName

public static String getOutputFormatClassName(HoodieFileFormat baseFileFormat)

getSerDeClassName

public static String getSerDeClassName(HoodieFileFormat baseFileFormat)

getInputFormat

public static org.apache.hadoop.mapred.FileInputFormat getInputFormat(String path,
                                                                      boolean realtime,
                                                                      org.apache.hadoop.conf.Configuration conf)

filterInstantsTimeline

public static HoodieDefaultTimeline filterInstantsTimeline(HoodieDefaultTimeline timeline)

Filter any specific instants that we do not want to process. example timeline: t0 -> create bucket1.parquet t1 -> create and append updates bucket1.log t2 -> request compaction t3 -> create bucket2.parquet if compaction at t2 takes a long time, incremental readers on RO tables can move to t3 and would skip updates in t1 To workaround this problem, we want to stop returning data belonging to commits > t2. After compaction is complete, incremental reader would see updates in t2, t3, so on.

Parameters:

timeline -

Returns:

getAffectedPartitions

public static Option<String> getAffectedPartitions(List<HoodieInstant> commitsToCheck,
                                                   HoodieTableMetaClient tableMetaClient,
                                                   HoodieTimeline timeline,
                                                   List<org.apache.hadoop.fs.Path> inputPaths)
                                            throws IOException

Extract partitions touched by the commitsToCheck.

Parameters:

commitsToCheck -

tableMetaClient -

timeline -

inputPaths -

Returns:

Throws:

IOException

getFilteredCommitsTimeline

public static Option<HoodieTimeline> getFilteredCommitsTimeline(org.apache.hadoop.mapreduce.JobContext job,
                                                                HoodieTableMetaClient tableMetaClient)

Extract HoodieTimeline based on HoodieTableMetaClient.

Parameters:

job -

tableMetaClient -

Returns:

getCommitsForIncrementalQuery

public static Option<List<HoodieInstant>> getCommitsForIncrementalQuery(org.apache.hadoop.mapreduce.Job job,
                                                                        String tableName,
                                                                        HoodieTimeline timeline)

Get commits for incremental query from Hive map reduce configuration.

Parameters:

job -

tableName -

timeline -

Returns:

getHoodieTimelineForIncrementalQuery

public static HoodieTimeline getHoodieTimelineForIncrementalQuery(org.apache.hadoop.mapreduce.JobContext job,
                                                                  String tableName,
                                                                  HoodieTimeline timeline)

Get HoodieTimeline for incremental query from Hive map reduce configuration.

Parameters:

job -

tableName -

timeline -

Returns:

getTableMetaClientByPartitionPath

public static Map<org.apache.hadoop.fs.Path,HoodieTableMetaClient> getTableMetaClientByPartitionPath(org.apache.hadoop.conf.Configuration conf,
                                                                                                     Set<org.apache.hadoop.fs.Path> partitions)

Extract HoodieTableMetaClient by partition path.

Parameters:

conf - The hadoop conf

partitions - The partitions

Returns:

partition path to table meta client mapping

getTableMetaClientForBasePathUnchecked

public static HoodieTableMetaClient getTableMetaClientForBasePathUnchecked(org.apache.hadoop.conf.Configuration conf,
                                                                           org.apache.hadoop.fs.Path partitionPath)
                                                                    throws IOException

Extract HoodieTableMetaClient from a partition path (not base path)

Throws:

IOException

getFileStatus

public static org.apache.hadoop.fs.FileStatus getFileStatus(HoodieBaseFile baseFile)
                                                     throws IOException

Throws:

IOException

filterIncrementalFileStatus

public static List<org.apache.hadoop.fs.FileStatus> filterIncrementalFileStatus(org.apache.hadoop.mapreduce.Job job,
                                                                                HoodieTableMetaClient tableMetaClient,
                                                                                HoodieTimeline timeline,
                                                                                org.apache.hadoop.fs.FileStatus[] fileStatuses,
                                                                                List<HoodieInstant> commitsToCheck)
                                                                         throws IOException

Filter a list of FileStatus based on commitsToCheck for incremental view.

Parameters:

job -

tableMetaClient -

timeline -

fileStatuses -

commitsToCheck -

Returns:

Throws:

IOException

groupFileStatusForSnapshotPaths

public static Map<HoodieTableMetaClient,List<org.apache.hadoop.fs.FileStatus>> groupFileStatusForSnapshotPaths(org.apache.hadoop.fs.FileStatus[] fileStatuses,
                                                                                                               String fileExtension,
                                                                                                               Collection<HoodieTableMetaClient> metaClientList)

Takes in a list of filesStatus and a list of table metadatas. Groups the files status list based on given table metadata.

Parameters:

fileStatuses -

fileExtension -

metaClientList -

Returns:

Throws:

IOException

groupSnapshotPathsByMetaClient

public static Map<HoodieTableMetaClient,List<org.apache.hadoop.fs.Path>> groupSnapshotPathsByMetaClient(Collection<HoodieTableMetaClient> metaClientList,
                                                                                                        List<org.apache.hadoop.fs.Path> snapshotPaths)

buildMetadataConfig

public static HoodieMetadataConfig buildMetadataConfig(org.apache.hadoop.conf.Configuration conf)

refreshFileStatus

public static HoodieBaseFile refreshFileStatus(org.apache.hadoop.conf.Configuration conf,
                                               HoodieBaseFile dataFile)

Checks the file status for a race condition which can set the file size to 0. 1. HiveInputFormat does super.listStatus() and gets back a FileStatus[] 2. Then it creates the HoodieTableMetaClient for the paths listed. 3. Generation of splits looks at FileStatus size to create splits, which skips this file

Parameters:

conf -

dataFile -

Returns:

listAffectedFilesForCommits

public static org.apache.hadoop.fs.FileStatus[] listAffectedFilesForCommits(org.apache.hadoop.conf.Configuration hadoopConf,
                                                                            org.apache.hadoop.fs.Path basePath,
                                                                            List<HoodieCommitMetadata> metadataList)

Iterate through a list of commit metadata in natural order, and extract the file status of all affected files from the commits metadata grouping by file full path. If the files has been touched multiple times in the given commits, the return value will keep the one from the latest commit.

Parameters:

basePath - The table base path

metadataList - The metadata list to read the data from

Returns:

the affected file status array

getWritePartitionPaths

public static Set<String> getWritePartitionPaths(List<HoodieCommitMetadata> metadataList)

Returns all the incremental write partition paths as a set with the given commits metadata.

Parameters:

metadataList - The commits metadata

Returns:

the partition path set

getCommitMetadata

public static HoodieCommitMetadata getCommitMetadata(HoodieInstant instant,
                                                     HoodieTimeline timeline)
                                              throws IOException

Returns the commit metadata of the given instant.

Parameters:

instant - The hoodie instant

timeline - The timeline

Returns:

the commit metadata

Throws:

IOException