org.apache.hudi.hadoop.utils

Class HoodieRealtimeInputFormatUtils

java.lang.Object

org.apache.hudi.hadoop.utils.HoodieInputFormatUtils

org.apache.hudi.hadoop.utils.HoodieRealtimeInputFormatUtils

public class HoodieRealtimeInputFormatUtils
extends HoodieInputFormatUtils

Field Summary

Fields inherited from class org.apache.hudi.hadoop.utils.HoodieInputFormatUtils

HOODIE_COMMIT_TIME_COL_POS, HOODIE_PARTITION_PATH_COL_POS, HOODIE_READ_COLUMNS_PROP, HOODIE_RECORD_KEY_COL_POS

Constructor Summary

Constructors

Constructor and Description
HoodieRealtimeInputFormatUtils()

Method Summary

Modifier and Type	Method and Description
static void	addRequiredProjectionFields(org.apache.hadoop.conf.Configuration configuration, Option<HoodieVirtualKeyInfo> hoodieVirtualKeyInfo)
static boolean	canAddProjectionToJobConf(RealtimeSplit realtimeSplit, org.apache.hadoop.mapred.JobConf jobConf)
static void	cleanProjectionColumnIds(org.apache.hadoop.conf.Configuration conf) Hive will append read columns' ids to old columns' ids during getRecordReader.
static boolean	doesBelongToIncrementalQuery(org.apache.hadoop.mapred.FileSplit s)
static boolean	requiredProjectionFieldsExistInConf(org.apache.hadoop.conf.Configuration configuration, Option<HoodieVirtualKeyInfo> hoodieVirtualKeyInfo)

Methods inherited from class org.apache.hudi.hadoop.utils.HoodieInputFormatUtils

buildMetadataConfig, filterIncrementalFileStatus, filterInstantsTimeline, getAffectedPartitions, getCommitMetadata, getCommitsForIncrementalQuery, getFileStatus, getFilteredCommitsTimeline, getHoodieTimelineForIncrementalQuery, getInputFormat, getInputFormat, getInputFormatClassName, getOutputFormatClassName, getSerDeClassName, getTableMetaClientByPartitionPath, getTableMetaClientForBasePathUnchecked, getWritePartitionPaths, groupFileStatusForSnapshotPaths, groupSnapshotPathsByMetaClient, listAffectedFilesForCommits, refreshFileStatus

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HoodieRealtimeInputFormatUtils

public HoodieRealtimeInputFormatUtils()

Method Detail

doesBelongToIncrementalQuery

public static boolean doesBelongToIncrementalQuery(org.apache.hadoop.mapred.FileSplit s)

addRequiredProjectionFields

public static void addRequiredProjectionFields(org.apache.hadoop.conf.Configuration configuration,
                                               Option<HoodieVirtualKeyInfo> hoodieVirtualKeyInfo)

requiredProjectionFieldsExistInConf

public static boolean requiredProjectionFieldsExistInConf(org.apache.hadoop.conf.Configuration configuration,
                                                          Option<HoodieVirtualKeyInfo> hoodieVirtualKeyInfo)

canAddProjectionToJobConf

public static boolean canAddProjectionToJobConf(RealtimeSplit realtimeSplit,
                                                org.apache.hadoop.mapred.JobConf jobConf)

cleanProjectionColumnIds

public static void cleanProjectionColumnIds(org.apache.hadoop.conf.Configuration conf)

Hive will append read columns' ids to old columns' ids during getRecordReader. In some cases, e.g. SELECT COUNT(*), the read columns' id is an empty string and Hive will combine it with Hoodie required projection ids and becomes e.g. ",2,0,3" and will cause an error. Actually this method is a temporary solution because the real bug is from Hive. Hive has fixed this bug after 3.0.0, but the version before that would still face this problem. (HIVE-22438)