org.apache.hudi.common.table

Class HoodieTableConfig

java.lang.Object

org.apache.hudi.common.config.HoodieConfig

org.apache.hudi.common.table.HoodieTableConfig

All Implemented Interfaces:

Serializable

public class HoodieTableConfig
extends HoodieConfig

Configurations on the Hoodie Table like type of ingestion, storage formats, hive table name etc Configurations are loaded from hoodie.properties, these properties are usually set during initializing a path as hoodie base path and never changes during the lifetime of a hoodie table.

Since:

0.3.0

See Also:

HoodieTableMetaClient, Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static ConfigProperty<String>	ARCHIVELOG_FOLDER
static ConfigProperty<HoodieFileFormat>	BASE_FILE_FORMAT
static ConfigProperty<String>	BOOTSTRAP_BASE_PATH
static ConfigProperty<String>	BOOTSTRAP_INDEX_CLASS_NAME
static ConfigProperty<Boolean>	BOOTSTRAP_INDEX_ENABLE
static ConfigProperty<String>	CREATE_SCHEMA
static ConfigProperty<String>	DATABASE_NAME
static String	DEFAULT_ARCHIVELOG_FOLDER Deprecated. Use ARCHIVELOG_FOLDER and its methods.
static HoodieFileFormat	DEFAULT_BASE_FILE_FORMAT Deprecated. Use BASE_FILE_FORMAT and its methods.
static String	DEFAULT_BOOTSTRAP_INDEX_CLASS Deprecated. Use BOOTSTRAP_INDEX_CLASS_NAME and its methods.
static HoodieFileFormat	DEFAULT_LOG_FILE_FORMAT Deprecated. Use LOG_FILE_FORMAT and its methods.
static String	DEFAULT_PAYLOAD_CLASS Deprecated. Use PAYLOAD_CLASS_NAME and its methods.
static HoodieTableType	DEFAULT_TABLE_TYPE Deprecated. Use TYPE and its methods.
static HoodieTableVersion	DEFAULT_TABLE_VERSION Deprecated. Use VERSION and its methods.
static ConfigProperty<Boolean>	DROP_PARTITION_COLUMNS
static ConfigProperty<String>	HIVE_STYLE_PARTITIONING_ENABLE
static String	HOODIE_ARCHIVELOG_FOLDER_PROP_NAME Deprecated. Use ARCHIVELOG_FOLDER and its methods.
static String	HOODIE_BASE_FILE_FORMAT_PROP_NAME Deprecated. Use BASE_FILE_FORMAT and its methods.
static String	HOODIE_BOOTSTRAP_BASE_PATH Deprecated. Use BOOTSTRAP_BASE_PATH and its methods.
static String	HOODIE_BOOTSTRAP_INDEX_CLASS_PROP_NAME Deprecated. Use BOOTSTRAP_INDEX_CLASS_NAME and its methods.
static String	HOODIE_LOG_FILE_FORMAT_PROP_NAME Deprecated. Use LOG_FILE_FORMAT and its methods.
static String	HOODIE_PAYLOAD_CLASS_PROP_NAME Deprecated. Use PAYLOAD_CLASS_NAME and its methods.
static String	HOODIE_PROPERTIES_FILE
static String	HOODIE_PROPERTIES_FILE_BACKUP
static String	HOODIE_RO_FILE_FORMAT_PROP_NAME Deprecated. Use BASE_FILE_FORMAT and its methods.
static String	HOODIE_RT_FILE_FORMAT_PROP_NAME Deprecated. Use LOG_FILE_FORMAT and its methods.
static String	HOODIE_TABLE_NAME_KEY
static String	HOODIE_TABLE_NAME_PROP_NAME Deprecated. Use NAME and its methods.
static String	HOODIE_TABLE_PRECOMBINE_FIELD Deprecated. Use PRECOMBINE_FIELD and its methods.
static String	HOODIE_TABLE_TYPE_PROP_NAME Deprecated. Use TYPE and its methods.
static String	HOODIE_TABLE_VERSION_PROP_NAME Deprecated. Use VERSION and its methods.
static String	HOODIE_TIMELINE_LAYOUT_VERSION Deprecated. Use TIMELINE_LAYOUT_VERSION and its methods.
static String	HOODIE_WRITE_TABLE_NAME_KEY
static ConfigProperty<String>	KEY_GENERATOR_CLASS_NAME
static ConfigProperty<HoodieFileFormat>	LOG_FILE_FORMAT
static ConfigProperty<String>	NAME
static String	NO_OP_BOOTSTRAP_INDEX_CLASS
static ConfigProperty<String>	PARTITION_FIELDS
static ConfigProperty<Boolean>	PARTITION_METAFILE_USE_BASE_FORMAT
static ConfigProperty<String>	PAYLOAD_CLASS_NAME
static List<String>	PERSISTED_CONFIG_LIST
static ConfigProperty<Boolean>	POPULATE_META_FIELDS
static ConfigProperty<String>	PRECOMBINE_FIELD
static ConfigProperty<String>	RECORDKEY_FIELDS
static ConfigProperty<String>	TABLE_CHECKSUM
static ConfigProperty<String>	TABLE_METADATA_PARTITIONS
static ConfigProperty<String>	TABLE_METADATA_PARTITIONS_INFLIGHT
static ConfigProperty<String>	TIMELINE_LAYOUT_VERSION
static ConfigProperty<HoodieTimelineTimeZone>	TIMELINE_TIMEZONE
static ConfigProperty<HoodieTableType>	TYPE
static ConfigProperty<String>	URL_ENCODE_PARTITIONING
static ConfigProperty<HoodieTableVersion>	VERSION

Fields inherited from class org.apache.hudi.common.config.HoodieConfig

CONFIG_VALUES_DELIMITER, props

Constructor Summary

Constructors

Constructor and Description
HoodieTableConfig() For serializing and de-serializing.
HoodieTableConfig(org.apache.hadoop.fs.FileSystem fs, String metaPath, String payloadClassName)

Method Summary

Modifier and Type	Method and Description
static void	create(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path metadataFolder, Properties properties) Initialize the hoodie meta directory and any necessary files inside the meta (including the hoodie.properties).
static void	delete(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path metadataFolder, Set<String> deletedProps)
static long	generateChecksum(Properties props)
String	getArchivelogFolder() Get the relative path of archive log folder under metafolder, for this table.
HoodieFileFormat	getBaseFileFormat() Get the base file storage format.
Option<String>	getBootstrapBasePath()
String	getBootstrapIndexClass() Read the payload class for HoodieRecords from the table properties.
String	getDatabaseName() Read the database name.
static String	getDefaultBootstrapIndexClass(Properties props)
String	getHiveStylePartitioningEnable()
String	getKeyGeneratorClassName()
HoodieFileFormat	getLogFileFormat() Get the log Storage Format.
Set<String>	getMetadataPartitions()
List<String>	getMetadataPartitionsInflight()
String	getPartitionFieldProp() Deprecated. please use getPartitionFields() instead
Option<String[]>	getPartitionFields()
Option<HoodieFileFormat>	getPartitionMetafileFormat() Returns the format to use for partition meta files.
String	getPayloadClass() Read the payload class for HoodieRecords from the table properties.
String	getPreCombineField()
String	getRecordKeyFieldProp()
Option<String[]>	getRecordKeyFields()
Option<org.apache.avro.Schema>	getTableCreateSchema()
String	getTableName() Read the table name.
HoodieTableType	getTableType() Read the table type from the table properties and if not found, return the default.
HoodieTableVersion	getTableVersion()
Option<TimelineLayoutVersion>	getTimelineLayoutVersion()
String	getUrlEncodePartitioning()
boolean	isTablePartitioned()
boolean	populateMetaFields()
Map<String,String>	propsMap()
static void	recover(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path metadataFolder)
void	setTableVersion(HoodieTableVersion tableVersion)
Boolean	shouldDropPartitionColumns()
static void	update(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path metadataFolder, Properties updatedProps) Upserts the table config with the set of properties passed in.
static boolean	validateChecksum(Properties props)

Methods inherited from class org.apache.hudi.common.config.HoodieConfig

contains, contains, create, getBoolean, getBooleanOrDefault, getBooleanOrDefault, getBooleanOrDefault, getDouble, getFloat, getInt, getIntOrDefault, getLong, getProps, getProps, getSplitStrings, getSplitStrings, getString, getString, getStringOrDefault, getStringOrDefault, getStringOrThrow, setAll, setDefault, setDefaultOnCondition, setDefaults, setDefaultValue, setDefaultValue, setValue, setValue

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

HOODIE_PROPERTIES_FILE

public static final String HOODIE_PROPERTIES_FILE

See Also:

Constant Field Values

HOODIE_PROPERTIES_FILE_BACKUP

public static final String HOODIE_PROPERTIES_FILE_BACKUP

See Also:

Constant Field Values

HOODIE_WRITE_TABLE_NAME_KEY

public static final String HOODIE_WRITE_TABLE_NAME_KEY

See Also:

Constant Field Values

HOODIE_TABLE_NAME_KEY

public static final String HOODIE_TABLE_NAME_KEY

See Also:

Constant Field Values

DATABASE_NAME

public static final ConfigProperty<String> DATABASE_NAME

NAME

public static final ConfigProperty<String> NAME

TYPE

public static final ConfigProperty<HoodieTableType> TYPE

VERSION

public static final ConfigProperty<HoodieTableVersion> VERSION

PRECOMBINE_FIELD

public static final ConfigProperty<String> PRECOMBINE_FIELD

PARTITION_FIELDS

public static final ConfigProperty<String> PARTITION_FIELDS

RECORDKEY_FIELDS

public static final ConfigProperty<String> RECORDKEY_FIELDS

CREATE_SCHEMA

public static final ConfigProperty<String> CREATE_SCHEMA

BASE_FILE_FORMAT

public static final ConfigProperty<HoodieFileFormat> BASE_FILE_FORMAT

LOG_FILE_FORMAT

public static final ConfigProperty<HoodieFileFormat> LOG_FILE_FORMAT

TIMELINE_LAYOUT_VERSION

public static final ConfigProperty<String> TIMELINE_LAYOUT_VERSION

PAYLOAD_CLASS_NAME

public static final ConfigProperty<String> PAYLOAD_CLASS_NAME

ARCHIVELOG_FOLDER

public static final ConfigProperty<String> ARCHIVELOG_FOLDER

BOOTSTRAP_INDEX_ENABLE

public static final ConfigProperty<Boolean> BOOTSTRAP_INDEX_ENABLE

BOOTSTRAP_INDEX_CLASS_NAME

public static final ConfigProperty<String> BOOTSTRAP_INDEX_CLASS_NAME

BOOTSTRAP_BASE_PATH

public static final ConfigProperty<String> BOOTSTRAP_BASE_PATH

POPULATE_META_FIELDS

public static final ConfigProperty<Boolean> POPULATE_META_FIELDS

KEY_GENERATOR_CLASS_NAME

public static final ConfigProperty<String> KEY_GENERATOR_CLASS_NAME

TIMELINE_TIMEZONE

public static final ConfigProperty<HoodieTimelineTimeZone> TIMELINE_TIMEZONE

PARTITION_METAFILE_USE_BASE_FORMAT

public static final ConfigProperty<Boolean> PARTITION_METAFILE_USE_BASE_FORMAT

DROP_PARTITION_COLUMNS

public static final ConfigProperty<Boolean> DROP_PARTITION_COLUMNS

URL_ENCODE_PARTITIONING

public static final ConfigProperty<String> URL_ENCODE_PARTITIONING

HIVE_STYLE_PARTITIONING_ENABLE

public static final ConfigProperty<String> HIVE_STYLE_PARTITIONING_ENABLE

PERSISTED_CONFIG_LIST

public static final List<String> PERSISTED_CONFIG_LIST

NO_OP_BOOTSTRAP_INDEX_CLASS

public static final String NO_OP_BOOTSTRAP_INDEX_CLASS

TABLE_CHECKSUM

public static final ConfigProperty<String> TABLE_CHECKSUM

TABLE_METADATA_PARTITIONS_INFLIGHT

public static final ConfigProperty<String> TABLE_METADATA_PARTITIONS_INFLIGHT

TABLE_METADATA_PARTITIONS

public static final ConfigProperty<String> TABLE_METADATA_PARTITIONS

HOODIE_RO_FILE_FORMAT_PROP_NAME

@Deprecated
public static final String HOODIE_RO_FILE_FORMAT_PROP_NAME

Deprecated. Use BASE_FILE_FORMAT and its methods.

See Also:

Constant Field Values

HOODIE_RT_FILE_FORMAT_PROP_NAME

@Deprecated
public static final String HOODIE_RT_FILE_FORMAT_PROP_NAME

Deprecated. Use LOG_FILE_FORMAT and its methods.

See Also:

Constant Field Values

HOODIE_TABLE_NAME_PROP_NAME

@Deprecated
public static final String HOODIE_TABLE_NAME_PROP_NAME

Deprecated. Use NAME and its methods.

HOODIE_TABLE_TYPE_PROP_NAME

@Deprecated
public static final String HOODIE_TABLE_TYPE_PROP_NAME

Deprecated. Use TYPE and its methods.

HOODIE_TABLE_VERSION_PROP_NAME

@Deprecated
public static final String HOODIE_TABLE_VERSION_PROP_NAME

Deprecated. Use VERSION and its methods.

HOODIE_TABLE_PRECOMBINE_FIELD

@Deprecated
public static final String HOODIE_TABLE_PRECOMBINE_FIELD

Deprecated. Use PRECOMBINE_FIELD and its methods.

HOODIE_BASE_FILE_FORMAT_PROP_NAME

@Deprecated
public static final String HOODIE_BASE_FILE_FORMAT_PROP_NAME

Deprecated. Use BASE_FILE_FORMAT and its methods.

HOODIE_LOG_FILE_FORMAT_PROP_NAME

@Deprecated
public static final String HOODIE_LOG_FILE_FORMAT_PROP_NAME

Deprecated. Use LOG_FILE_FORMAT and its methods.

HOODIE_TIMELINE_LAYOUT_VERSION

@Deprecated
public static final String HOODIE_TIMELINE_LAYOUT_VERSION

Deprecated. Use TIMELINE_LAYOUT_VERSION and its methods.

HOODIE_PAYLOAD_CLASS_PROP_NAME

@Deprecated
public static final String HOODIE_PAYLOAD_CLASS_PROP_NAME

Deprecated. Use PAYLOAD_CLASS_NAME and its methods.

HOODIE_ARCHIVELOG_FOLDER_PROP_NAME

@Deprecated
public static final String HOODIE_ARCHIVELOG_FOLDER_PROP_NAME

Deprecated. Use ARCHIVELOG_FOLDER and its methods.

HOODIE_BOOTSTRAP_INDEX_CLASS_PROP_NAME

@Deprecated
public static final String HOODIE_BOOTSTRAP_INDEX_CLASS_PROP_NAME

Deprecated. Use BOOTSTRAP_INDEX_CLASS_NAME and its methods.

HOODIE_BOOTSTRAP_BASE_PATH

@Deprecated
public static final String HOODIE_BOOTSTRAP_BASE_PATH

Deprecated. Use BOOTSTRAP_BASE_PATH and its methods.

DEFAULT_TABLE_TYPE

@Deprecated
public static final HoodieTableType DEFAULT_TABLE_TYPE

Deprecated. Use TYPE and its methods.

DEFAULT_TABLE_VERSION

@Deprecated
public static final HoodieTableVersion DEFAULT_TABLE_VERSION

Deprecated. Use VERSION and its methods.

DEFAULT_BASE_FILE_FORMAT

@Deprecated
public static final HoodieFileFormat DEFAULT_BASE_FILE_FORMAT

Deprecated. Use BASE_FILE_FORMAT and its methods.

DEFAULT_LOG_FILE_FORMAT

@Deprecated
public static final HoodieFileFormat DEFAULT_LOG_FILE_FORMAT

Deprecated. Use LOG_FILE_FORMAT and its methods.

DEFAULT_PAYLOAD_CLASS

@Deprecated
public static final String DEFAULT_PAYLOAD_CLASS

Deprecated. Use PAYLOAD_CLASS_NAME and its methods.

DEFAULT_BOOTSTRAP_INDEX_CLASS

@Deprecated
public static final String DEFAULT_BOOTSTRAP_INDEX_CLASS

Deprecated. Use BOOTSTRAP_INDEX_CLASS_NAME and its methods.

DEFAULT_ARCHIVELOG_FOLDER

@Deprecated
public static final String DEFAULT_ARCHIVELOG_FOLDER

Deprecated. Use ARCHIVELOG_FOLDER and its methods.

Constructor Detail

HoodieTableConfig

public HoodieTableConfig(org.apache.hadoop.fs.FileSystem fs,
                         String metaPath,
                         String payloadClassName)

HoodieTableConfig

public HoodieTableConfig()

For serializing and de-serializing.

Method Detail

recover

public static void recover(org.apache.hadoop.fs.FileSystem fs,
                           org.apache.hadoop.fs.Path metadataFolder)
                    throws IOException

Throws:

IOException

update

public static void update(org.apache.hadoop.fs.FileSystem fs,
                          org.apache.hadoop.fs.Path metadataFolder,
                          Properties updatedProps)

Upserts the table config with the set of properties passed in. We implement a fail-safe backup protocol here for safely updating with recovery and also ensuring the table config continues to be readable.

delete

public static void delete(org.apache.hadoop.fs.FileSystem fs,
                          org.apache.hadoop.fs.Path metadataFolder,
                          Set<String> deletedProps)

create

public static void create(org.apache.hadoop.fs.FileSystem fs,
                          org.apache.hadoop.fs.Path metadataFolder,
                          Properties properties)
                   throws IOException

Initialize the hoodie meta directory and any necessary files inside the meta (including the hoodie.properties).

Throws:

IOException

generateChecksum

public static long generateChecksum(Properties props)

validateChecksum

public static boolean validateChecksum(Properties props)

getTableType

public HoodieTableType getTableType()

Read the table type from the table properties and if not found, return the default.

getTimelineLayoutVersion

public Option<TimelineLayoutVersion> getTimelineLayoutVersion()

getTableVersion

public HoodieTableVersion getTableVersion()

Returns:

the hoodie.table.version from hoodie.properties file.

setTableVersion

public void setTableVersion(HoodieTableVersion tableVersion)

getPayloadClass

public String getPayloadClass()

Read the payload class for HoodieRecords from the table properties.

getPreCombineField

public String getPreCombineField()

getRecordKeyFields

public Option<String[]> getRecordKeyFields()

getPartitionFields

public Option<String[]> getPartitionFields()

isTablePartitioned

public boolean isTablePartitioned()

getPartitionFieldProp

@Deprecated
public String getPartitionFieldProp()

Deprecated. please use getPartitionFields() instead

getBootstrapIndexClass

public String getBootstrapIndexClass()

Read the payload class for HoodieRecords from the table properties.

getDefaultBootstrapIndexClass

public static String getDefaultBootstrapIndexClass(Properties props)

getBootstrapBasePath

public Option<String> getBootstrapBasePath()

getTableCreateSchema

public Option<org.apache.avro.Schema> getTableCreateSchema()

getDatabaseName

public String getDatabaseName()

Read the database name.

getTableName

public String getTableName()

Read the table name.

getBaseFileFormat

public HoodieFileFormat getBaseFileFormat()

Get the base file storage format.

Returns:

HoodieFileFormat for the base file Storage format

getLogFileFormat

public HoodieFileFormat getLogFileFormat()

Get the log Storage Format.

Returns:

HoodieFileFormat for the log Storage format

getArchivelogFolder

public String getArchivelogFolder()

Get the relative path of archive log folder under metafolder, for this table.

populateMetaFields

public boolean populateMetaFields()

getRecordKeyFieldProp

public String getRecordKeyFieldProp()

getKeyGeneratorClassName

public String getKeyGeneratorClassName()

getHiveStylePartitioningEnable

public String getHiveStylePartitioningEnable()

getUrlEncodePartitioning

public String getUrlEncodePartitioning()

shouldDropPartitionColumns

public Boolean shouldDropPartitionColumns()

getMetadataPartitionsInflight

public List<String> getMetadataPartitionsInflight()

getMetadataPartitions

public Set<String> getMetadataPartitions()

getPartitionMetafileFormat

public Option<HoodieFileFormat> getPartitionMetafileFormat()

Returns the format to use for partition meta files.

propsMap

public Map<String,String> propsMap()