org.apache.hudi.keygen

Class KeyGenUtils

java.lang.Object

org.apache.hudi.keygen.KeyGenUtils

public class KeyGenUtils
extends Object

Field Summary

Fields

Modifier and Type	Field and Description
static String	DEFAULT_COMPOSITE_KEY_FILED_VALUE
static String	DEFAULT_PARTITION_PATH_SEPARATOR
static String	DEFAULT_RECORD_KEY_PARTS_SEPARATOR
protected static String	EMPTY_RECORDKEY_PLACEHOLDER
protected static String	HUDI_DEFAULT_PARTITION_PATH
protected static String	NULL_RECORDKEY_PLACEHOLDER
static String	RECORD_KEY_GEN_INSTANT_TIME_CONFIG
static String	RECORD_KEY_GEN_PARTITION_ID_CONFIG

Constructor Summary

Constructors

Constructor and Description
KeyGenUtils()

Method Summary

Modifier and Type	Method and Description
static BaseHoodieDateTimeParser	createDateTimeParser(TypedProperties props, String parserClass) Create a date time parser class for TimestampBasedKeyGenerator, passing in any configs needed.
static KeyGenerator	createKeyGeneratorByClassName(TypedProperties props) Create a key generator class via reflection, passing in any configs needed.
static boolean	enableAutoGenerateRecordKeys(TypedProperties props)
static String[]	extractRecordKeys(String recordKey) Extracts the record key fields in strings out of the given record key, this is the reverse operation of getRecordKey(GenericRecord, String, boolean).
static String[]	extractRecordKeysByFields(String recordKey, List<String> fields)
static String	getPartitionPath(org.apache.avro.generic.GenericRecord record, String partitionPathField, boolean hiveStylePartitioning, boolean encodePartitionPath, boolean consistentLogicalTimestampEnabled)
static String	getPartitionPathFromGenericRecord(org.apache.avro.generic.GenericRecord genericRecord, Option<BaseKeyGenerator> keyGeneratorOpt) Fetches partition path from the GenericRecord.
static String	getRecordKey(org.apache.avro.generic.GenericRecord record, List<String> recordKeyFields, boolean consistentLogicalTimestampEnabled)
static String	getRecordKey(org.apache.avro.generic.GenericRecord record, String recordKeyField, boolean consistentLogicalTimestampEnabled)
static List<String>	getRecordKeyFields(TypedProperties props)
static String	getRecordKeyFromGenericRecord(org.apache.avro.generic.GenericRecord genericRecord, Option<BaseKeyGenerator> keyGeneratorOpt) Fetches record key from the GenericRecord.
static String	getRecordPartitionPath(org.apache.avro.generic.GenericRecord record, List<String> partitionPathFields, boolean hiveStylePartitioning, boolean encodePartitionPath, boolean consistentLogicalTimestampEnabled)
static KeyGeneratorType	inferKeyGeneratorType(Option<String> recordsKeyFields, String partitionFields) Infers the key generator type based on the record key and partition fields.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

NULL_RECORDKEY_PLACEHOLDER

protected static final String NULL_RECORDKEY_PLACEHOLDER

See Also:

Constant Field Values

EMPTY_RECORDKEY_PLACEHOLDER

protected static final String EMPTY_RECORDKEY_PLACEHOLDER

See Also:

Constant Field Values

HUDI_DEFAULT_PARTITION_PATH

protected static final String HUDI_DEFAULT_PARTITION_PATH

See Also:

Constant Field Values

DEFAULT_PARTITION_PATH_SEPARATOR

public static final String DEFAULT_PARTITION_PATH_SEPARATOR

See Also:

Constant Field Values

DEFAULT_RECORD_KEY_PARTS_SEPARATOR

public static final String DEFAULT_RECORD_KEY_PARTS_SEPARATOR

See Also:

Constant Field Values

DEFAULT_COMPOSITE_KEY_FILED_VALUE

public static final String DEFAULT_COMPOSITE_KEY_FILED_VALUE

See Also:

Constant Field Values

RECORD_KEY_GEN_PARTITION_ID_CONFIG

public static final String RECORD_KEY_GEN_PARTITION_ID_CONFIG

See Also:

Constant Field Values

RECORD_KEY_GEN_INSTANT_TIME_CONFIG

public static final String RECORD_KEY_GEN_INSTANT_TIME_CONFIG

See Also:

Constant Field Values

Constructor Detail

KeyGenUtils

public KeyGenUtils()

Method Detail

inferKeyGeneratorType

public static KeyGeneratorType inferKeyGeneratorType(Option<String> recordsKeyFields,
                                                     String partitionFields)

Infers the key generator type based on the record key and partition fields.

(1) partition field is empty: KeyGeneratorType.NON_PARTITION; (2) Only one partition field and one record key field: KeyGeneratorType.SIMPLE; (3) More than one partition and/or record key fields: KeyGeneratorType.COMPLEX.

Parameters:

recordsKeyFields - Record key field list.

partitionFields - Partition field list.

Returns:

Inferred key generator type.

getRecordKeyFromGenericRecord

public static String getRecordKeyFromGenericRecord(org.apache.avro.generic.GenericRecord genericRecord,
                                                   Option<BaseKeyGenerator> keyGeneratorOpt)

Fetches record key from the GenericRecord.

Parameters:

genericRecord - generic record of interest.

keyGeneratorOpt - Optional BaseKeyGenerator. If not, meta field will be used.

Returns:

the record key for the passed in generic record.

getPartitionPathFromGenericRecord

public static String getPartitionPathFromGenericRecord(org.apache.avro.generic.GenericRecord genericRecord,
                                                       Option<BaseKeyGenerator> keyGeneratorOpt)

Fetches partition path from the GenericRecord.

Parameters:

genericRecord - generic record of interest.

keyGeneratorOpt - Optional BaseKeyGenerator. If not, meta field will be used.

Returns:

the partition path for the passed in generic record.

extractRecordKeys

public static String[] extractRecordKeys(String recordKey)

Extracts the record key fields in strings out of the given record key, this is the reverse operation of getRecordKey(GenericRecord, String, boolean).

See Also:

SimpleAvroKeyGenerator, ComplexAvroKeyGenerator

extractRecordKeysByFields

public static String[] extractRecordKeysByFields(String recordKey,
                                                 List<String> fields)

getRecordKey

public static String getRecordKey(org.apache.avro.generic.GenericRecord record,
                                  List<String> recordKeyFields,
                                  boolean consistentLogicalTimestampEnabled)

getRecordPartitionPath

public static String getRecordPartitionPath(org.apache.avro.generic.GenericRecord record,
                                            List<String> partitionPathFields,
                                            boolean hiveStylePartitioning,
                                            boolean encodePartitionPath,
                                            boolean consistentLogicalTimestampEnabled)

getRecordKey

public static String getRecordKey(org.apache.avro.generic.GenericRecord record,
                                  String recordKeyField,
                                  boolean consistentLogicalTimestampEnabled)

getPartitionPath

public static String getPartitionPath(org.apache.avro.generic.GenericRecord record,
                                      String partitionPathField,
                                      boolean hiveStylePartitioning,
                                      boolean encodePartitionPath,
                                      boolean consistentLogicalTimestampEnabled)

createDateTimeParser

public static BaseHoodieDateTimeParser createDateTimeParser(TypedProperties props,
                                                            String parserClass)
                                                     throws IOException

Create a date time parser class for TimestampBasedKeyGenerator, passing in any configs needed.

Throws:

IOException

createKeyGeneratorByClassName

public static KeyGenerator createKeyGeneratorByClassName(TypedProperties props)
                                                  throws IOException

Create a key generator class via reflection, passing in any configs needed.

This method is for user-defined classes. To create hudi's built-in key generators, please set proper KeyGeneratorType conf, and use the relevant factory, see HoodieAvroKeyGeneratorFactory.

Throws:

IOException

getRecordKeyFields

public static List<String> getRecordKeyFields(TypedProperties props)

enableAutoGenerateRecordKeys

public static boolean enableAutoGenerateRecordKeys(TypedProperties props)

Parameters:

props - props of interest.

Returns:

true if record keys need to be auto generated. false otherwise.