org.apache.hudi.keygen

Class KeyGenUtils

java.lang.Object

org.apache.hudi.keygen.KeyGenUtils

public class KeyGenUtils
extends Object

Field Summary

Fields

Modifier and Type	Field and Description
static String	DEFAULT_PARTITION_PATH_SEPARATOR
static String	DEFAULT_RECORD_KEY_PARTS_SEPARATOR
protected static String	EMPTY_RECORDKEY_PLACEHOLDER
protected static String	HUDI_DEFAULT_PARTITION_PATH
protected static String	NULL_RECORDKEY_PLACEHOLDER

Constructor Summary

Constructors

Constructor and Description
KeyGenUtils()

Method Summary

Modifier and Type	Method and Description
static void	checkRequiredProperties(TypedProperties props, List<String> checkPropNames)
static BaseHoodieDateTimeParser	createDateTimeParser(TypedProperties props, String parserClass) Create a date time parser class for TimestampBasedKeyGenerator, passing in any configs needed.
static KeyGenerator	createKeyGeneratorByClassName(TypedProperties props) Create a key generator class via reflection, passing in any configs needed.
static String[]	extractRecordKeys(String recordKey) Extracts the record key fields in strings out of the given record key, this is the reverse operation of getRecordKey(GenericRecord, String, boolean).
static String	getPartitionPath(org.apache.avro.generic.GenericRecord record, String partitionPathField, boolean hiveStylePartitioning, boolean encodePartitionPath, boolean consistentLogicalTimestampEnabled)
static String	getPartitionPathFromGenericRecord(org.apache.avro.generic.GenericRecord genericRecord, Option<BaseKeyGenerator> keyGeneratorOpt) Fetches partition path from the GenericRecord.
static String	getRecordKey(org.apache.avro.generic.GenericRecord record, List<String> recordKeyFields, boolean consistentLogicalTimestampEnabled)
static String	getRecordKey(org.apache.avro.generic.GenericRecord record, String recordKeyField, boolean consistentLogicalTimestampEnabled)
static String	getRecordKeyFromGenericRecord(org.apache.avro.generic.GenericRecord genericRecord, Option<BaseKeyGenerator> keyGeneratorOpt) Fetches record key from the GenericRecord.
static String	getRecordPartitionPath(org.apache.avro.generic.GenericRecord record, List<String> partitionPathFields, boolean hiveStylePartitioning, boolean encodePartitionPath, boolean consistentLogicalTimestampEnabled)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

NULL_RECORDKEY_PLACEHOLDER

protected static final String NULL_RECORDKEY_PLACEHOLDER

See Also:

Constant Field Values

EMPTY_RECORDKEY_PLACEHOLDER

protected static final String EMPTY_RECORDKEY_PLACEHOLDER

See Also:

Constant Field Values

HUDI_DEFAULT_PARTITION_PATH

protected static final String HUDI_DEFAULT_PARTITION_PATH

See Also:

Constant Field Values

DEFAULT_PARTITION_PATH_SEPARATOR

public static final String DEFAULT_PARTITION_PATH_SEPARATOR

See Also:

Constant Field Values

DEFAULT_RECORD_KEY_PARTS_SEPARATOR

public static final String DEFAULT_RECORD_KEY_PARTS_SEPARATOR

See Also:

Constant Field Values

Constructor Detail

KeyGenUtils

public KeyGenUtils()

Method Detail

getRecordKeyFromGenericRecord

public static String getRecordKeyFromGenericRecord(org.apache.avro.generic.GenericRecord genericRecord,
                                                   Option<BaseKeyGenerator> keyGeneratorOpt)

Fetches record key from the GenericRecord.

Parameters:

genericRecord - generic record of interest.

keyGeneratorOpt - Optional BaseKeyGenerator. If not, meta field will be used.

Returns:

the record key for the passed in generic record.

getPartitionPathFromGenericRecord

public static String getPartitionPathFromGenericRecord(org.apache.avro.generic.GenericRecord genericRecord,
                                                       Option<BaseKeyGenerator> keyGeneratorOpt)

Fetches partition path from the GenericRecord.

Parameters:

genericRecord - generic record of interest.

keyGeneratorOpt - Optional BaseKeyGenerator. If not, meta field will be used.

Returns:

the partition path for the passed in generic record.

extractRecordKeys

public static String[] extractRecordKeys(String recordKey)

Extracts the record key fields in strings out of the given record key, this is the reverse operation of getRecordKey(GenericRecord, String, boolean).

See Also:

SimpleAvroKeyGenerator, ComplexAvroKeyGenerator

getRecordKey

public static String getRecordKey(org.apache.avro.generic.GenericRecord record,
                                  List<String> recordKeyFields,
                                  boolean consistentLogicalTimestampEnabled)

getRecordPartitionPath

public static String getRecordPartitionPath(org.apache.avro.generic.GenericRecord record,
                                            List<String> partitionPathFields,
                                            boolean hiveStylePartitioning,
                                            boolean encodePartitionPath,
                                            boolean consistentLogicalTimestampEnabled)

getRecordKey

public static String getRecordKey(org.apache.avro.generic.GenericRecord record,
                                  String recordKeyField,
                                  boolean consistentLogicalTimestampEnabled)

getPartitionPath

public static String getPartitionPath(org.apache.avro.generic.GenericRecord record,
                                      String partitionPathField,
                                      boolean hiveStylePartitioning,
                                      boolean encodePartitionPath,
                                      boolean consistentLogicalTimestampEnabled)

createDateTimeParser

public static BaseHoodieDateTimeParser createDateTimeParser(TypedProperties props,
                                                            String parserClass)
                                                     throws IOException

Create a date time parser class for TimestampBasedKeyGenerator, passing in any configs needed.

Throws:

IOException

checkRequiredProperties

public static void checkRequiredProperties(TypedProperties props,
                                           List<String> checkPropNames)

createKeyGeneratorByClassName

public static KeyGenerator createKeyGeneratorByClassName(TypedProperties props)
                                                  throws IOException

Create a key generator class via reflection, passing in any configs needed.

This method is for user-defined classes. To create hudi's built-in key generators, please set proper KeyGeneratorType conf, and use the relevant factory, see HoodieAvroKeyGeneratorFactory.

Throws:

IOException