org.apache.hudi.avro

Class HoodieAvroUtils

java.lang.Object

org.apache.hudi.avro.HoodieAvroUtils

public class HoodieAvroUtils
extends Object

Helper class to do common stuff across Avro.

Field Summary

Fields

Modifier and Type	Field and Description
static String	AVRO_VERSION
static org.apache.avro.Conversions.DecimalConversion	DECIMAL_CONVERSION
static org.apache.avro.Schema	METADATA_FIELD_SCHEMA
static org.apache.avro.Schema	RECORD_KEY_SCHEMA

Constructor Summary

Constructors

Constructor and Description
HoodieAvroUtils()

Method Summary

Modifier and Type	Method and Description
static org.apache.avro.generic.GenericRecord	addCommitMetadataToRecord(org.apache.avro.generic.GenericRecord record, String instantTime, String commitSeqno) Adds the Hoodie commit metadata into the provided Generic Record.
static org.apache.avro.generic.GenericRecord	addHoodieKeyToRecord(org.apache.avro.generic.GenericRecord record, String recordKey, String partitionPath, String fileName)
static String	addMetadataColumnTypes(String hiveColumnTypes)
static org.apache.avro.Schema	addMetadataFields(org.apache.avro.Schema schema) Adds the Hoodie metadata fields to the given schema.
static org.apache.avro.Schema	addMetadataFields(org.apache.avro.Schema schema, boolean withOperationField) Adds the Hoodie metadata fields to the given schema.
static org.apache.avro.generic.GenericRecord	addOperationToRecord(org.apache.avro.generic.GenericRecord record, HoodieOperation operation)
static byte[]	avroToBytes(org.apache.avro.generic.GenericRecord record) Convert a given avro record to bytes.
static byte[]	avroToJson(org.apache.avro.generic.GenericRecord record, boolean pretty) Convert a given avro record to json and return the encoded bytes.
static org.apache.avro.generic.GenericRecord	bytesToAvro(byte[] bytes, org.apache.avro.Schema schema) Convert serialized bytes back into avro record.
static org.apache.avro.generic.GenericRecord	bytesToAvro(byte[] bytes, org.apache.avro.Schema writerSchema, org.apache.avro.Schema readerSchema) Convert serialized bytes back into avro record.
static Object	convertValueForSpecificDataTypes(org.apache.avro.Schema fieldSchema, Object fieldValue, boolean consistentLogicalTimestampEnabled) This method converts values for fields with certain Avro/Parquet data types that require special handling.
static org.apache.avro.Schema	createHoodieWriteSchema(org.apache.avro.Schema originalSchema)
static org.apache.avro.Schema	createHoodieWriteSchema(String originalSchema)
static org.apache.avro.Schema	createHoodieWriteSchema(String originalSchema, boolean withOperationField)
static int	fromJavaDate(Date date) convert Date to days NOTE: This method could only be used in tests
static org.apache.avro.Schema	generateProjectionSchema(org.apache.avro.Schema originalSchema, List<String> fieldNames) Generate a reader schema off the provided writeSchema, to just project out the provided columns.
static Object	getFieldVal(org.apache.avro.generic.GenericRecord record, String key) Obtain value of the provided key, which is consistent with avro before 1.10
static Object	getFieldVal(org.apache.avro.generic.GenericRecord record, String key, boolean returnNullIfNotFound) Obtain value of the provided key, when set returnNullIfNotFound false, it is consistent with avro after 1.10
static org.apache.avro.Schema	getNestedFieldSchemaFromRecord(org.apache.avro.generic.GenericRecord record, String fieldName) Get schema for the given field and record.
static org.apache.avro.Schema	getNestedFieldSchemaFromWriteSchema(org.apache.avro.Schema writeSchema, String fieldName) Get schema for the given field and write schema.
static Object	getNestedFieldVal(org.apache.avro.generic.GenericRecord record, String fieldName, boolean returnNullIfNotFound, boolean consistentLogicalTimestampEnabled) Obtain value of the provided field, denoted by dot notation.
static String	getNestedFieldValAsString(org.apache.avro.generic.GenericRecord record, String fieldName, boolean returnNullIfNotFound, boolean consistentLogicalTimestampEnabled) Obtain value of the provided field as string, denoted by dot notation.
static Option<String>	getNullableValAsString(org.apache.avro.generic.GenericRecord rec, String fieldName) Returns the string value of the given record rec and field fieldName.
static org.apache.avro.Schema	getNullSchema()
static Object	getRecordColumnValues(HoodieRecord<? extends HoodieRecordPayload> record, String[] columns, org.apache.avro.Schema schema, boolean consistentLogicalTimestampEnabled) Gets record column values into one object.
static Object	getRecordColumnValues(HoodieRecord<? extends HoodieRecordPayload> record, String[] columns, SerializableSchema schema, boolean consistentLogicalTimestampEnabled) Gets record column values into one object.
static org.apache.avro.Schema	getRecordKeyPartitionPathSchema() Fetch schema for record key and partition path.
static org.apache.avro.Schema	getRecordKeySchema()
static String	getRootLevelFieldName(String fieldName) Obtain the root-level field name of a full field name, possibly a nested field.
static org.apache.avro.Schema	getSchemaForFields(org.apache.avro.Schema fileSchema, List<String> fields) Fetch schema for record key and partition path.
static boolean	gteqAvro1_10()
static boolean	gteqAvro1_9()
static <T extends org.apache.avro.generic.IndexedRecord> byte[]	indexedRecordToBytes(T record)
static boolean	isMetadataField(String fieldName)
static org.apache.avro.generic.GenericRecord	jsonBytesToAvro(byte[] bytes, org.apache.avro.Schema schema) Convert json bytes back into avro record.
static org.apache.avro.generic.GenericRecord	removeFields(org.apache.avro.generic.GenericRecord record, Set<String> fieldsToRemove) Given an Avro record and list of columns to remove, this method removes the list of columns from the given avro record using rewriteRecord method.
static org.apache.avro.Schema	removeFields(org.apache.avro.Schema schema, Set<String> fieldsToRemove)
static org.apache.avro.Schema	removeMetadataFields(org.apache.avro.Schema schema)
static org.apache.avro.generic.GenericRecord	rewriteEvolutionRecordWithMetadata(org.apache.avro.generic.GenericRecord genericRecord, org.apache.avro.Schema newSchema, String fileName)
static org.apache.avro.generic.GenericRecord	rewriteRecord(org.apache.avro.generic.GenericRecord oldRecord, org.apache.avro.Schema newSchema) Given an Avro record with a given schema, rewrites it into the new schema while setting fields only from the new schema.
static org.apache.avro.generic.GenericRecord	rewriteRecordDeep(org.apache.avro.generic.GenericRecord oldRecord, org.apache.avro.Schema newSchema)
static List<org.apache.avro.generic.GenericRecord>	rewriteRecords(List<org.apache.avro.generic.GenericRecord> records, org.apache.avro.Schema newSchema) Converts list of GenericRecord provided into the GenericRecord adhering to the provided newSchema.
static org.apache.avro.generic.GenericRecord	rewriteRecordWithMetadata(org.apache.avro.generic.GenericRecord genericRecord, org.apache.avro.Schema newSchema, String fileName)
static org.apache.avro.generic.GenericRecord	rewriteRecordWithNewSchema(org.apache.avro.generic.IndexedRecord oldRecord, org.apache.avro.Schema newSchema, Map<String,String> renameCols) Given a avro record with a given schema, rewrites it into the new schema while setting fields only from the new schema.
static Iterator<org.apache.avro.generic.GenericRecord>	rewriteRecordWithNewSchema(Iterator<org.apache.avro.generic.GenericRecord> oldRecords, org.apache.avro.Schema newSchema, Map<String,String> renameCols) Given avro records, rewrites them with new schema.
static String	sanitizeName(String name) Sanitizes Name according to Avro rule for names.
static org.apache.avro.generic.GenericRecord	stitchRecords(org.apache.avro.generic.GenericRecord left, org.apache.avro.generic.GenericRecord right, org.apache.avro.Schema stitchedSchema)
static Date	toJavaDate(int days) convert days to Date NOTE: This method could only be used in tests

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

AVRO_VERSION

public static final String AVRO_VERSION

DECIMAL_CONVERSION

public static final org.apache.avro.Conversions.DecimalConversion DECIMAL_CONVERSION

METADATA_FIELD_SCHEMA

public static final org.apache.avro.Schema METADATA_FIELD_SCHEMA

RECORD_KEY_SCHEMA

public static final org.apache.avro.Schema RECORD_KEY_SCHEMA

Constructor Detail

HoodieAvroUtils

public HoodieAvroUtils()

Method Detail

avroToBytes

public static byte[] avroToBytes(org.apache.avro.generic.GenericRecord record)

Convert a given avro record to bytes.

indexedRecordToBytes

public static <T extends org.apache.avro.generic.IndexedRecord> byte[] indexedRecordToBytes(T record)

avroToJson

public static byte[] avroToJson(org.apache.avro.generic.GenericRecord record,
                                boolean pretty)
                         throws IOException

Convert a given avro record to json and return the encoded bytes.

Parameters:

record - The GenericRecord to convert

pretty - Whether to pretty-print the json output

Throws:

IOException

bytesToAvro

public static org.apache.avro.generic.GenericRecord bytesToAvro(byte[] bytes,
                                                                org.apache.avro.Schema schema)
                                                         throws IOException

Convert serialized bytes back into avro record.

Throws:

IOException

bytesToAvro

public static org.apache.avro.generic.GenericRecord bytesToAvro(byte[] bytes,
                                                                org.apache.avro.Schema writerSchema,
                                                                org.apache.avro.Schema readerSchema)
                                                         throws IOException

Convert serialized bytes back into avro record.

Throws:

IOException

jsonBytesToAvro

public static org.apache.avro.generic.GenericRecord jsonBytesToAvro(byte[] bytes,
                                                                    org.apache.avro.Schema schema)
                                                             throws IOException

Convert json bytes back into avro record.

Throws:

IOException

isMetadataField

public static boolean isMetadataField(String fieldName)

createHoodieWriteSchema

public static org.apache.avro.Schema createHoodieWriteSchema(org.apache.avro.Schema originalSchema)

createHoodieWriteSchema

public static org.apache.avro.Schema createHoodieWriteSchema(String originalSchema)

createHoodieWriteSchema

public static org.apache.avro.Schema createHoodieWriteSchema(String originalSchema,
                                                             boolean withOperationField)

addMetadataFields

public static org.apache.avro.Schema addMetadataFields(org.apache.avro.Schema schema)

Adds the Hoodie metadata fields to the given schema.

Parameters:

schema - The schema

addMetadataFields

public static org.apache.avro.Schema addMetadataFields(org.apache.avro.Schema schema,
                                                       boolean withOperationField)

Adds the Hoodie metadata fields to the given schema.

Parameters:

schema - The schema

withOperationField - Whether to include the '_hoodie_operation' field

removeMetadataFields

public static org.apache.avro.Schema removeMetadataFields(org.apache.avro.Schema schema)

removeFields

public static org.apache.avro.Schema removeFields(org.apache.avro.Schema schema,
                                                  Set<String> fieldsToRemove)

addMetadataColumnTypes

public static String addMetadataColumnTypes(String hiveColumnTypes)

getRecordKeySchema

public static org.apache.avro.Schema getRecordKeySchema()

getRecordKeyPartitionPathSchema

public static org.apache.avro.Schema getRecordKeyPartitionPathSchema()

Fetch schema for record key and partition path.

getSchemaForFields

public static org.apache.avro.Schema getSchemaForFields(org.apache.avro.Schema fileSchema,
                                                        List<String> fields)

Fetch schema for record key and partition path.

addHoodieKeyToRecord

public static org.apache.avro.generic.GenericRecord addHoodieKeyToRecord(org.apache.avro.generic.GenericRecord record,
                                                                         String recordKey,
                                                                         String partitionPath,
                                                                         String fileName)

addOperationToRecord

public static org.apache.avro.generic.GenericRecord addOperationToRecord(org.apache.avro.generic.GenericRecord record,
                                                                         HoodieOperation operation)

addCommitMetadataToRecord

public static org.apache.avro.generic.GenericRecord addCommitMetadataToRecord(org.apache.avro.generic.GenericRecord record,
                                                                              String instantTime,
                                                                              String commitSeqno)

Adds the Hoodie commit metadata into the provided Generic Record.

stitchRecords

public static org.apache.avro.generic.GenericRecord stitchRecords(org.apache.avro.generic.GenericRecord left,
                                                                  org.apache.avro.generic.GenericRecord right,
                                                                  org.apache.avro.Schema stitchedSchema)

rewriteRecord

public static org.apache.avro.generic.GenericRecord rewriteRecord(org.apache.avro.generic.GenericRecord oldRecord,
                                                                  org.apache.avro.Schema newSchema)

Given an Avro record with a given schema, rewrites it into the new schema while setting fields only from the new schema. NOTE: This method is rewriting every record's field that is record itself recursively. It's caller's responsibility to make sure that no unnecessary re-writing occurs (by preemptively checking whether the record does require re-writing to adhere to the new schema) NOTE: Here, the assumption is that you cannot go from an evolved schema (schema with (N) fields) to an older schema (schema with (N-1) fields). All fields present in the older record schema MUST be present in the new schema and the default/existing values are carried over. This particular method does the following:

1. Create a new empty GenericRecord with the new schema.
2. For GenericRecord, copy over the data from the old schema to the new schema or set default values for all fields of this transformed schema
3. For SpecificRecord, hoodie_metadata_fields have a special treatment (see below)

For SpecificRecord we ignore Hudi Metadata fields, because for code generated avro classes (HoodieMetadataRecord), the avro record is a SpecificBaseRecord type instead of a GenericRecord. SpecificBaseRecord throws null pointer exception for record.get(name) if name is not present in the schema of the record (which happens when converting a SpecificBaseRecord without hoodie_metadata_fields to a new record with it). In this case, we do NOT set the defaults for the hoodie_metadata_fields explicitly, instead, the new record assumes the default defined in the avro schema itself. TODO: See if we can always pass GenericRecord instead of SpecificBaseRecord in some cases.

rewriteRecordWithMetadata

public static org.apache.avro.generic.GenericRecord rewriteRecordWithMetadata(org.apache.avro.generic.GenericRecord genericRecord,
                                                                              org.apache.avro.Schema newSchema,
                                                                              String fileName)

rewriteEvolutionRecordWithMetadata

public static org.apache.avro.generic.GenericRecord rewriteEvolutionRecordWithMetadata(org.apache.avro.generic.GenericRecord genericRecord,
                                                                                       org.apache.avro.Schema newSchema,
                                                                                       String fileName)

rewriteRecords

public static List<org.apache.avro.generic.GenericRecord> rewriteRecords(List<org.apache.avro.generic.GenericRecord> records,
                                                                         org.apache.avro.Schema newSchema)

Converts list of GenericRecord provided into the GenericRecord adhering to the provided newSchema.

To better understand conversion rules please check rewriteRecord(GenericRecord, Schema)

removeFields

public static org.apache.avro.generic.GenericRecord removeFields(org.apache.avro.generic.GenericRecord record,
                                                                 Set<String> fieldsToRemove)

Given an Avro record and list of columns to remove, this method removes the list of columns from the given avro record using rewriteRecord method.

To better understand how it removes please check rewriteRecord(GenericRecord, Schema)

generateProjectionSchema

public static org.apache.avro.Schema generateProjectionSchema(org.apache.avro.Schema originalSchema,
                                                              List<String> fieldNames)

Generate a reader schema off the provided writeSchema, to just project out the provided columns.

getRootLevelFieldName

public static String getRootLevelFieldName(String fieldName)

Obtain the root-level field name of a full field name, possibly a nested field. For example, given "a.b.c", the output is "a"; given "a", the output is "a".

Parameters:

fieldName - The field name.

Returns:

Root-level field name

getFieldVal

public static Object getFieldVal(org.apache.avro.generic.GenericRecord record,
                                 String key)

Obtain value of the provided key, which is consistent with avro before 1.10

getFieldVal

public static Object getFieldVal(org.apache.avro.generic.GenericRecord record,
                                 String key,
                                 boolean returnNullIfNotFound)

Obtain value of the provided key, when set returnNullIfNotFound false, it is consistent with avro after 1.10

getNestedFieldValAsString

public static String getNestedFieldValAsString(org.apache.avro.generic.GenericRecord record,
                                               String fieldName,
                                               boolean returnNullIfNotFound,
                                               boolean consistentLogicalTimestampEnabled)

Obtain value of the provided field as string, denoted by dot notation. e.g: a.b.c

getNestedFieldVal

public static Object getNestedFieldVal(org.apache.avro.generic.GenericRecord record,
                                       String fieldName,
                                       boolean returnNullIfNotFound,
                                       boolean consistentLogicalTimestampEnabled)

Obtain value of the provided field, denoted by dot notation. e.g: a.b.c

getNestedFieldSchemaFromRecord

public static org.apache.avro.Schema getNestedFieldSchemaFromRecord(org.apache.avro.generic.GenericRecord record,
                                                                    String fieldName)

Get schema for the given field and record. Field can be nested, denoted by dot notation. e.g: a.b.c

Parameters:

record - - record containing the value of the given field

fieldName - - name of the field

Returns:

getNestedFieldSchemaFromWriteSchema

public static org.apache.avro.Schema getNestedFieldSchemaFromWriteSchema(org.apache.avro.Schema writeSchema,
                                                                         String fieldName)

Get schema for the given field and write schema. Field can be nested, denoted by dot notation. e.g: a.b.c Use this method when record is not available. Otherwise, prefer to use getNestedFieldSchemaFromRecord(GenericRecord, String)

Parameters:

writeSchema - - write schema of the record

fieldName - - name of the field

Returns:

getNullableValAsString

public static Option<String> getNullableValAsString(org.apache.avro.generic.GenericRecord rec,
                                                    String fieldName)

Returns the string value of the given record rec and field fieldName. The field and value both could be missing.

Parameters:

rec - The record

fieldName - The field name

Returns:

the string form of the field or empty if the schema does not contain the field name or the value is null

convertValueForSpecificDataTypes

public static Object convertValueForSpecificDataTypes(org.apache.avro.Schema fieldSchema,
                                                      Object fieldValue,
                                                      boolean consistentLogicalTimestampEnabled)

This method converts values for fields with certain Avro/Parquet data types that require special handling.

Parameters:

fieldSchema - avro field schema

fieldValue - avro field value

Returns:

field value either converted (for certain data types) or as it is.

getNullSchema

public static org.apache.avro.Schema getNullSchema()

sanitizeName

public static String sanitizeName(String name)

Sanitizes Name according to Avro rule for names. Removes characters other than the ones mentioned in https://avro.apache.org/docs/current/spec.html#names .

Parameters:

name - input name

Returns:

sanitized name

getRecordColumnValues

public static Object getRecordColumnValues(HoodieRecord<? extends HoodieRecordPayload> record,
                                           String[] columns,
                                           org.apache.avro.Schema schema,
                                           boolean consistentLogicalTimestampEnabled)

Gets record column values into one object.

Parameters:

record - Hoodie record.

columns - Names of the columns to get values.

schema - Schema instance.

Returns:

Column value if a single column, or concatenated String values by comma.

getRecordColumnValues

public static Object getRecordColumnValues(HoodieRecord<? extends HoodieRecordPayload> record,
                                           String[] columns,
                                           SerializableSchema schema,
                                           boolean consistentLogicalTimestampEnabled)

Gets record column values into one object.

Parameters:

record - Hoodie record.

columns - Names of the columns to get values.

schema - SerializableSchema instance.

Returns:

Column value if a single column, or concatenated String values by comma.

rewriteRecordWithNewSchema

public static org.apache.avro.generic.GenericRecord rewriteRecordWithNewSchema(org.apache.avro.generic.IndexedRecord oldRecord,
                                                                               org.apache.avro.Schema newSchema,
                                                                               Map<String,String> renameCols)

Given a avro record with a given schema, rewrites it into the new schema while setting fields only from the new schema. support deep rewrite for nested record. This particular method does the following things : a) Create a new empty GenericRecord with the new schema. b) For GenericRecord, copy over the data from the old schema to the new schema or set default values for all fields of this transformed schema

Parameters:

oldRecord - oldRecord to be rewritten

newSchema - newSchema used to rewrite oldRecord

renameCols - a map store all rename cols, (k, v)-> (colNameFromNewSchema, colNameFromOldSchema)

Returns:

newRecord for new Schema

toJavaDate

public static Date toJavaDate(int days)

convert days to Date NOTE: This method could only be used in tests

fromJavaDate

public static int fromJavaDate(Date date)

convert Date to days NOTE: This method could only be used in tests

rewriteRecordWithNewSchema

public static Iterator<org.apache.avro.generic.GenericRecord> rewriteRecordWithNewSchema(Iterator<org.apache.avro.generic.GenericRecord> oldRecords,
                                                                                         org.apache.avro.Schema newSchema,
                                                                                         Map<String,String> renameCols)

Given avro records, rewrites them with new schema.

Parameters:

oldRecords - oldRecords to be rewrite

newSchema - newSchema used to rewrite oldRecord

renameCols - a map store all rename cols, (k, v)-> (colNameFromNewSchema, colNameFromOldSchema)

Returns:

a iterator of rewrote GeneriRcords

rewriteRecordDeep

public static org.apache.avro.generic.GenericRecord rewriteRecordDeep(org.apache.avro.generic.GenericRecord oldRecord,
                                                                      org.apache.avro.Schema newSchema)

gteqAvro1_9

public static boolean gteqAvro1_9()

gteqAvro1_10

public static boolean gteqAvro1_10()