org.apache.hudi.internal.schema.utils

Class AvroSchemaEvolutionUtils

java.lang.Object

org.apache.hudi.internal.schema.utils.AvroSchemaEvolutionUtils

public class AvroSchemaEvolutionUtils
extends Object

Utility methods to support evolve old avro schema based on a given schema.

Constructor Summary

Constructors

Constructor and Description
AvroSchemaEvolutionUtils()

Method Summary

Modifier and Type	Method and Description
static org.apache.avro.Schema	canonicalizeColumnNullability(org.apache.avro.Schema writeSchema, org.apache.avro.Schema readSchema) Canonical the nullability.
static InternalSchema	reconcileSchema(org.apache.avro.Schema incomingSchema, InternalSchema oldTableSchema) Support reconcile from a new avroSchema.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AvroSchemaEvolutionUtils

public AvroSchemaEvolutionUtils()

Method Detail

reconcileSchema

public static InternalSchema reconcileSchema(org.apache.avro.Schema incomingSchema,
                                             InternalSchema oldTableSchema)

Support reconcile from a new avroSchema. 1) incoming data has missing columns that were already defined in the table –> null values will be injected into missing columns 2) incoming data contains new columns not defined yet in the table -> columns will be added to the table schema (incoming dataframe?) 3) incoming data has missing columns that are already defined in the table and new columns not yet defined in the table -> new columns will be added to the table schema, missing columns will be injected with null values 4) support nested schema change. Notice: the incoming schema should not have delete/rename semantics. for example: incoming schema: int a, int b, int d; oldTableSchema int a, int b, int c, int d we must guarantee the column c is missing semantic, instead of delete semantic.

Parameters:

incomingSchema - implicitly evolution of avro when hoodie write operation

oldTableSchema - old internalSchema

Returns:

reconcile Schema

canonicalizeColumnNullability

public static org.apache.avro.Schema canonicalizeColumnNullability(org.apache.avro.Schema writeSchema,
                                                                   org.apache.avro.Schema readSchema)

Canonical the nullability. Do not allow change cols Nullability field from optional to required. If above problem occurs, try to correct it.

Parameters:

writeSchema - writeSchema hoodie used to write data.

readSchema - read schema

Returns:

canonical Schema