org.apache.hudi.avro

Class AvroSchemaUtils

java.lang.Object

org.apache.hudi.avro.AvroSchemaUtils

public class AvroSchemaUtils
extends Object

Utils for Avro Schema.

Method Summary

Modifier and Type	Method and Description
static org.apache.avro.Schema	appendFieldsToSchema(org.apache.avro.Schema schema, List<org.apache.avro.Schema.Field> newFields) Appends provided new fields at the end of the given schema NOTE: No deduplication is made, this method simply appends fields at the end of the list of the source schema as is
static boolean	canProject(org.apache.avro.Schema prevSchema, org.apache.avro.Schema newSchema) Check that each field in the prevSchema can be populated in the newSchema
static void	checkSchemaCompatible(org.apache.avro.Schema tableSchema, org.apache.avro.Schema writerSchema, boolean shouldValidate, boolean allowProjection, Set<String> dropPartitionColNames) Checks whether writer schema is compatible with table schema considering AVRO_SCHEMA_VALIDATE_ENABLE and SCHEMA_ALLOW_AUTO_EVOLUTION_COLUMN_DROP options.
static void	checkValidEvolution(org.apache.avro.Schema incomingSchema, org.apache.avro.Schema tableSchema) Validate whether the incomingSchema is a valid evolution of tableSchema.
static boolean	containsFieldInSchema(org.apache.avro.Schema schema, String fieldName) Returns true in case when schema contains the field w/ provided name
static org.apache.avro.Schema	createNullableSchema(org.apache.avro.Schema.Type avroType) Creates schema following Avro's typical nullable schema definition: Union(Schema.Type.NULL, <NonNullType>), wrapping around provided target non-null type
static org.apache.avro.Schema	createNullableSchema(org.apache.avro.Schema schema)
static String	createSchemaErrorString(String errorMessage, org.apache.avro.Schema writerSchema, org.apache.avro.Schema tableSchema)
static String	getAvroRecordQualifiedName(String tableName) Generates fully-qualified name for the Avro's schema based on the Table's name NOTE: PLEASE READ CAREFULLY BEFORE CHANGING This method should not change for compatibility reasons as older versions of Avro might be comparing fully-qualified names rather than just the record names
static boolean	isCompatibleProjectionOf(org.apache.avro.Schema sourceSchema, org.apache.avro.Schema targetSchema) Validate whether the targetSchema is a "compatible" projection of sourceSchema.
static boolean	isNullable(org.apache.avro.Schema schema) Returns true in case provided Schema is nullable (ie accepting null values), returns false otherwise
static boolean	isSchemaCompatible(org.apache.avro.Schema prevSchema, org.apache.avro.Schema newSchema) See isSchemaCompatible(Schema, Schema, boolean, boolean) doc for more details
static boolean	isSchemaCompatible(org.apache.avro.Schema prevSchema, org.apache.avro.Schema newSchema, boolean allowProjection) See isSchemaCompatible(Schema, Schema, boolean, boolean) doc for more details
static boolean	isSchemaCompatible(org.apache.avro.Schema prevSchema, org.apache.avro.Schema newSchema, boolean checkNaming, boolean allowProjection) Establishes whether newSchema is compatible w/ prevSchema, as defined by Avro's AvroSchemaCompatibility.
static boolean	isStrictProjectionOf(org.apache.avro.Schema sourceSchema, org.apache.avro.Schema targetSchema) Validate whether the targetSchema is a strict projection of sourceSchema.
static org.apache.avro.Schema	resolveNullableSchema(org.apache.avro.Schema schema) Resolves typical Avro's nullable schema definition: Union(Schema.Type.NULL, <NonNullType>), decomposing union and returning the target non-null type
static org.apache.avro.Schema	resolveUnionSchema(org.apache.avro.Schema schema, String fieldSchemaFullName) Passed in Union schema and will try to resolve the field with the fieldSchemaFullName w/in the union returning its corresponding schema

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

isSchemaCompatible

public static boolean isSchemaCompatible(org.apache.avro.Schema prevSchema,
                                         org.apache.avro.Schema newSchema)

See isSchemaCompatible(Schema, Schema, boolean, boolean) doc for more details

isSchemaCompatible

public static boolean isSchemaCompatible(org.apache.avro.Schema prevSchema,
                                         org.apache.avro.Schema newSchema,
                                         boolean allowProjection)

See isSchemaCompatible(Schema, Schema, boolean, boolean) doc for more details

isSchemaCompatible

public static boolean isSchemaCompatible(org.apache.avro.Schema prevSchema,
                                         org.apache.avro.Schema newSchema,
                                         boolean checkNaming,
                                         boolean allowProjection)

Establishes whether newSchema is compatible w/ prevSchema, as defined by Avro's AvroSchemaCompatibility. From avro's compatibility standpoint, prevSchema is writer schema and new schema is reader schema. newSchema is considered compatible to prevSchema, iff data written using prevSchema could be read by newSchema

Parameters:

prevSchema - previous instance of the schema

newSchema - new instance of the schema

checkNaming - controls whether schemas fully-qualified names should be checked

canProject

public static boolean canProject(org.apache.avro.Schema prevSchema,
                                 org.apache.avro.Schema newSchema)

Check that each field in the prevSchema can be populated in the newSchema

Parameters:

prevSchema - prev schema.

newSchema - new schema

Returns:

true if prev schema is a projection of new schema.

getAvroRecordQualifiedName

public static String getAvroRecordQualifiedName(String tableName)

Generates fully-qualified name for the Avro's schema based on the Table's name NOTE: PLEASE READ CAREFULLY BEFORE CHANGING This method should not change for compatibility reasons as older versions of Avro might be comparing fully-qualified names rather than just the record names

isCompatibleProjectionOf

public static boolean isCompatibleProjectionOf(org.apache.avro.Schema sourceSchema,
                                               org.apache.avro.Schema targetSchema)

Validate whether the targetSchema is a "compatible" projection of sourceSchema. Only difference of this method from isStrictProjectionOf(Schema, Schema) is the fact that it allows some legitimate type promotions (like int -> long, decimal(3, 2) -> decimal(5, 2), etc) that allows projection to have a "wider" atomic type (whereas strict projection requires atomic type to be identical)

isStrictProjectionOf

public static boolean isStrictProjectionOf(org.apache.avro.Schema sourceSchema,
                                           org.apache.avro.Schema targetSchema)

Validate whether the targetSchema is a strict projection of sourceSchema. Schema B is considered a strict projection of schema A iff

1. Schemas A and B are equal, or
2. Schemas A and B are array schemas and element-type of B is a strict projection of the element-type of A, or
3. Schemas A and B are map schemas and value-type of B is a strict projection of the value-type of A, or
4. Schemas A and B are union schemas (of the same size) and every element-type of B is a strict projection of the corresponding element-type of A, or
5. Schemas A and B are record schemas and every field of the record B has corresponding counterpart (w/ the same name) in the schema A, such that the schema of the field of the schema B is also a strict projection of the A field's schema

appendFieldsToSchema

public static org.apache.avro.Schema appendFieldsToSchema(org.apache.avro.Schema schema,
                                                          List<org.apache.avro.Schema.Field> newFields)

Appends provided new fields at the end of the given schema NOTE: No deduplication is made, this method simply appends fields at the end of the list of the source schema as is

resolveUnionSchema

public static org.apache.avro.Schema resolveUnionSchema(org.apache.avro.Schema schema,
                                                        String fieldSchemaFullName)

Passed in Union schema and will try to resolve the field with the fieldSchemaFullName w/in the union returning its corresponding schema

Parameters:

schema - target schema to be inspected

fieldSchemaFullName - target field-name to be looked up w/in the union

Returns:

schema of the field w/in the union identified by the fieldSchemaFullName

isNullable

public static boolean isNullable(org.apache.avro.Schema schema)

Returns true in case provided Schema is nullable (ie accepting null values), returns false otherwise

resolveNullableSchema

public static org.apache.avro.Schema resolveNullableSchema(org.apache.avro.Schema schema)

Resolves typical Avro's nullable schema definition: Union(Schema.Type.NULL, <NonNullType>), decomposing union and returning the target non-null type

createNullableSchema

public static org.apache.avro.Schema createNullableSchema(org.apache.avro.Schema.Type avroType)

Creates schema following Avro's typical nullable schema definition: Union(Schema.Type.NULL, <NonNullType>), wrapping around provided target non-null type

createNullableSchema

public static org.apache.avro.Schema createNullableSchema(org.apache.avro.Schema schema)

containsFieldInSchema

public static boolean containsFieldInSchema(org.apache.avro.Schema schema,
                                            String fieldName)

Returns true in case when schema contains the field w/ provided name

checkSchemaCompatible

public static void checkSchemaCompatible(org.apache.avro.Schema tableSchema,
                                         org.apache.avro.Schema writerSchema,
                                         boolean shouldValidate,
                                         boolean allowProjection,
                                         Set<String> dropPartitionColNames)
                                  throws SchemaCompatibilityException

Checks whether writer schema is compatible with table schema considering AVRO_SCHEMA_VALIDATE_ENABLE and SCHEMA_ALLOW_AUTO_EVOLUTION_COLUMN_DROP options. To avoid collision of SCHEMA_ALLOW_AUTO_EVOLUTION_COLUMN_DROP and DROP_PARTITION_COLUMNS partition column names should be passed as dropPartitionColNames. Passed empty set means DROP_PARTITION_COLUMNS is disabled.

Parameters:

tableSchema - the latest dataset schema

writerSchema - writer schema

shouldValidate - whether AvroSchemaCompatibility check being performed

allowProjection - whether column dropping check being performed

dropPartitionColNames - partition column names to being excluded from column dropping check

Throws:

SchemaCompatibilityException - if writer schema is not compatible

checkValidEvolution

public static void checkValidEvolution(org.apache.avro.Schema incomingSchema,
                                       org.apache.avro.Schema tableSchema)

Validate whether the incomingSchema is a valid evolution of tableSchema.

Parameters:

incomingSchema - schema of the incoming dataset

tableSchema - latest table schema

createSchemaErrorString

public static String createSchemaErrorString(String errorMessage,
                                             org.apache.avro.Schema writerSchema,
                                             org.apache.avro.Schema tableSchema)