org.apache.hudi.internal.schema.convert

Class AvroInternalSchemaConverter

java.lang.Object

org.apache.hudi.internal.schema.convert.AvroInternalSchemaConverter

public class AvroInternalSchemaConverter
extends Object

Auxiliary class. Converts an avro schema into InternalSchema, or convert InternalSchema to an avro schema

Constructor Summary

Constructors

Constructor and Description
AvroInternalSchemaConverter()

Method Summary

Modifier and Type	Method and Description
static org.apache.avro.Schema	buildAvroSchemaFromInternalSchema(InternalSchema schema, String recordName) Converts hudi internal Schema into an Avro Schema.
static org.apache.avro.Schema	buildAvroSchemaFromType(Type type, String recordName) Converts hudi type into an Avro Schema.
static Type	buildTypeFromAvroSchema(org.apache.avro.Schema schema) Build hudi type from avro schema.
static org.apache.avro.Schema	convert(InternalSchema internalSchema, String name) Convert internalSchema to avro Schema.
static InternalSchema	convert(org.apache.avro.Schema schema) Convert an avro schema into internalSchema.
static org.apache.avro.Schema	convert(Types.RecordType type, String name) Convert RecordType to avro Schema.
static org.apache.avro.Schema	convert(Type type, String name) Convert internal type to avro Schema.
static Type	convertToField(org.apache.avro.Schema schema) Convert an avro schema into internal type.
static org.apache.avro.Schema	fixNullOrdering(org.apache.avro.Schema schema) Converting from avro -> internal schema -> avro causes null to always be first in unions.
static boolean	isOptional(org.apache.avro.Schema schema) Check whether current avro schema is optional?.
static org.apache.avro.Schema	nullableSchema(org.apache.avro.Schema schema) Returns schema with nullable true.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AvroInternalSchemaConverter

public AvroInternalSchemaConverter()

Method Detail

convert

public static org.apache.avro.Schema convert(InternalSchema internalSchema,
                                             String name)

Convert internalSchema to avro Schema.

Parameters:

internalSchema - internal schema.

name - the record name.

Returns:

an avro Schema.

fixNullOrdering

public static org.apache.avro.Schema fixNullOrdering(org.apache.avro.Schema schema)

Converting from avro -> internal schema -> avro causes null to always be first in unions. if we compare a schema that has not been converted to internal schema at any stage, the difference in ordering can cause issues. To resolve this, we order null to be first for any avro schema that enters into hudi. AvroSchemaUtils.isProjectionOfInternal uses index based comparison for unions. Spark and flink don't support complex unions so this would not be an issue but for the metadata table HoodieMetadata.avsc uses a trick where we have a bunch of different types wrapped in record for col stats.

Parameters:

schema - avro schema.

Returns:

an avro Schema where null is the first.

convert

public static org.apache.avro.Schema convert(Types.RecordType type,
                                             String name)

Convert RecordType to avro Schema.

Parameters:

type - internal schema.

name - the record name.

Returns:

an avro Schema.

convert

public static org.apache.avro.Schema convert(Type type,
                                             String name)

Convert internal type to avro Schema.

Parameters:

type - internal type.

name - the record name.

Returns:

an avro Schema.

convertToField

public static Type convertToField(org.apache.avro.Schema schema)

Convert an avro schema into internal type.

convert

public static InternalSchema convert(org.apache.avro.Schema schema)

Convert an avro schema into internalSchema.

isOptional

public static boolean isOptional(org.apache.avro.Schema schema)

Check whether current avro schema is optional?.

nullableSchema

public static org.apache.avro.Schema nullableSchema(org.apache.avro.Schema schema)

Returns schema with nullable true.

buildTypeFromAvroSchema

public static Type buildTypeFromAvroSchema(org.apache.avro.Schema schema)

Build hudi type from avro schema.

Parameters:

schema - a avro schema.

Returns:

a hudi type.

buildAvroSchemaFromType

public static org.apache.avro.Schema buildAvroSchemaFromType(Type type,
                                                             String recordName)

Converts hudi type into an Avro Schema.

Parameters:

type - a hudi type.

recordName - the record name

Returns:

a Avro schema match this type

buildAvroSchemaFromInternalSchema

public static org.apache.avro.Schema buildAvroSchemaFromInternalSchema(InternalSchema schema,
                                                                       String recordName)

Converts hudi internal Schema into an Avro Schema.

Parameters:

schema - a hudi internal Schema.

recordName - the record name

Returns:

a Avro schema match hudi internal schema.