org.apache.beam.sdk.extensions.avro.schemas.utils

Class AvroUtils

java.lang.Object

org.apache.beam.sdk.extensions.avro.schemas.utils.AvroUtils

public class AvroUtils
extends java.lang.Object

Utils to convert AVRO records to Beam rows. Imposes a mapping between common avro types and Beam portable schemas (https://s.apache.org/beam-schemas):

   Avro                Beam Field Type
   INT         <-----> INT32
   LONG        <-----> INT64
   FLOAT       <-----> FLOAT
   DOUBLE      <-----> DOUBLE
   BOOLEAN     <-----> BOOLEAN
   STRING      <-----> STRING
   BYTES       <-----> BYTES
               <------ LogicalType(urn="beam:logical_type:var_bytes:v1")
   FIXED       <-----> LogicalType(urn="beam:logical_type:fixed_bytes:v1")
   ARRAY       <-----> ARRAY
   ENUM        <-----> LogicalType(EnumerationType)
   MAP         <-----> MAP
   RECORD      <-----> ROW
   UNION       <-----> LogicalType(OneOfType)
   LogicalTypes.Date              <-----> LogicalType(DATE)
                                  <------ LogicalType(urn="beam:logical_type:date:v1")
   LogicalTypes.TimestampMillis   <-----> DATETIME
   LogicalTypes.Decimal           <-----> DECIMAL
 

For SQL CHAR/VARCHAR types, an Avro schema

   LogicalType({"type":"string","logicalType":"char","maxLength":MAX_LENGTH}) or
   LogicalType({"type":"string","logicalType":"varchar","maxLength":MAX_LENGTH})
 

is used.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	AvroUtils.AvroConvertType
static class	AvroUtils.AvroConvertValueForGetter
static class	AvroUtils.AvroConvertValueForSetter
static class	AvroUtils.FixedBytesField Wrapper for fixed byte fields.
static class	AvroUtils.TypeWithNullability

Method Summary

Modifier and Type	Method and Description
static void	addLogicalTypeConversions(org.apache.avro.generic.GenericData data)
static @Nullable java.lang.Object	convertAvroFieldStrict(@Nullable java.lang.Object value, org.apache.avro.Schema avroSchema, org.apache.beam.sdk.schemas.Schema.FieldType fieldType) Strict conversion from AVRO to Beam, strict because it doesn't do widening or narrowing during conversion.
static org.apache.beam.sdk.transforms.SimpleFunction<byte[],org.apache.beam.sdk.values.Row>	getAvroBytesToRowFunction(org.apache.beam.sdk.schemas.Schema beamSchema) Returns a function mapping encoded AVRO GenericRecords to Beam Rows.
static <T> org.apache.beam.sdk.schemas.SchemaUserTypeCreator	getCreator(java.lang.Class<T> clazz, org.apache.beam.sdk.schemas.Schema schema) Get an object creator for an AVRO-generated SpecificRecord.
static <T> java.util.List<org.apache.beam.sdk.schemas.FieldValueTypeInformation>	getFieldTypes(java.lang.Class<T> clazz, org.apache.beam.sdk.schemas.Schema schema) Get field types for an AVRO-generated SpecificRecord or a POJO.
static <T> org.apache.beam.sdk.transforms.SerializableFunction<org.apache.beam.sdk.values.Row,T>	getFromRowFunction(java.lang.Class<T> clazz)
static org.apache.beam.sdk.transforms.SerializableFunction<org.apache.avro.generic.GenericRecord,org.apache.beam.sdk.values.Row>	getGenericRecordToRowFunction(@Nullable org.apache.beam.sdk.schemas.Schema schema) Returns a function mapping AVRO GenericRecords to Beam Rows for use in PCollection.setSchema(org.apache.beam.sdk.schemas.Schema, org.apache.beam.sdk.values.TypeDescriptor<T>, org.apache.beam.sdk.transforms.SerializableFunction<T, org.apache.beam.sdk.values.Row>, org.apache.beam.sdk.transforms.SerializableFunction<org.apache.beam.sdk.values.Row, T>).
static <T> java.util.List<org.apache.beam.sdk.schemas.FieldValueGetter>	getGetters(java.lang.Class<T> clazz, org.apache.beam.sdk.schemas.Schema schema) Get generated getters for an AVRO-generated SpecificRecord or a POJO.
static org.apache.beam.sdk.transforms.SimpleFunction<org.apache.beam.sdk.values.Row,byte[]>	getRowToAvroBytesFunction(org.apache.beam.sdk.schemas.Schema beamSchema) Returns a function mapping Beam Rows to encoded AVRO GenericRecords.
static org.apache.beam.sdk.transforms.SerializableFunction<org.apache.beam.sdk.values.Row,org.apache.avro.generic.GenericRecord>	getRowToGenericRecordFunction(org.apache.avro.Schema avroSchema) Returns a function mapping Beam Rows to AVRO GenericRecords for use in PCollection.setSchema(org.apache.beam.sdk.schemas.Schema, org.apache.beam.sdk.values.TypeDescriptor<T>, org.apache.beam.sdk.transforms.SerializableFunction<T, org.apache.beam.sdk.values.Row>, org.apache.beam.sdk.transforms.SerializableFunction<org.apache.beam.sdk.values.Row, T>).
static <T> @Nullable org.apache.beam.sdk.schemas.Schema	getSchema(java.lang.Class<T> clazz, org.apache.avro.Schema schema)
static <T> org.apache.beam.sdk.transforms.SerializableFunction<T,org.apache.beam.sdk.values.Row>	getToRowFunction(java.lang.Class<T> clazz, org.apache.avro.Schema schema)
static <T> org.apache.beam.sdk.schemas.SchemaCoder<T>	schemaCoder(AvroCoder<T> avroCoder) Returns an SchemaCoder instance based on the provided AvroCoder for the element type.
static <T> org.apache.beam.sdk.schemas.SchemaCoder<T>	schemaCoder(java.lang.Class<T> clazz) Returns an SchemaCoder instance for the provided element class.
static <T> org.apache.beam.sdk.schemas.SchemaCoder<T>	schemaCoder(java.lang.Class<T> clazz, org.apache.avro.Schema schema) Returns an SchemaCoder instance for the provided element type using the provided Avro schema.
static org.apache.beam.sdk.schemas.SchemaCoder<org.apache.avro.generic.GenericRecord>	schemaCoder(org.apache.avro.Schema schema) Returns an SchemaCoder instance for the Avro schema.
static <T> org.apache.beam.sdk.schemas.SchemaCoder<T>	schemaCoder(org.apache.beam.sdk.values.TypeDescriptor<T> type) Returns an SchemaCoder instance for the provided element type.
static org.apache.avro.Schema.Field	toAvroField(org.apache.beam.sdk.schemas.Schema.Field field, java.lang.String namespace) Get Avro Field from Beam Field.
static org.apache.avro.Schema	toAvroSchema(org.apache.beam.sdk.schemas.Schema beamSchema)
static org.apache.avro.Schema	toAvroSchema(org.apache.beam.sdk.schemas.Schema beamSchema, @Nullable java.lang.String name, @Nullable java.lang.String namespace) Converts a Beam Schema into an AVRO schema.
static org.apache.beam.sdk.schemas.Schema.Field	toBeamField(org.apache.avro.Schema.Field field) Get Beam Field from avro Field.
static org.apache.beam.sdk.values.Row	toBeamRowStrict(org.apache.avro.generic.GenericRecord record, @Nullable org.apache.beam.sdk.schemas.Schema schema) Strict conversion from AVRO to Beam, strict because it doesn't do widening or narrowing during conversion.
static org.apache.beam.sdk.schemas.Schema	toBeamSchema(java.lang.Class<?> clazz) Converts AVRO schema to Beam row schema.
static org.apache.beam.sdk.schemas.Schema	toBeamSchema(org.apache.avro.Schema schema) Converts AVRO schema to Beam row schema.
static org.apache.avro.generic.GenericRecord	toGenericRecord(org.apache.beam.sdk.values.Row row) Convert from a Beam Row to an AVRO GenericRecord.
static org.apache.avro.generic.GenericRecord	toGenericRecord(org.apache.beam.sdk.values.Row row, org.apache.avro.Schema avroSchema) Convert from a Beam Row to an AVRO GenericRecord.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

addLogicalTypeConversions

public static void addLogicalTypeConversions(org.apache.avro.generic.GenericData data)

toBeamField

public static org.apache.beam.sdk.schemas.Schema.Field toBeamField(org.apache.avro.Schema.Field field)

Get Beam Field from avro Field.

toAvroField

public static org.apache.avro.Schema.Field toAvroField(org.apache.beam.sdk.schemas.Schema.Field field,
                                                       java.lang.String namespace)

Get Avro Field from Beam Field.

toBeamSchema

public static org.apache.beam.sdk.schemas.Schema toBeamSchema(java.lang.Class<?> clazz)

Converts AVRO schema to Beam row schema.

Parameters:

clazz - avro class

toBeamSchema

public static org.apache.beam.sdk.schemas.Schema toBeamSchema(org.apache.avro.Schema schema)

Converts AVRO schema to Beam row schema.

Parameters:

schema - schema of type RECORD

toAvroSchema

public static org.apache.avro.Schema toAvroSchema(org.apache.beam.sdk.schemas.Schema beamSchema,
                                                  @Nullable java.lang.String name,
                                                  @Nullable java.lang.String namespace)

Converts a Beam Schema into an AVRO schema.

toAvroSchema

public static org.apache.avro.Schema toAvroSchema(org.apache.beam.sdk.schemas.Schema beamSchema)

toBeamRowStrict

public static org.apache.beam.sdk.values.Row toBeamRowStrict(org.apache.avro.generic.GenericRecord record,
                                                             @Nullable org.apache.beam.sdk.schemas.Schema schema)

Strict conversion from AVRO to Beam, strict because it doesn't do widening or narrowing during conversion. If Schema is not provided, one is inferred from the AVRO schema.

toGenericRecord

public static org.apache.avro.generic.GenericRecord toGenericRecord(org.apache.beam.sdk.values.Row row)

Convert from a Beam Row to an AVRO GenericRecord. The Avro Schema is inferred from the Beam schema on the row.

toGenericRecord

public static org.apache.avro.generic.GenericRecord toGenericRecord(org.apache.beam.sdk.values.Row row,
                                                                    org.apache.avro.Schema avroSchema)

Convert from a Beam Row to an AVRO GenericRecord. If a Schema is not provided, one is inferred from the Beam schema on the row.

getToRowFunction

public static <T> org.apache.beam.sdk.transforms.SerializableFunction<T,org.apache.beam.sdk.values.Row> getToRowFunction(java.lang.Class<T> clazz,
                                                                                                                         org.apache.avro.Schema schema)

getFromRowFunction

public static <T> org.apache.beam.sdk.transforms.SerializableFunction<org.apache.beam.sdk.values.Row,T> getFromRowFunction(java.lang.Class<T> clazz)

getSchema

public static <T> @Nullable org.apache.beam.sdk.schemas.Schema getSchema(java.lang.Class<T> clazz,
                                                                         org.apache.avro.Schema schema)

getAvroBytesToRowFunction

public static org.apache.beam.sdk.transforms.SimpleFunction<byte[],org.apache.beam.sdk.values.Row> getAvroBytesToRowFunction(org.apache.beam.sdk.schemas.Schema beamSchema)

Returns a function mapping encoded AVRO GenericRecords to Beam Rows.

getRowToAvroBytesFunction

public static org.apache.beam.sdk.transforms.SimpleFunction<org.apache.beam.sdk.values.Row,byte[]> getRowToAvroBytesFunction(org.apache.beam.sdk.schemas.Schema beamSchema)

Returns a function mapping Beam Rows to encoded AVRO GenericRecords.

getGenericRecordToRowFunction

public static org.apache.beam.sdk.transforms.SerializableFunction<org.apache.avro.generic.GenericRecord,org.apache.beam.sdk.values.Row> getGenericRecordToRowFunction(@Nullable org.apache.beam.sdk.schemas.Schema schema)

Returns a function mapping AVRO GenericRecords to Beam Rows for use in PCollection.setSchema(org.apache.beam.sdk.schemas.Schema, org.apache.beam.sdk.values.TypeDescriptor<T>, org.apache.beam.sdk.transforms.SerializableFunction<T, org.apache.beam.sdk.values.Row>, org.apache.beam.sdk.transforms.SerializableFunction<org.apache.beam.sdk.values.Row, T>).

getRowToGenericRecordFunction

public static org.apache.beam.sdk.transforms.SerializableFunction<org.apache.beam.sdk.values.Row,org.apache.avro.generic.GenericRecord> getRowToGenericRecordFunction(org.apache.avro.Schema avroSchema)

Returns a function mapping Beam Rows to AVRO GenericRecords for use in PCollection.setSchema(org.apache.beam.sdk.schemas.Schema, org.apache.beam.sdk.values.TypeDescriptor<T>, org.apache.beam.sdk.transforms.SerializableFunction<T, org.apache.beam.sdk.values.Row>, org.apache.beam.sdk.transforms.SerializableFunction<org.apache.beam.sdk.values.Row, T>).

schemaCoder

public static <T> org.apache.beam.sdk.schemas.SchemaCoder<T> schemaCoder(org.apache.beam.sdk.values.TypeDescriptor<T> type)

Returns an SchemaCoder instance for the provided element type.

Type Parameters:

T - the element type

schemaCoder

public static <T> org.apache.beam.sdk.schemas.SchemaCoder<T> schemaCoder(java.lang.Class<T> clazz)

Returns an SchemaCoder instance for the provided element class.

Type Parameters:

T - the element type

schemaCoder

public static org.apache.beam.sdk.schemas.SchemaCoder<org.apache.avro.generic.GenericRecord> schemaCoder(org.apache.avro.Schema schema)

Returns an SchemaCoder instance for the Avro schema. The implicit type is GenericRecord.

schemaCoder

public static <T> org.apache.beam.sdk.schemas.SchemaCoder<T> schemaCoder(java.lang.Class<T> clazz,
                                                                         org.apache.avro.Schema schema)

Returns an SchemaCoder instance for the provided element type using the provided Avro schema.

If the type argument is GenericRecord, the schema may be arbitrary. Otherwise, the schema must correspond to the type provided.

Type Parameters:

T - the element type

schemaCoder

public static <T> org.apache.beam.sdk.schemas.SchemaCoder<T> schemaCoder(AvroCoder<T> avroCoder)

Returns an SchemaCoder instance based on the provided AvroCoder for the element type.

Type Parameters:

T - the element type

getFieldTypes

public static <T> java.util.List<org.apache.beam.sdk.schemas.FieldValueTypeInformation> getFieldTypes(java.lang.Class<T> clazz,
                                                                                                      org.apache.beam.sdk.schemas.Schema schema)

Get field types for an AVRO-generated SpecificRecord or a POJO.

getGetters

public static <T> java.util.List<org.apache.beam.sdk.schemas.FieldValueGetter> getGetters(java.lang.Class<T> clazz,
                                                                                          org.apache.beam.sdk.schemas.Schema schema)

Get generated getters for an AVRO-generated SpecificRecord or a POJO.

getCreator

public static <T> org.apache.beam.sdk.schemas.SchemaUserTypeCreator getCreator(java.lang.Class<T> clazz,
                                                                               org.apache.beam.sdk.schemas.Schema schema)

Get an object creator for an AVRO-generated SpecificRecord.

convertAvroFieldStrict

public static @Nullable java.lang.Object convertAvroFieldStrict(@Nullable java.lang.Object value,
                                                                @Nonnull
                                                                org.apache.avro.Schema avroSchema,
                                                                @Nonnull
                                                                org.apache.beam.sdk.schemas.Schema.FieldType fieldType)

Strict conversion from AVRO to Beam, strict because it doesn't do widening or narrowing during conversion.

Parameters:

value - GenericRecord or any nested value

avroSchema - schema for value

fieldType - target beam field type

Returns:

value converted for Row