org.apache.hudi.common.util

Class AvroOrcUtils

java.lang.Object

org.apache.hudi.common.util.AvroOrcUtils

public class AvroOrcUtils
extends Object

Methods including addToVector, addUnionValue, createOrcSchema are originally from https://github.com/streamsets/datacollector. Source classes: - com.streamsets.pipeline.lib.util.avroorc.AvroToOrcRecordConverter - com.streamsets.pipeline.lib.util.avroorc.AvroToOrcSchemaConverter Changes made: 1. Flatten nullable Avro schema type when the value is not null in `addToVector`. 2. Use getLogicalType(), constants from LogicalTypes instead of getJsonProp() to handle Avro logical types.

Constructor Summary

Constructors

Constructor and Description
AvroOrcUtils()

Method Summary

Modifier and Type	Method and Description
static void	addToVector(org.apache.orc.TypeDescription type, org.apache.orc.storage.ql.exec.vector.ColumnVector colVector, org.apache.avro.Schema avroSchema, Object value, int vectorPos) Add an object (of a given ORC type) to the column vector at a given position.
static boolean	addUnionValue(org.apache.orc.storage.ql.exec.vector.UnionColumnVector unionVector, List<org.apache.orc.TypeDescription> unionChildTypes, org.apache.avro.Schema avroSchema, Object value, int vectorPos) Match value with its ORC type and add to the union vector at a given position.
static org.apache.avro.Schema	createAvroSchema(org.apache.orc.TypeDescription orcSchema)
static org.apache.avro.Schema	createAvroSchemaWithDefaultValue(org.apache.orc.TypeDescription orcSchema, String recordName, String namespace, boolean nullable)
static org.apache.orc.TypeDescription	createOrcSchema(org.apache.avro.Schema avroSchema)
static Object	readFromVector(org.apache.orc.TypeDescription type, org.apache.orc.storage.ql.exec.vector.ColumnVector colVector, org.apache.avro.Schema avroSchema, int vectorPos) Read the Column vector at a given position conforming to a given ORC schema.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AvroOrcUtils

public AvroOrcUtils()

Method Detail

addToVector

public static void addToVector(org.apache.orc.TypeDescription type,
                               org.apache.orc.storage.ql.exec.vector.ColumnVector colVector,
                               org.apache.avro.Schema avroSchema,
                               Object value,
                               int vectorPos)

Add an object (of a given ORC type) to the column vector at a given position.

Parameters:

type - ORC schema of the value Object.

colVector - The column vector to store the value Object.

avroSchema - Avro schema of the value Object. Only used to check logical types for timestamp unit conversion.

value - Object to be added to the column vector

vectorPos - The position in the vector where value will be stored at.

addUnionValue

public static boolean addUnionValue(org.apache.orc.storage.ql.exec.vector.UnionColumnVector unionVector,
                                    List<org.apache.orc.TypeDescription> unionChildTypes,
                                    org.apache.avro.Schema avroSchema,
                                    Object value,
                                    int vectorPos)

Match value with its ORC type and add to the union vector at a given position.

Parameters:

unionVector - The vector to store value.

unionChildTypes - All possible types for the value Object.

avroSchema - Avro union schema for the value Object.

value - Object to be added to the unionVector

vectorPos - The position in the vector where value will be stored at.

Returns:

succeeded or failed

readFromVector

public static Object readFromVector(org.apache.orc.TypeDescription type,
                                    org.apache.orc.storage.ql.exec.vector.ColumnVector colVector,
                                    org.apache.avro.Schema avroSchema,
                                    int vectorPos)

Read the Column vector at a given position conforming to a given ORC schema.

Parameters:

type - ORC schema of the object to read.

colVector - The column vector to read.

avroSchema - Avro schema of the object to read. Only used to check logical types for timestamp unit conversion.

vectorPos - The position in the vector where the value to read is stored at.

Returns:

The object being read.

createOrcSchema

public static org.apache.orc.TypeDescription createOrcSchema(org.apache.avro.Schema avroSchema)

createAvroSchema

public static org.apache.avro.Schema createAvroSchema(org.apache.orc.TypeDescription orcSchema)

createAvroSchemaWithDefaultValue

public static org.apache.avro.Schema createAvroSchemaWithDefaultValue(org.apache.orc.TypeDescription orcSchema,
                                                                      String recordName,
                                                                      String namespace,
                                                                      boolean nullable)