org.apache.hadoop.hive.ql.io.arrow

Class ArrowColumnarBatchSerDe

java.lang.Object

org.apache.hadoop.hive.serde2.AbstractSerDe

org.apache.hadoop.hive.ql.io.arrow.ArrowColumnarBatchSerDe

All Implemented Interfaces:

Deserializer, SerDe, Serializer

public class ArrowColumnarBatchSerDe
extends AbstractSerDe

ArrowColumnarBatchSerDe converts Apache Hive rows to Apache Arrow columns. Its serialized class is ArrowWrapperWritable, which doesn't support Writable.readFields(DataInput) and Writable.write(DataOutput). Followings are known issues of current implementation. A list column cannot have a decimal column. UnionListWriter doesn't have an implementation for BaseWriter.ListWriter#decimal(). A union column can have only one of string, char, varchar fields at a same time. Apache Arrow doesn't have string and char, so ArrowColumnarBatchSerDe uses varchar to simulate string and char. They will be considered as a same data type in UnionVector. Timestamp with local timezone is not supported. VectorAssignRow doesn't support it.

Field Summary

Fields inherited from class org.apache.hadoop.hive.serde2.AbstractSerDe

configuration, log, partitionProperties, properties, tableProperties

Constructor Summary

Constructors

Constructor and Description
ArrowColumnarBatchSerDe()

Method Summary

Modifier and Type	Method and Description
Object	deserialize(org.apache.hadoop.io.Writable writable)
ObjectInspector	getObjectInspector()
Class<? extends org.apache.hadoop.io.Writable>	getSerializedClass()
void	initialize(org.apache.hadoop.conf.Configuration configuration, Properties tableProperties, Properties partitionProperties)
ArrowWrapperWritable	serialize(Object obj, ObjectInspector objInspector)

Methods inherited from class org.apache.hadoop.hive.serde2.AbstractSerDe

getColumnComments, getColumnNames, getColumnTypes, getConfiguration, getPartitionColumnComments, getPartitionColumnNames, getPartitionColumnTypes, getSerDeStats, parseColumnComments, parseColumnNames, parseColumnNames, parseColumnTypes, parseColumnTypes, shouldStoreFieldsInMetastore, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.hive.serde2.SerDe

handleJobLevelConfiguration

Constructor Detail

ArrowColumnarBatchSerDe

public ArrowColumnarBatchSerDe()

Method Detail

initialize

public void initialize(org.apache.hadoop.conf.Configuration configuration,
                       Properties tableProperties,
                       Properties partitionProperties)
                throws SerDeException

Overrides:

initialize in class AbstractSerDe

Throws:

SerDeException

getSerializedClass

public Class<? extends org.apache.hadoop.io.Writable> getSerializedClass()

Specified by:

getSerializedClass in interface Serializer

Specified by:

getSerializedClass in class AbstractSerDe

serialize

public ArrowWrapperWritable serialize(Object obj,
                                      ObjectInspector objInspector)

Specified by:

serialize in interface Serializer

Overrides:

serialize in class AbstractSerDe

deserialize

public Object deserialize(org.apache.hadoop.io.Writable writable)

Specified by:

deserialize in interface Deserializer

Overrides:

deserialize in class AbstractSerDe

getObjectInspector

public ObjectInspector getObjectInspector()