org.apache.hadoop.hive.serde2.avro

Class AvroSerdeUtils

java.lang.Object

org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils

public class AvroSerdeUtils
extends java.lang.Object

Utilities useful only to the AvroSerde itself. Not mean to be used by end-users but public for interop to the ql package.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	AvroSerdeUtils.AvroTableProperties Enum container for all avro table properties.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	AVRO_SERDE_SCHEMA Deprecated.
static java.lang.String	EXCEPTION_MESSAGE
static java.lang.String	SCHEMA_DOC Deprecated.
static java.lang.String	SCHEMA_LITERAL Deprecated.
static java.lang.String	SCHEMA_NAME Deprecated.
static java.lang.String	SCHEMA_NAMESPACE Deprecated.
static java.lang.String	SCHEMA_NONE
static java.lang.String	SCHEMA_RETRIEVER Deprecated.
static java.lang.String	SCHEMA_URL Deprecated.

Constructor Summary

Constructors

Constructor and Description
AvroSerdeUtils()

Method Summary

Modifier and Type	Method and Description
static org.apache.avro.Schema	determineSchemaOrThrowException(org.apache.hadoop.conf.Configuration conf, java.util.Properties properties) Determine the schema to that's been provided for Avro serde work.
static java.nio.Buffer	getBufferFromBytes(byte[] input)
static java.nio.Buffer	getBufferFromDecimal(org.apache.hadoop.hive.common.type.HiveDecimal dec, int scale)
static byte[]	getBytesFromByteBuffer(java.nio.ByteBuffer byteBuffer)
static org.apache.hadoop.hive.common.type.HiveDecimal	getHiveDecimalFromByteBuffer(java.nio.ByteBuffer byteBuffer, int scale)
static org.apache.avro.Schema	getOtherTypeFromNullableType(org.apache.avro.Schema unionSchema) If the union schema is a nullable union, get the schema for the non-nullable type.
static org.apache.avro.Schema	getSchemaFor(java.io.File file)
static org.apache.avro.Schema	getSchemaFor(java.io.InputStream stream)
static org.apache.avro.Schema	getSchemaFor(java.lang.String str)
static org.apache.avro.Schema	getSchemaFor(java.net.URL url)
protected static org.apache.avro.Schema	getSchemaFromFS(java.lang.String schemaFSUrl, org.apache.hadoop.conf.Configuration conf)
static void	handleAlterTableForAvro(org.apache.hadoop.hive.conf.HiveConf conf, java.lang.String serializationLib, java.util.Map<java.lang.String,java.lang.String> parameters) Called on specific alter table events, removes schema url and schema literal from given tblproperties After the change, HMS solely will be responsible for handling the schema
static boolean	insideMRJob(org.apache.hadoop.mapred.JobConf job) Determine if we're being executed from within an MR job or as part of a select * statement.
static boolean	isNullableType(org.apache.avro.Schema schema) Determine if an Avro schema is of type Union[T, NULL].

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SCHEMA_LITERAL

@Deprecated
public static final java.lang.String SCHEMA_LITERAL

Deprecated.

See Also:

Constant Field Values

SCHEMA_URL

@Deprecated
public static final java.lang.String SCHEMA_URL

Deprecated.

See Also:

Constant Field Values

SCHEMA_NAMESPACE

@Deprecated
public static final java.lang.String SCHEMA_NAMESPACE

Deprecated.

See Also:

Constant Field Values

SCHEMA_NAME

@Deprecated
public static final java.lang.String SCHEMA_NAME

Deprecated.

See Also:

Constant Field Values

SCHEMA_DOC

@Deprecated
public static final java.lang.String SCHEMA_DOC

Deprecated.

See Also:

Constant Field Values

AVRO_SERDE_SCHEMA

@Deprecated
public static final java.lang.String AVRO_SERDE_SCHEMA

Deprecated.

SCHEMA_RETRIEVER

@Deprecated
public static final java.lang.String SCHEMA_RETRIEVER

Deprecated.

SCHEMA_NONE

public static final java.lang.String SCHEMA_NONE

See Also:

Constant Field Values

EXCEPTION_MESSAGE

public static final java.lang.String EXCEPTION_MESSAGE

Constructor Detail

AvroSerdeUtils

public AvroSerdeUtils()

Method Detail

determineSchemaOrThrowException

public static org.apache.avro.Schema determineSchemaOrThrowException(org.apache.hadoop.conf.Configuration conf,
                                                                     java.util.Properties properties)
                                                              throws java.io.IOException,
                                                                     org.apache.hadoop.hive.serde2.avro.AvroSerdeException

Determine the schema to that's been provided for Avro serde work.

Parameters:

properties - containing a key pointing to the schema, one way or another

Returns:

schema to use while serdeing the avro file

Throws:

java.io.IOException - if error while trying to read the schema from another location

AvroSerdeException - if unable to find a schema or pointer to it in the properties

getSchemaFromFS

protected static org.apache.avro.Schema getSchemaFromFS(java.lang.String schemaFSUrl,
                                                        org.apache.hadoop.conf.Configuration conf)
                                                 throws java.io.IOException,
                                                        java.net.URISyntaxException

Throws:

java.io.IOException

java.net.URISyntaxException

isNullableType

public static boolean isNullableType(org.apache.avro.Schema schema)

Determine if an Avro schema is of type Union[T, NULL]. Avro supports nullable types via a union of type T and null. This is a very common use case. As such, we want to silently convert it to just T and allow the value to be null. When a Hive union type is used with AVRO, the schema type becomes Union[NULL, T1, T2, ...]. The NULL in the union should be silently removed

Returns:

true if type represents Union[T, Null], false otherwise

getOtherTypeFromNullableType

public static org.apache.avro.Schema getOtherTypeFromNullableType(org.apache.avro.Schema unionSchema)

If the union schema is a nullable union, get the schema for the non-nullable type. This method does no checking that the provided Schema is nullable. If the provided union schema is non-nullable, it simply returns the union schema

insideMRJob

public static boolean insideMRJob(org.apache.hadoop.mapred.JobConf job)

Determine if we're being executed from within an MR job or as part of a select * statement. The signals for this varies between Hive versions.

Parameters:

job - that contains things that are or are not set in a job

Returns:

Are we in a job or not?

getBufferFromBytes

public static java.nio.Buffer getBufferFromBytes(byte[] input)

getBufferFromDecimal

public static java.nio.Buffer getBufferFromDecimal(org.apache.hadoop.hive.common.type.HiveDecimal dec,
                                                   int scale)

getBytesFromByteBuffer

public static byte[] getBytesFromByteBuffer(java.nio.ByteBuffer byteBuffer)

getHiveDecimalFromByteBuffer

public static org.apache.hadoop.hive.common.type.HiveDecimal getHiveDecimalFromByteBuffer(java.nio.ByteBuffer byteBuffer,
                                                                                          int scale)

getSchemaFor

public static org.apache.avro.Schema getSchemaFor(java.lang.String str)

getSchemaFor

public static org.apache.avro.Schema getSchemaFor(java.io.File file)

getSchemaFor

public static org.apache.avro.Schema getSchemaFor(java.io.InputStream stream)

getSchemaFor

public static org.apache.avro.Schema getSchemaFor(java.net.URL url)

handleAlterTableForAvro

public static void handleAlterTableForAvro(org.apache.hadoop.hive.conf.HiveConf conf,
                                           java.lang.String serializationLib,
                                           java.util.Map<java.lang.String,java.lang.String> parameters)

Called on specific alter table events, removes schema url and schema literal from given tblproperties After the change, HMS solely will be responsible for handling the schema

Parameters:

conf -

serializationLib -

parameters -