org.apache.hudi.hadoop.utils

Class HoodieRealtimeRecordReaderUtils

java.lang.Object

org.apache.hudi.hadoop.utils.HoodieRealtimeRecordReaderUtils

public class HoodieRealtimeRecordReaderUtils
extends Object

Constructor Summary

Constructors

Constructor and Description
HoodieRealtimeRecordReaderUtils()

Method Summary

Modifier and Type	Method and Description
static org.apache.avro.Schema	addPartitionFields(org.apache.avro.Schema schema, List<String> partitioningFields) Hive implementation of ParquetRecordReader results in partition columns not present in the original parquet file to also be part of the projected schema.
static String	arrayWritableToString(org.apache.hadoop.io.ArrayWritable writable) Prints a JSON representation of the ArrayWritable for easier debuggability.
static org.apache.hadoop.io.Writable	avroToArrayWritable(Object value, org.apache.avro.Schema schema) Convert the projected read from delta record into an array writable.
static org.apache.avro.Schema	generateProjectionSchema(org.apache.avro.Schema writeSchema, Map<String,org.apache.avro.Schema.Field> schemaFieldsMap, List<String> fieldNames) Generate a reader schema off the provided writeSchema, to just project out the provided columns.
static long	getMaxCompactionMemoryInBytes(org.apache.hadoop.mapred.JobConf jobConf) get the max compaction memory in bytes from JobConf.
static Map<String,org.apache.avro.Schema.Field>	getNameToFieldMap(org.apache.avro.Schema schema)
static List<String>	orderFields(String fieldNameCsv, String fieldOrderCsv, List<String> partitioningFields) Given a comma separated list of field names and positions at which they appear on Hive, return an ordered list of field names, that can be passed onto storage.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HoodieRealtimeRecordReaderUtils

public HoodieRealtimeRecordReaderUtils()

Method Detail

getMaxCompactionMemoryInBytes

public static long getMaxCompactionMemoryInBytes(org.apache.hadoop.mapred.JobConf jobConf)

get the max compaction memory in bytes from JobConf.

arrayWritableToString

public static String arrayWritableToString(org.apache.hadoop.io.ArrayWritable writable)

Prints a JSON representation of the ArrayWritable for easier debuggability.

generateProjectionSchema

public static org.apache.avro.Schema generateProjectionSchema(org.apache.avro.Schema writeSchema,
                                                              Map<String,org.apache.avro.Schema.Field> schemaFieldsMap,
                                                              List<String> fieldNames)

Generate a reader schema off the provided writeSchema, to just project out the provided columns.

getNameToFieldMap

public static Map<String,org.apache.avro.Schema.Field> getNameToFieldMap(org.apache.avro.Schema schema)

avroToArrayWritable

public static org.apache.hadoop.io.Writable avroToArrayWritable(Object value,
                                                                org.apache.avro.Schema schema)

Convert the projected read from delta record into an array writable.

orderFields

public static List<String> orderFields(String fieldNameCsv,
                                       String fieldOrderCsv,
                                       List<String> partitioningFields)

Given a comma separated list of field names and positions at which they appear on Hive, return an ordered list of field names, that can be passed onto storage.

addPartitionFields

public static org.apache.avro.Schema addPartitionFields(org.apache.avro.Schema schema,
                                                        List<String> partitioningFields)

Hive implementation of ParquetRecordReader results in partition columns not present in the original parquet file to also be part of the projected schema. Hive expects the record reader implementation to return the row in its entirety (with un-projected column having null values). As we use writerSchema for this, make sure writer schema also includes partition columns

Parameters:

schema - Schema to be changed