parquet.hadoop.api

Class ReadSupport<T>

java.lang.Object

parquet.hadoop.api.ReadSupport<T>

Type Parameters:

T - the type of the materialized record

Direct Known Subclasses:

GroupReadSupport

public abstract class ReadSupport<T>
extends Object

Abstraction used by the ParquetInputFormat to materialize records

Author:

Julien Le Dem

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ReadSupport.ReadContext information to read the file

Field Summary

Fields

Modifier and Type	Field and Description
static String	PARQUET_READ_SCHEMA configuration key for a parquet read projection schema

Constructor Summary

Constructors

Constructor and Description
ReadSupport()

Method Summary

Methods

Modifier and Type	Method and Description
static MessageType	getSchemaForRead(MessageType fileMessageType, MessageType projectedMessageType)
static MessageType	getSchemaForRead(MessageType fileMessageType, String partialReadSchemaString) attempts to validate and construct a MessageType from a read projection schema
ReadSupport.ReadContext	init(org.apache.hadoop.conf.Configuration configuration, Map<String,String> keyValueMetaData, MessageType fileSchema) Deprecated. override init(InitContext) instead
ReadSupport.ReadContext	init(InitContext context) called in InputFormat.getSplits(org.apache.hadoop.mapreduce.JobContext) in the front end
abstract RecordMaterializer<T>	prepareForRead(org.apache.hadoop.conf.Configuration configuration, Map<String,String> keyValueMetaData, MessageType fileSchema, ReadSupport.ReadContext readContext) called in RecordReader.initialize(org.apache.hadoop.mapreduce.InputSplit, org.apache.hadoop.mapreduce.TaskAttemptContext) in the back end the returned RecordConsumer will materialize the records and add them to the destination

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PARQUET_READ_SCHEMA

public static final String PARQUET_READ_SCHEMA

configuration key for a parquet read projection schema

See Also:

Constant Field Values

Constructor Detail

ReadSupport

public ReadSupport()

Method Detail

getSchemaForRead

public static MessageType getSchemaForRead(MessageType fileMessageType,
                           String partialReadSchemaString)

attempts to validate and construct a MessageType from a read projection schema

Parameters:

fileMessageType - the typed schema of the source

partialReadSchemaString - the requested projection schema

Returns:

the typed schema that should be used to read

getSchemaForRead

public static MessageType getSchemaForRead(MessageType fileMessageType,
                           MessageType projectedMessageType)

init

@Deprecated
public ReadSupport.ReadContext init(org.apache.hadoop.conf.Configuration configuration,
                                      Map<String,String> keyValueMetaData,
                                      MessageType fileSchema)

Deprecated. override init(InitContext) instead

called in InputFormat.getSplits(org.apache.hadoop.mapreduce.JobContext) in the front end

Parameters:

configuration - the job configuration

keyValueMetaData - the app specific metadata from the file

fileSchema - the schema of the file

Returns:

the readContext that defines how to read the file

init

public ReadSupport.ReadContext init(InitContext context)

called in InputFormat.getSplits(org.apache.hadoop.mapreduce.JobContext) in the front end

Parameters:

context - the initialisation context

Returns:

the readContext that defines how to read the file

prepareForRead

public abstract RecordMaterializer<T> prepareForRead(org.apache.hadoop.conf.Configuration configuration,
                                   Map<String,String> keyValueMetaData,
                                   MessageType fileSchema,
                                   ReadSupport.ReadContext readContext)

called in RecordReader.initialize(org.apache.hadoop.mapreduce.InputSplit, org.apache.hadoop.mapreduce.TaskAttemptContext) in the back end the returned RecordConsumer will materialize the records and add them to the destination

Parameters:

configuration - the job configuration

keyValueMetaData - the app specific metadata from the file

fileSchema - the schema of the file

readContext - returned by the init method

Returns:

the recordConsumer that will receive the events