org.apache.parquet.avro

Class AvroParquetOutputFormat<T>

java.lang.Object

org.apache.hadoop.mapreduce.OutputFormat<K,V>

org.apache.hadoop.mapreduce.lib.output.FileOutputFormat<Void,T>

org.apache.parquet.hadoop.ParquetOutputFormat<T>

org.apache.parquet.avro.AvroParquetOutputFormat<T>

public class AvroParquetOutputFormat<T>
extends ParquetOutputFormat<T>

A Hadoop OutputFormat for Parquet files.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.parquet.hadoop.ParquetOutputFormat

ParquetOutputFormat.JobSummaryLevel

Nested classes/interfaces inherited from class org.apache.hadoop.mapreduce.lib.output.FileOutputFormat

org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.Counter

Field Summary

Fields inherited from class org.apache.parquet.hadoop.ParquetOutputFormat

ADAPTIVE_BLOOM_FILTER_ENABLED, BLOCK_SIZE, BLOOM_FILTER_CANDIDATES_NUMBER, BLOOM_FILTER_ENABLED, BLOOM_FILTER_EXPECTED_NDV, BLOOM_FILTER_FPP, BLOOM_FILTER_MAX_BYTES, COLUMN_INDEX_TRUNCATE_LENGTH, COMPRESSION, DICTIONARY_PAGE_SIZE, ENABLE_DICTIONARY, ENABLE_JOB_SUMMARY, ESTIMATE_PAGE_SIZE_CHECK, JOB_SUMMARY_LEVEL, MAX_PADDING_BYTES, MAX_ROW_COUNT_FOR_PAGE_SIZE_CHECK, MEMORY_POOL_RATIO, MIN_MEMORY_ALLOCATION, MIN_ROW_COUNT_FOR_PAGE_SIZE_CHECK, PAGE_ROW_COUNT_LIMIT, PAGE_SIZE, PAGE_VALUE_COUNT_THRESHOLD, PAGE_WRITE_CHECKSUM_ENABLED, SIZE_STATISTICS_ENABLED, STATISTICS_ENABLED, STATISTICS_TRUNCATE_LENGTH, VALIDATION, WRITE_SUPPORT_CLASS, WRITER_VERSION

Fields inherited from class org.apache.hadoop.mapreduce.lib.output.FileOutputFormat

BASE_OUTPUT_NAME, COMPRESS, COMPRESS_CODEC, COMPRESS_TYPE, OUTDIR, PART

Constructor Summary

Constructors

Constructor and Description
AvroParquetOutputFormat()

Method Summary

Modifier and Type	Method and Description
static void	setAvroDataSupplier(org.apache.hadoop.mapreduce.Job job, Class<? extends AvroDataSupplier> supplierClass) Sets the AvroDataSupplier class that will be used.
static void	setSchema(org.apache.hadoop.mapreduce.Job job, org.apache.avro.Schema schema) Set the Avro schema to use for writing.

Methods inherited from class org.apache.parquet.hadoop.ParquetOutputFormat

createEncryptionProperties, getAdaptiveBloomFilterEnabled, getBlockSize, getBlockSize, getBloomFilterEnabled, getBloomFilterMaxBytes, getCompression, getCompression, getDictionaryPageSize, getDictionaryPageSize, getEnableDictionary, getEnableDictionary, getEstimatePageSizeCheck, getJobSummaryLevel, getLongBlockSize, getMaxRowCountForPageSizeCheck, getMemoryManager, getMinRowCountForPageSizeCheck, getOutputCommitter, getPageSize, getPageSize, getPageWriteChecksumEnabled, getRecordWriter, getRecordWriter, getRecordWriter, getRecordWriter, getRecordWriter, getRecordWriter, getSizeStatisticsEnabled, getSizeStatisticsEnabled, getStatisticsEnabled, getStatisticsEnabled, getValidation, getValidation, getValueCountThreshold, getWriterVersion, getWriteSupport, getWriteSupportClass, isCompressionSet, isCompressionSet, setBlockSize, setColumnIndexTruncateLength, setColumnIndexTruncateLength, setCompression, setDictionaryPageSize, setEnableDictionary, setMaxPaddingSize, setMaxPaddingSize, setPageRowCountLimit, setPageRowCountLimit, setPageSize, setPageWriteChecksumEnabled, setPageWriteChecksumEnabled, setSizeStatisticsEnabled, setSizeStatisticsEnabled, setStatisticsEnabled, setStatisticsEnabled, setStatisticsTruncateLength, setValidation, setValidation, setWriteSupportClass, setWriteSupportClass

Methods inherited from class org.apache.hadoop.mapreduce.lib.output.FileOutputFormat

checkOutputSpecs, getCompressOutput, getDefaultWorkFile, getOutputCompressorClass, getOutputName, getOutputPath, getPathForWorkFile, getUniqueFile, getWorkOutputPath, setCompressOutput, setOutputCompressorClass, setOutputName, setOutputPath

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AvroParquetOutputFormat

public AvroParquetOutputFormat()

Method Detail

setSchema

public static void setSchema(org.apache.hadoop.mapreduce.Job job,
                             org.apache.avro.Schema schema)

Set the Avro schema to use for writing. The schema is translated into a Parquet schema so that the records can be written in Parquet format. It is also stored in the Parquet metadata so that records can be reconstructed as Avro objects at read time without specifying a read schema.

Parameters:

job - a job

schema - a schema for the data that will be written

See Also:

AvroParquetInputFormat.setAvroReadSchema(org.apache.hadoop.mapreduce.Job, org.apache.avro.Schema)

setAvroDataSupplier

public static void setAvroDataSupplier(org.apache.hadoop.mapreduce.Job job,
                                       Class<? extends AvroDataSupplier> supplierClass)

Sets the AvroDataSupplier class that will be used. The data supplier provides instances of GenericData that are used to deconstruct records.

Parameters:

job - a Job to configure

supplierClass - a supplier class