Package org.apache.parquet.avro

Class AvroParquetWriter<T>

java.lang.Object

org.apache.parquet.hadoop.ParquetWriter<T>

org.apache.parquet.avro.AvroParquetWriter<T>

All Implemented Interfaces:

Closeable, AutoCloseable

public class AvroParquetWriter<T>
extends org.apache.parquet.hadoop.ParquetWriter<T>

Write Avro records to a Parquet file.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	AvroParquetWriter.Builder<T>

Field Summary

Fields inherited from class org.apache.parquet.hadoop.ParquetWriter

DEFAULT_BLOCK_SIZE, DEFAULT_COMPRESSION_CODEC_NAME, DEFAULT_IS_DICTIONARY_ENABLED, DEFAULT_IS_VALIDATING_ENABLED, DEFAULT_PAGE_SIZE, DEFAULT_WRITER_VERSION, MAX_PADDING_SIZE_DEFAULT, OBJECT_MODEL_NAME_PROP

Constructor Summary

Constructors

Constructor	Description
AvroParquetWriter(org.apache.hadoop.fs.Path file, org.apache.avro.Schema avroSchema)	Deprecated.
AvroParquetWriter(org.apache.hadoop.fs.Path file, org.apache.avro.Schema avroSchema, org.apache.parquet.hadoop.metadata.CompressionCodecName compressionCodecName, int blockSize, int pageSize)	Deprecated.
AvroParquetWriter(org.apache.hadoop.fs.Path file, org.apache.avro.Schema avroSchema, org.apache.parquet.hadoop.metadata.CompressionCodecName compressionCodecName, int blockSize, int pageSize, boolean enableDictionary)	Deprecated.
AvroParquetWriter(org.apache.hadoop.fs.Path file, org.apache.avro.Schema avroSchema, org.apache.parquet.hadoop.metadata.CompressionCodecName compressionCodecName, int blockSize, int pageSize, boolean enableDictionary, org.apache.hadoop.conf.Configuration conf)	Deprecated.

Method Summary

Modifier and Type	Method	Description
static <T> AvroParquetWriter.Builder<T>	builder(org.apache.hadoop.fs.Path file)	Deprecated. will be removed in 2.0.0; use builder(OutputFile) instead.
static <T> AvroParquetWriter.Builder<T>	builder(org.apache.parquet.io.OutputFile file)

Methods inherited from class org.apache.parquet.hadoop.ParquetWriter

close, getDataSize, getFooter, write

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AvroParquetWriter

@Deprecated
public AvroParquetWriter(org.apache.hadoop.fs.Path file,
                         org.apache.avro.Schema avroSchema,
                         org.apache.parquet.hadoop.metadata.CompressionCodecName compressionCodecName,
                         int blockSize,
                         int pageSize)
                  throws IOException

Deprecated.

Create a new AvroParquetWriter.

Parameters:

file - a file path

avroSchema - a schema for the write

compressionCodecName - compression codec

blockSize - target block size

pageSize - target page size

Throws:

IOException - if there is an error while writing

AvroParquetWriter

@Deprecated
public AvroParquetWriter(org.apache.hadoop.fs.Path file,
                         org.apache.avro.Schema avroSchema,
                         org.apache.parquet.hadoop.metadata.CompressionCodecName compressionCodecName,
                         int blockSize,
                         int pageSize,
                         boolean enableDictionary)
                  throws IOException

Deprecated.

Create a new AvroParquetWriter.

Parameters:

file - The file name to write to.

avroSchema - The schema to write with.

compressionCodecName - Compression code to use, or CompressionCodecName.UNCOMPRESSED

blockSize - the block size threshold.

pageSize - See parquet write up. Blocks are subdivided into pages for alignment and other purposes.

enableDictionary - Whether to use a dictionary to compress columns.

Throws:

IOException - if there is an error while writing

AvroParquetWriter

@Deprecated
public AvroParquetWriter(org.apache.hadoop.fs.Path file,
                         org.apache.avro.Schema avroSchema)
                  throws IOException

Deprecated.

Create a new AvroParquetWriter. The default block size is 128 MB. The default page size is 1 MB. Default compression is no compression. (Inherited from ParquetWriter)

Parameters:

file - The file name to write to.

avroSchema - The schema to write with.

Throws:

IOException - if there is an error while writing

AvroParquetWriter

@Deprecated
public AvroParquetWriter(org.apache.hadoop.fs.Path file,
                         org.apache.avro.Schema avroSchema,
                         org.apache.parquet.hadoop.metadata.CompressionCodecName compressionCodecName,
                         int blockSize,
                         int pageSize,
                         boolean enableDictionary,
                         org.apache.hadoop.conf.Configuration conf)
                  throws IOException

Deprecated.

Create a new AvroParquetWriter.

Parameters:

file - The file name to write to.

avroSchema - The schema to write with.

compressionCodecName - Compression code to use, or CompressionCodecName.UNCOMPRESSED

blockSize - the block size threshold.

pageSize - See parquet write up. Blocks are subdivided into pages for alignment and other purposes.

enableDictionary - Whether to use a dictionary to compress columns.

conf - The Configuration to use.

Throws:

IOException - if there is an error while writing

Method Detail

builder

@Deprecated
public static <T> AvroParquetWriter.Builder<T> builder(org.apache.hadoop.fs.Path file)

Deprecated.

will be removed in 2.0.0; use builder(OutputFile) instead.

Type Parameters:

T - the Java type of records to read from the file

Parameters:

file - a file path

Returns:

an Avro reader builder

builder

public static <T> AvroParquetWriter.Builder<T> builder(org.apache.parquet.io.OutputFile file)