org.apache.parquet.hadoop

Class CodecFactory

java.lang.Object

org.apache.parquet.hadoop.CodecFactory

All Implemented Interfaces:

CompressionCodecFactory

public class CodecFactory
extends Object
implements CompressionCodecFactory

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CodecFactory.BytesCompressor Deprecated. will be removed in 2.0.0; use CompressionCodecFactory.BytesInputCompressor instead.
static class	CodecFactory.BytesDecompressor Deprecated. will be removed in 2.0.0; use CompressionCodecFactory.BytesInputDecompressor instead.

Nested classes/interfaces inherited from interface org.apache.parquet.compression.CompressionCodecFactory

CompressionCodecFactory.BytesInputCompressor, CompressionCodecFactory.BytesInputDecompressor

Field Summary

Fields

Modifier and Type	Field and Description
protected static Map<String,org.apache.hadoop.io.compress.CompressionCodec>	CODEC_BY_NAME
protected ParquetConfiguration	conf
protected org.apache.hadoop.conf.Configuration	configuration Deprecated.
protected int	pageSize

Constructor Summary

Constructors

Constructor and Description
CodecFactory(org.apache.hadoop.conf.Configuration configuration, int pageSize) Create a new codec factory.
CodecFactory(ParquetConfiguration configuration, int pageSize) Create a new codec factory.

Method Summary

Modifier and Type	Method and Description
protected CodecFactory.BytesCompressor	createCompressor(CompressionCodecName codecName)
protected CodecFactory.BytesDecompressor	createDecompressor(CompressionCodecName codecName)
static CodecFactory	createDirectCodecFactory(org.apache.hadoop.conf.Configuration config, ByteBufferAllocator allocator, int pageSize) Create a codec factory that will provide compressors and decompressors that will work natively with ByteBuffers backed by direct memory.
protected org.apache.hadoop.io.compress.CompressionCodec	getCodec(CompressionCodecName codecName)
CodecFactory.BytesCompressor	getCompressor(CompressionCodecName codecName)
CodecFactory.BytesDecompressor	getDecompressor(CompressionCodecName codecName)
void	release()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CODEC_BY_NAME

protected static final Map<String,org.apache.hadoop.io.compress.CompressionCodec> CODEC_BY_NAME

conf

protected final ParquetConfiguration conf

pageSize

protected final int pageSize

configuration

@Deprecated
protected final org.apache.hadoop.conf.Configuration configuration

Deprecated.

Constructor Detail

CodecFactory

public CodecFactory(org.apache.hadoop.conf.Configuration configuration,
                    int pageSize)

Create a new codec factory.

Parameters:

configuration - used to pass compression codec configuration information

pageSize - the expected page size, does not set a hard limit, currently just used to set the initial size of the output stream used when compressing a buffer. If this factory is only used to construct decompressors this parameter has no impact on the function of the factory

CodecFactory

public CodecFactory(ParquetConfiguration configuration,
                    int pageSize)

Create a new codec factory.

Parameters:

configuration - used to pass compression codec configuration information

pageSize - the expected page size, does not set a hard limit, currently just used to set the initial size of the output stream used when compressing a buffer. If this factory is only used to construct decompressors this parameter has no impact on the function of the factory

Method Detail

createDirectCodecFactory

public static CodecFactory createDirectCodecFactory(org.apache.hadoop.conf.Configuration config,
                                                    ByteBufferAllocator allocator,
                                                    int pageSize)

Create a codec factory that will provide compressors and decompressors that will work natively with ByteBuffers backed by direct memory.

Parameters:

config - configuration options for different compression codecs

allocator - an allocator for creating result buffers during compression and decompression, must provide buffers backed by Direct memory and return true for the isDirect() method on the ByteBufferAllocator interface

pageSize - the default page size. This does not set a hard limit on the size of buffers that can be compressed, but performance may be improved by setting it close to the expected size of buffers (in the case of parquet, pages) that will be compressed. This setting is unused in the case of decompressing data, as parquet always records the uncompressed size of a buffer. If this CodecFactory is only going to be used for decompressors, this parameter will not impact the function of the factory.

Returns:

a configured direct codec factory

getCompressor

public CodecFactory.BytesCompressor getCompressor(CompressionCodecName codecName)

Specified by:

getCompressor in interface CompressionCodecFactory

getDecompressor

public CodecFactory.BytesDecompressor getDecompressor(CompressionCodecName codecName)

Specified by:

getDecompressor in interface CompressionCodecFactory

createCompressor

protected CodecFactory.BytesCompressor createCompressor(CompressionCodecName codecName)

createDecompressor

protected CodecFactory.BytesDecompressor createDecompressor(CompressionCodecName codecName)

getCodec

protected org.apache.hadoop.io.compress.CompressionCodec getCodec(CompressionCodecName codecName)

Parameters:

codecName - the requested codec

Returns:

the corresponding hadoop codec. null if UNCOMPRESSED

release

public void release()

Specified by:

release in interface CompressionCodecFactory