org.apache.parquet.hadoop.codec

Class ZstandardCodec

java.lang.Object

org.apache.parquet.hadoop.codec.ZstandardCodec

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, org.apache.hadoop.io.compress.CompressionCodec

public class ZstandardCodec
extends Object
implements org.apache.hadoop.conf.Configurable, org.apache.hadoop.io.compress.CompressionCodec

ZSTD compression codec for Parquet. We do not use the default hadoop one because it requires 1) to set up hadoop on local development machine; 2) to upgrade hadoop to the newer version to have ZSTD support which is more cumbersome than upgrading parquet version.

This implementation relies on ZSTD JNI(https://github.com/luben/zstd-jni) which is already a dependency for Parquet. ZSTD JNI ZstdOutputStream and ZstdInputStream use Zstd internally. So no need to create compressor and decompressor in ZstandardCodec.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.io.compress.CompressionCodec

org.apache.hadoop.io.compress.CompressionCodec.Util

Field Summary

Fields

Modifier and Type	Field and Description
static boolean	DEFAULT_PARQUET_COMPRESS_ZSTD_BUFFERPOOL_ENABLED
static int	DEFAULT_PARQUET_COMPRESS_ZSTD_LEVEL
static int	DEFAULTPARQUET_COMPRESS_ZSTD_WORKERS
static String	PARQUET_COMPRESS_ZSTD_BUFFERPOOL_ENABLED
static String	PARQUET_COMPRESS_ZSTD_LEVEL
static String	PARQUET_COMPRESS_ZSTD_WORKERS

Constructor Summary

Constructors

Constructor and Description
ZstandardCodec()

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.io.compress.Compressor	createCompressor()
org.apache.hadoop.io.compress.Decompressor	createDecompressor()
org.apache.hadoop.io.compress.CompressionInputStream	createInputStream(InputStream stream)
org.apache.hadoop.io.compress.CompressionInputStream	createInputStream(InputStream stream, org.apache.hadoop.io.compress.Decompressor decompressor)
org.apache.hadoop.io.compress.CompressionOutputStream	createOutputStream(OutputStream stream)
org.apache.hadoop.io.compress.CompressionOutputStream	createOutputStream(OutputStream stream, org.apache.hadoop.io.compress.Compressor compressor)
Class<? extends org.apache.hadoop.io.compress.Compressor>	getCompressorType()
org.apache.hadoop.conf.Configuration	getConf()
Class<? extends org.apache.hadoop.io.compress.Decompressor>	getDecompressorType()
String	getDefaultExtension()
void	setConf(org.apache.hadoop.conf.Configuration conf)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PARQUET_COMPRESS_ZSTD_BUFFERPOOL_ENABLED

public static final String PARQUET_COMPRESS_ZSTD_BUFFERPOOL_ENABLED

See Also:

Constant Field Values

DEFAULT_PARQUET_COMPRESS_ZSTD_BUFFERPOOL_ENABLED

public static final boolean DEFAULT_PARQUET_COMPRESS_ZSTD_BUFFERPOOL_ENABLED

See Also:

Constant Field Values

PARQUET_COMPRESS_ZSTD_LEVEL

public static final String PARQUET_COMPRESS_ZSTD_LEVEL

See Also:

Constant Field Values

DEFAULT_PARQUET_COMPRESS_ZSTD_LEVEL

public static final int DEFAULT_PARQUET_COMPRESS_ZSTD_LEVEL

See Also:

Constant Field Values

PARQUET_COMPRESS_ZSTD_WORKERS

public static final String PARQUET_COMPRESS_ZSTD_WORKERS

See Also:

Constant Field Values

DEFAULTPARQUET_COMPRESS_ZSTD_WORKERS

public static final int DEFAULTPARQUET_COMPRESS_ZSTD_WORKERS

See Also:

Constant Field Values

Constructor Detail

ZstandardCodec

public ZstandardCodec()

Method Detail

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

getConf

public org.apache.hadoop.conf.Configuration getConf()

Specified by:

getConf in interface org.apache.hadoop.conf.Configurable

createCompressor

public org.apache.hadoop.io.compress.Compressor createCompressor()

Specified by:

createCompressor in interface org.apache.hadoop.io.compress.CompressionCodec

createDecompressor

public org.apache.hadoop.io.compress.Decompressor createDecompressor()

Specified by:

createDecompressor in interface org.apache.hadoop.io.compress.CompressionCodec

createInputStream

public org.apache.hadoop.io.compress.CompressionInputStream createInputStream(InputStream stream,
                                                                              org.apache.hadoop.io.compress.Decompressor decompressor)
                                                                       throws IOException

Specified by:

createInputStream in interface org.apache.hadoop.io.compress.CompressionCodec

Throws:

IOException

createInputStream

public org.apache.hadoop.io.compress.CompressionInputStream createInputStream(InputStream stream)
                                                                       throws IOException

Specified by:

createInputStream in interface org.apache.hadoop.io.compress.CompressionCodec

Throws:

IOException

createOutputStream

public org.apache.hadoop.io.compress.CompressionOutputStream createOutputStream(OutputStream stream,
                                                                                org.apache.hadoop.io.compress.Compressor compressor)
                                                                         throws IOException

Specified by:

createOutputStream in interface org.apache.hadoop.io.compress.CompressionCodec

Throws:

IOException

createOutputStream

public org.apache.hadoop.io.compress.CompressionOutputStream createOutputStream(OutputStream stream)
                                                                         throws IOException

Specified by:

createOutputStream in interface org.apache.hadoop.io.compress.CompressionCodec

Throws:

IOException

getCompressorType

public Class<? extends org.apache.hadoop.io.compress.Compressor> getCompressorType()

Specified by:

getCompressorType in interface org.apache.hadoop.io.compress.CompressionCodec

getDecompressorType

public Class<? extends org.apache.hadoop.io.compress.Decompressor> getDecompressorType()

Specified by:

getDecompressorType in interface org.apache.hadoop.io.compress.CompressionCodec

getDefaultExtension

public String getDefaultExtension()

Specified by:

getDefaultExtension in interface org.apache.hadoop.io.compress.CompressionCodec