org.apache.hudi.io.hadoop

Class HoodieBaseParquetWriter<R>

java.lang.Object

org.apache.hudi.io.hadoop.HoodieBaseParquetWriter<R>

Type Parameters:

R - target type of the object being written into Parquet files (for ex, IndexedRecord, InternalRow)

All Implemented Interfaces:

Closeable, AutoCloseable

Direct Known Subclasses:

HoodieAvroParquetWriter

public abstract class HoodieBaseParquetWriter<R>
extends Object
implements Closeable

Base class of Hudi's custom ParquetWriter implementations

Field Summary

Fields

Modifier and Type	Field and Description
static String	BLOOM_FILTER_ENABLED
static String	BLOOM_FILTER_EXPECTED_NDV

Constructor Summary

Constructors

Constructor and Description
HoodieBaseParquetWriter(StoragePath file, HoodieParquetConfig<? extends org.apache.parquet.hadoop.api.WriteSupport<R>> parquetConfig)

Method Summary

Modifier and Type	Method and Description
boolean	canWrite()
void	close()
long	getDataSize()
protected long	getRecordCountForNextSizeCheck()
protected long	getWrittenRecordCount()
protected void	handleParquetBloomFilters(org.apache.parquet.hadoop.ParquetWriter.Builder parquetWriterbuilder, org.apache.hadoop.conf.Configuration hadoopConf) Once we get parquet version >= 1.12 among all engines we can cleanup the reflexion hack.
void	write(R object)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

BLOOM_FILTER_EXPECTED_NDV

public static final String BLOOM_FILTER_EXPECTED_NDV

See Also:

Constant Field Values

BLOOM_FILTER_ENABLED

public static final String BLOOM_FILTER_ENABLED

See Also:

Constant Field Values

Constructor Detail

HoodieBaseParquetWriter

public HoodieBaseParquetWriter(StoragePath file,
                               HoodieParquetConfig<? extends org.apache.parquet.hadoop.api.WriteSupport<R>> parquetConfig)
                        throws IOException

Throws:

IOException

Method Detail

handleParquetBloomFilters

protected void handleParquetBloomFilters(org.apache.parquet.hadoop.ParquetWriter.Builder parquetWriterbuilder,
                                         org.apache.hadoop.conf.Configuration hadoopConf)

Once we get parquet version >= 1.12 among all engines we can cleanup the reflexion hack.

Parameters:

parquetWriterbuilder -

hadoopConf -

canWrite

public boolean canWrite()

getDataSize

public long getDataSize()

write

public void write(R object)
           throws IOException

Throws:

IOException

getWrittenRecordCount

protected long getWrittenRecordCount()

getRecordCountForNextSizeCheck

protected long getRecordCountForNextSizeCheck()

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException