org.apache.hudi.table.action.commit

Class JavaUpsertPartitioner<T>

java.lang.Object

org.apache.hudi.table.action.commit.JavaUpsertPartitioner<T>

All Implemented Interfaces:

Serializable, Partitioner

public class JavaUpsertPartitioner<T>
extends Object
implements Partitioner

Packs incoming records to be upserted, into buckets.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected HoodieWriteConfig	config
protected List<SmallFile>	smallFiles List of all small files to be corrected.
protected HoodieTable	table

Constructor Summary

Constructors

Constructor and Description
JavaUpsertPartitioner(WorkloadProfile workloadProfile, org.apache.hudi.common.engine.HoodieEngineContext context, HoodieTable table, HoodieWriteConfig config)

Method Summary

Modifier and Type	Method and Description
protected static long	averageBytesPerRecord(org.apache.hudi.common.table.timeline.HoodieTimeline commitTimeline, HoodieWriteConfig hoodieWriteConfig) Obtains the average record size based on records written during previous commits.
BucketInfo	getBucketInfo(int bucketNumber)
List<InsertBucketCumulativeWeightPair>	getInsertBuckets(String partitionPath)
int	getNumPartitions()
int	getPartition(Object key)
List<String>	getSmallFileIds()
protected List<SmallFile>	getSmallFiles(String partitionPath) Returns a list of small files in the given partition path.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

smallFiles

protected List<SmallFile> smallFiles

List of all small files to be corrected.

table

protected final HoodieTable table

config

protected final HoodieWriteConfig config

Constructor Detail

JavaUpsertPartitioner

public JavaUpsertPartitioner(WorkloadProfile workloadProfile,
                             org.apache.hudi.common.engine.HoodieEngineContext context,
                             HoodieTable table,
                             HoodieWriteConfig config)

Method Detail

getSmallFiles

protected List<SmallFile> getSmallFiles(String partitionPath)

Returns a list of small files in the given partition path.

getBucketInfo

public BucketInfo getBucketInfo(int bucketNumber)

getInsertBuckets

public List<InsertBucketCumulativeWeightPair> getInsertBuckets(String partitionPath)

getNumPartitions

public int getNumPartitions()

Specified by:

getNumPartitions in interface Partitioner

getPartition

public int getPartition(Object key)

Specified by:

getPartition in interface Partitioner

averageBytesPerRecord

protected static long averageBytesPerRecord(org.apache.hudi.common.table.timeline.HoodieTimeline commitTimeline,
                                            HoodieWriteConfig hoodieWriteConfig)

Obtains the average record size based on records written during previous commits. Used for estimating how many records pack into one file.

getSmallFileIds

public List<String> getSmallFileIds()