Package io.trino.operator.exchange

Class UniformPartitionRebalancer

java.lang.Object

io.trino.operator.exchange.UniformPartitionRebalancer

@ThreadSafe
public class UniformPartitionRebalancer
extends Object

Help in finding the skewness across writers when writing partitioned data using preferred partitioning. It then tries to uniformly distribute the biggest partitions from skewed writers to all the available writers.

Example:

Before: For three writers with skewed partitions Writer 1 -> No partition assigned -> 0 bytes Writer 2 -> No partition assigned -> 0 bytes Writer 3 -> Partition 1 (100MB) + Partition 2 (100MB) + Partition 3 (100MB) -> 300 MB

After scaling: Writer 1 -> Partition 1 (50MB) + Partition 3 (50MB) -> 100 MB Writer 2 -> Partition 2 (50MB) -> 50 MB Writer 3 -> Partition 1 (150MB) + Partition 2 (150MB) + Partition 3 (150MB) -> 450 MB

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static final record	UniformPartitionRebalancer.WriterPartitionId

Constructor Summary

Constructors

Constructor	Description
UniformPartitionRebalancer(List<Supplier<Long>> writerPhysicalWrittenBytesSuppliers, Supplier<it.unimi.dsi.fastutil.longs.Long2LongMap> partitionRowCountsSupplier, int partitionCount, int numberOfWriters, long writerMinSize)

Method Summary

Modifier and Type	Method	Description
int	getWriterId(int partitionId, int index)
void	rebalancePartitions()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

UniformPartitionRebalancer

public UniformPartitionRebalancer(List<Supplier<Long>> writerPhysicalWrittenBytesSuppliers,
 Supplier<it.unimi.dsi.fastutil.longs.Long2LongMap> partitionRowCountsSupplier,
 int partitionCount,
 int numberOfWriters,
 long writerMinSize)

Method Details

getWriterId

public int getWriterId(int partitionId,
 int index)

rebalancePartitions

public void rebalancePartitions()