org.apache.iceberg.spark.actions

Class SparkSortStrategy

java.lang.Object

org.apache.iceberg.actions.BinPackStrategy

org.apache.iceberg.actions.SortStrategy

org.apache.iceberg.spark.actions.SparkSortStrategy

All Implemented Interfaces:

java.io.Serializable, org.apache.iceberg.actions.RewriteStrategy

Direct Known Subclasses:

SparkZOrderStrategy

public class SparkSortStrategy
extends org.apache.iceberg.actions.SortStrategy

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	COMPRESSION_FACTOR The number of shuffle partitions and consequently the number of output files created by the Spark Sort is based on the size of the input data files used in this rewrite operation.

Fields inherited from class org.apache.iceberg.actions.BinPackStrategy

DELETE_FILE_THRESHOLD, DELETE_FILE_THRESHOLD_DEFAULT, MAX_FILE_SIZE_BYTES, MAX_FILE_SIZE_DEFAULT_RATIO, MIN_FILE_SIZE_BYTES, MIN_FILE_SIZE_DEFAULT_RATIO, MIN_INPUT_FILES, MIN_INPUT_FILES_DEFAULT, REWRITE_ALL, REWRITE_ALL_DEFAULT

Constructor Summary

Constructors

Constructor and Description
SparkSortStrategy(org.apache.iceberg.Table table, org.apache.spark.sql.SparkSession spark)

Method Summary

Modifier and Type	Method and Description
protected FileScanTaskSetManager	manager()
org.apache.iceberg.actions.RewriteStrategy	options(java.util.Map<java.lang.String,java.lang.String> options)
protected FileRewriteCoordinator	rewriteCoordinator()
java.util.Set<org.apache.iceberg.DataFile>	rewriteFiles(java.util.List<org.apache.iceberg.FileScanTask> filesToRewrite)
protected double	sizeEstimateMultiple()
protected org.apache.spark.sql.catalyst.plans.logical.LogicalPlan	sortPlan(org.apache.spark.sql.connector.distributions.Distribution distribution, org.apache.spark.sql.connector.expressions.SortOrder[] ordering, org.apache.spark.sql.catalyst.plans.logical.LogicalPlan plan, org.apache.spark.sql.internal.SQLConf conf)
protected org.apache.spark.sql.SparkSession	spark()
org.apache.iceberg.Table	table()
protected SparkTableCache	tableCache()
java.util.Set<java.lang.String>	validOptions()

Methods inherited from class org.apache.iceberg.actions.SortStrategy

name, sortOrder, sortOrder, validateOptions

Methods inherited from class org.apache.iceberg.actions.BinPackStrategy

inputFileSize, numOutputFiles, planFileGroups, selectFilesToRewrite, splitSize, targetFileSize, writeMaxFileSize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

COMPRESSION_FACTOR

public static final java.lang.String COMPRESSION_FACTOR

The number of shuffle partitions and consequently the number of output files created by the Spark Sort is based on the size of the input data files used in this rewrite operation. Due to compression, the disk file sizes may not accurately represent the size of files in the output. This parameter lets the user adjust the file size used for estimating actual output data size. A factor greater than 1.0 would generate more files than we would expect based on the on-disk file size. A value less than 1.0 would create fewer files than we would expect due to the on-disk size.

See Also:

Constant Field Values

Constructor Detail

SparkSortStrategy

public SparkSortStrategy(org.apache.iceberg.Table table,
                         org.apache.spark.sql.SparkSession spark)

Method Detail

table

public org.apache.iceberg.Table table()

validOptions

public java.util.Set<java.lang.String> validOptions()

Specified by:

validOptions in interface org.apache.iceberg.actions.RewriteStrategy

Overrides:

validOptions in class org.apache.iceberg.actions.SortStrategy

options

public org.apache.iceberg.actions.RewriteStrategy options(java.util.Map<java.lang.String,java.lang.String> options)

Specified by:

options in interface org.apache.iceberg.actions.RewriteStrategy

Overrides:

options in class org.apache.iceberg.actions.SortStrategy

rewriteFiles

public java.util.Set<org.apache.iceberg.DataFile> rewriteFiles(java.util.List<org.apache.iceberg.FileScanTask> filesToRewrite)

spark

protected org.apache.spark.sql.SparkSession spark()

sortPlan

protected org.apache.spark.sql.catalyst.plans.logical.LogicalPlan sortPlan(org.apache.spark.sql.connector.distributions.Distribution distribution,
                                                                           org.apache.spark.sql.connector.expressions.SortOrder[] ordering,
                                                                           org.apache.spark.sql.catalyst.plans.logical.LogicalPlan plan,
                                                                           org.apache.spark.sql.internal.SQLConf conf)

sizeEstimateMultiple

protected double sizeEstimateMultiple()

tableCache

protected SparkTableCache tableCache()

manager

protected FileScanTaskSetManager manager()

rewriteCoordinator

protected FileRewriteCoordinator rewriteCoordinator()