trait OptimizeTableStrategy extends AnyRef
Defines set of utilities used in OptimizeTableCommand. The behavior of these utilities will change based on the OptimizeTableMode: COMPACTION, ZORDER and CLUSTERING.
- Alphabetic
- By Inheritance
- OptimizeTableStrategy
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Abstract Value Members
- abstract def curve: String
The clustering algorithm to be used by either by ZORDER or Liquid CLUSTERING.
The clustering algorithm to be used by either by ZORDER or Liquid CLUSTERING. An error is thrown for COMPACTION.
- abstract def maxBinSize: Long
Utility method to get max bin size in bytes to group files into.
- abstract def optimizeTableMode: OptimizeTableMode.Value
The optimize mode the strategy instance is created for.
- abstract def sparkSession: SparkSession
- abstract def updateOptimizeStats(optimizeStats: OptimizeStats, removedFiles: Seq[RemoveFile], bins: Seq[(Map[String, String], Seq[AddFile])]): Unit
Utility to update additional metrics after optimization.
Utility to update additional metrics after optimization.
- optimizeStats
The input stats to update on.
- removedFiles
Removed files.
- bins
Sequence of bin-packed file groups, where each group consists of a partition value and its associated files.
Concrete Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @native()
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable])
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def initNewBin: BinInfo
Prepare a new Bin and returns its initialized BinInfo.
Prepare a new Bin and returns its initialized BinInfo.
This function is expected to be called once for each bin before tagAddFile is called.
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- def prepareFilesPerPartition(inputFiles: Seq[AddFile]): Seq[AddFile]
Utility method to prepare files in a partition for optimization.
Utility method to prepare files in a partition for optimization.
By default it sorts files on the size for the binpack.
- returns
Prepared files for the subsequent optimization.
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def tagAddFile(file: AddFile, binInfo: BinInfo): AddFile
Incorporate essential tags for optimized files based on the OptimizeTableMode.
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()