class AutoCompactPartitionStats extends AnyRef

This singleton object collect the table partition statistic for each commit that creates AddFile or RemoveFile objects. To control the memory usage, there are maxNumTablePartitions per table and 'maxNumPartitions' partition entries across all tables. Note:

  1. Since the partition of each table is limited, if this limitation is reached, the least recently used table partitions will be evicted. 2. If all 'maxNumPartitions' are occupied, the partition stats of least recently used tables will be evicted until the used partitions fall back below to 'maxNumPartitions'. 3. The un-partitioned tables are treated as tables with single partition.
Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. AutoCompactPartitionStats
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new AutoCompactPartitionStats(maxNumTablePartitions: Int, maxNumPartitions: Int)

    maxNumTablePartitions

    The hash space of partition key to reduce memory usage per table.

    maxNumPartitions

    The maximum number of partition that can be occupied.

Type Members

  1. type PartitionFilesMap = LinkedHashMap[Int, Long]
  2. type PartitionKey = Map[String, String]
  3. type PartitionKeySet = Set[Map[String, String]]
  4. class PartitionStat extends AnyRef

    This class to store the states of one table partition.

    This class to store the states of one table partition. These state includes: -- the number of small files, -- the thread that assigned to compact this partition, and -- whether the partition was compacted.

    Note: Since this class keeps tracking of the statistics of the table partition and the state of the auto compaction thread that works on the table partition, any method that accesses any attribute of this class needs to be protected by synchronized context.

  5. type TablePartitionStats = LinkedHashMap[Int, PartitionStat]

    This hashtable is used to store all table partition states of a table, the key is the hashcode of the partition, the value is PartitionStat object.

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  6. def collectPartitionStats(collector: AutoCompactPartitionStatsCollector, tableId: String, actions: Iterator[Action]): Unit

    Collect the number of files, which are less than minFileSize, added to or removed from each partition from actions.

  7. def createStatsCollector(minFileSize: Long, errorReporter: (Throwable) => Unit): AutoCompactPartitionStatsCollector

    Helper class used to keep state regarding tracking auto-compaction stats of AddFile and RemoveFile actions in a single run that are greater than a passed-in minimum file size.

    Helper class used to keep state regarding tracking auto-compaction stats of AddFile and RemoveFile actions in a single run that are greater than a passed-in minimum file size. If the collector runs into any non-fatal errors, it will invoke the error reporter on the error and then skip further execution.

    minFileSize

    Minimum file size for files we track auto-compact stats

    errorReporter

    Function that reports the first error, if any

    returns

    A collector object that tracks the Add/Remove file actions of the current commit.

  8. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  9. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  10. def filterPartitionsWithSmallFiles(tableId: String, targetPartitions: Set[PartitionKey], minNumFiles: Long): Set[PartitionKey]

    returns

    Filter partitions from targetPartitions that have not been auto-compacted or that have enough small files.

  11. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  12. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  13. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  14. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  15. def markPartitionsAsCompacted(tableId: String, compactedPartitions: Set[PartitionKey]): Unit
  16. def maxNumFilesInTable(tableId: String): Long

    Get the maximum number of files among all partitions inside table tableId.

  17. def merge(tableId: String, inputPartitionFiles: PartitionFilesMap): Unit

    This method merges the inputPartitionFiles of current committed transaction to the global cache of table partition stats.

    This method merges the inputPartitionFiles of current committed transaction to the global cache of table partition stats. After merge is completed, tablePath will be moved to most recently used position. If the number of occupied partitions exceeds MAX_NUM_PARTITIONS, the least recently used tables will be evicted out.

    tableId

    The path of the table that contains inputPartitionFiles.

    inputPartitionFiles

    The number of files, which are qualified for Auto Compaction, in each partition.

  18. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  19. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  20. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  21. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  22. def toString(): String
    Definition Classes
    AnyRef → Any
  23. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  24. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  25. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()

Inherited from AnyRef

Inherited from Any

Ungrouped