Packages

  • package root
    Definition Classes
    root
  • package org
    Definition Classes
    root
  • package apache
    Definition Classes
    org
  • package spark
    Definition Classes
    apache
  • package sql
    Definition Classes
    spark
  • package delta
    Definition Classes
    sql
  • package stats
    Definition Classes
    delta
  • class AutoCompactPartitionStats extends AnyRef

    This singleton object collect the table partition statistic for each commit that creates AddFile or RemoveFile objects.

    This singleton object collect the table partition statistic for each commit that creates AddFile or RemoveFile objects. To control the memory usage, there are maxNumTablePartitions per table and 'maxNumPartitions' partition entries across all tables. Note:

    1. Since the partition of each table is limited, if this limitation is reached, the least recently used table partitions will be evicted. 2. If all 'maxNumPartitions' are occupied, the partition stats of least recently used tables will be evicted until the used partitions fall back below to 'maxNumPartitions'. 3. The un-partitioned tables are treated as tables with single partition.
    Definition Classes
    stats
  • PartitionStat

class PartitionStat extends AnyRef

This class to store the states of one table partition. These state includes: -- the number of small files, -- the thread that assigned to compact this partition, and -- whether the partition was compacted.

Note: Since this class keeps tracking of the statistics of the table partition and the state of the auto compaction thread that works on the table partition, any method that accesses any attribute of this class needs to be protected by synchronized context.

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. PartitionStat
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new PartitionStat(numFiles: Long, wasAutoCompacted: Boolean = false)

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  9. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  10. def hasSufficientFiles(minNumFiles: Long): Boolean
  11. def hasSufficientSmallFilesOrHasNotBeenCompacted(minNumFiles: Long): Boolean

    Determine whether this partition can be autocompacted based on the number of small files or if this AutoCompactPartitionStats instance has not auto compacted it yet.

    Determine whether this partition can be autocompacted based on the number of small files or if this AutoCompactPartitionStats instance has not auto compacted it yet.

    minNumFiles

    The minimum number of files this table-partition should have to trigger Auto Compaction in case it has already been compacted once.

  12. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  13. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  14. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  15. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  16. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  17. var numFiles: Long
  18. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  19. def toString(): String
    Definition Classes
    AnyRef → Any
  20. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  21. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  22. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  23. var wasAutoCompacted: Boolean

Inherited from AnyRef

Inherited from Any

Ungrouped