Packages

  • package root
    Definition Classes
    root
  • package org
    Definition Classes
    root
  • package apache
    Definition Classes
    org
  • package spark
    Definition Classes
    apache
  • package sql
    Definition Classes
    spark
  • package delta
    Definition Classes
    sql
  • package stats
    Definition Classes
    delta
  • abstract class StatsCollector extends Serializable

    A helper class to collect stats of parquet data files for Delta table and its equivalent (tables that can be converted into Delta table like Parquet/Iceberg table).

    A helper class to collect stats of parquet data files for Delta table and its equivalent (tables that can be converted into Delta table like Parquet/Iceberg table).

    Definition Classes
    stats
  • StatsCollectionMetrics

case class StatsCollectionMetrics(numMissingMax: Long, numMissingMin: Long, numMissingNullCount: Long, numMissingTypes: Long) extends Product with Serializable

Used to report number of missing fields per supported type and number of missing unsupported types in the collected statistics, currently the statistics collection supports 4 types of stats: NUM_RECORDS, MAX, MIN, NULL_COUNT.

numMissingMax

The number of missing fields for MAX

numMissingMin

The number of missing fields for MIN

numMissingNullCount

The number of missing fields for NULL_COUNT

numMissingTypes

The number of unsupported type being requested.

Linear Supertypes
Serializable, Product, Equals, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. StatsCollectionMetrics
  2. Serializable
  3. Product
  4. Equals
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new StatsCollectionMetrics(numMissingMax: Long, numMissingMin: Long, numMissingNullCount: Long, numMissingTypes: Long)

    numMissingMax

    The number of missing fields for MAX

    numMissingMin

    The number of missing fields for MIN

    numMissingNullCount

    The number of missing fields for NULL_COUNT

    numMissingTypes

    The number of unsupported type being requested.

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  8. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  9. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  10. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  11. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  12. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  13. val numMissingMax: Long
  14. val numMissingMin: Long
  15. val numMissingNullCount: Long
  16. val numMissingTypes: Long
  17. def productElementNames: Iterator[String]
    Definition Classes
    Product
  18. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  19. val totalMissingFields: Long
  20. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  21. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  22. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from AnyRef

Inherited from Any

Ungrouped