Packages

c

org.apache.spark.sql.delta.commands

DeltaPurgeOperation

class DeltaPurgeOperation extends DeltaReorgOperation with ReorgTableHelper

Reorg operation to purge files with soft deleted rows. This operation will also try finding and removing the dropped columns from parquet files, if ever exists such column that does not present in the current table schema.

Linear Supertypes
ReorgTableHelper, Serializable, Serializable, DeltaReorgOperation, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DeltaPurgeOperation
  2. ReorgTableHelper
  3. Serializable
  4. Serializable
  5. DeltaReorgOperation
  6. AnyRef
  7. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new DeltaPurgeOperation()

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  8. def fileHasDifferentTypes(fileSchema: StructType, tablePhysicalSchema: StructType): Boolean

    Determine whether fileSchema has any column that has a type that differs from tablePhysicalSchema.

    Determine whether fileSchema has any column that has a type that differs from tablePhysicalSchema.

    fileSchema

    the current parquet schema to be checked.

    tablePhysicalSchema

    the current table schema.

    returns

    whether the file has any column that has a different type from table column.

    Attributes
    protected
    Definition Classes
    ReorgTableHelper
  9. def fileHasExtraColumns(fileSchema: StructType, tablePhysicalSchema: StructType, protocol: Protocol, metadata: Metadata): Boolean

    Determine whether fileSchema has any column that does not exist in the tablePhysicalSchema, this is possible by running ALTER TABLE commands, e.g., ALTER TABLE DROP COLUMN.

    Determine whether fileSchema has any column that does not exist in the tablePhysicalSchema, this is possible by running ALTER TABLE commands, e.g., ALTER TABLE DROP COLUMN.

    fileSchema

    the current parquet schema to be checked.

    tablePhysicalSchema

    the current table schema.

    protocol

    the protocol used to check row_id and row_commit_version.

    metadata

    the metadata used to check row_id and row_commit_version.

    returns

    whether the file has any dropped column.

    Attributes
    protected
    Definition Classes
    ReorgTableHelper
  10. def filterFilesToReorg(spark: SparkSession, snapshot: Snapshot, files: Seq[AddFile]): Seq[AddFile]

    Collects files that need to be processed by the reorg operation from the list of candidate files.

    Collects files that need to be processed by the reorg operation from the list of candidate files.

    Definition Classes
    DeltaPurgeOperation → DeltaReorgOperation
  11. def filterParquetFiles(files: Seq[AddFile], dataPath: Path, configuration: Configuration, ignoreCorruptFiles: Boolean, assumeBinaryIsString: Boolean, assumeInt96IsTimestamp: Boolean)(filterFileFn: (StructType) ⇒ Boolean): Seq[AddFile]
    Attributes
    protected
    Definition Classes
    ReorgTableHelper
  12. def filterParquetFilesOnExecutors(spark: SparkSession, files: Seq[AddFile], snapshot: Snapshot, ignoreCorruptFiles: Boolean)(filterFileFn: (StructType) ⇒ Boolean): Seq[AddFile]

    Apply a filter on the list of AddFile to only keep the files that have physical parquet schema that satisfies the given filter function.

    Apply a filter on the list of AddFile to only keep the files that have physical parquet schema that satisfies the given filter function.

    Note: Filtering happens on the executors: **any variable captured by filterFileFn must be Serializable**

    Attributes
    protected
    Definition Classes
    ReorgTableHelper
  13. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  14. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  15. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  16. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  17. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  18. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  19. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  20. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  21. def toString(): String
    Definition Classes
    AnyRef → Any
  22. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  23. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  24. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from ReorgTableHelper

Inherited from Serializable

Inherited from Serializable

Inherited from DeltaReorgOperation

Inherited from AnyRef

Inherited from Any

Ungrouped