Value Members

1.  final def !=(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

2.  final def ##: Int

   Definition Classes

   AnyRef → Any

3.  final def ==(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

4.  final def asInstanceOf[T0]: T0

   Definition Classes

   Any

5.  def clone(): AnyRef

   Attributes

   protected[lang]

   Definition Classes

   AnyRef

   Annotations

   @throws(classOf[java.lang.CloneNotSupportedException]) @native()

6.  final def eq(arg0: AnyRef): Boolean

   Definition Classes

   AnyRef

7.  def equals(arg0: AnyRef): Boolean

   Definition Classes

   AnyRef → Any

8.  def fileHasDifferentTypes(fileSchema: StructType, tablePhysicalSchema: StructType): Boolean

   Determine whether fileSchema has any column that has a type that differs from tablePhysicalSchema.

   Determine whether fileSchema has any column that has a type that differs from tablePhysicalSchema.

   fileSchema

   the current parquet schema to be checked.

   tablePhysicalSchema

   the current table schema.

   returns

   whether the file has any column that has a different type from table column.

   Attributes

   protected

   Definition Classes

   ReorgTableHelper

9.  def fileHasExtraColumns(fileSchema: StructType, tablePhysicalSchema: StructType, protocol: Protocol, metadata: Metadata): Boolean

   Determine whether fileSchema has any column that does not exist in the tablePhysicalSchema, this is possible by running ALTER TABLE commands, e.g., ALTER TABLE DROP COLUMN.

   Determine whether fileSchema has any column that does not exist in the tablePhysicalSchema, this is possible by running ALTER TABLE commands, e.g., ALTER TABLE DROP COLUMN.

   fileSchema

   the current parquet schema to be checked.

   tablePhysicalSchema

   the current table schema.

   protocol

   the protocol used to check row_id and row_commit_version.

   metadata

   the metadata used to check row_id and row_commit_version.

   returns

   whether the file has any dropped column.

   Attributes

   protected

   Definition Classes

   ReorgTableHelper

10.  def filterFilesToReorg(spark: SparkSession, snapshot: Snapshot, files: Seq[AddFile]): Seq[AddFile]

    Collects files that need to be processed by the reorg operation from the list of candidate files.

    Collects files that need to be processed by the reorg operation from the list of candidate files.

    Definition Classes

    DeltaRewriteTypeWideningOperation → DeltaReorgOperation

11.  def filterParquetFiles(files: Seq[AddFile], dataPath: Path, configuration: Configuration, ignoreCorruptFiles: Boolean, assumeBinaryIsString: Boolean, assumeInt96IsTimestamp: Boolean)(filterFileFn: (StructType) => Boolean): Seq[AddFile]

    Attributes

    protected

    Definition Classes

    ReorgTableHelper

12.  def filterParquetFilesOnExecutors(spark: SparkSession, files: Seq[AddFile], snapshot: Snapshot, ignoreCorruptFiles: Boolean)(filterFileFn: (StructType) => Boolean): Seq[AddFile]

    Apply a filter on the list of AddFile to only keep the files that have physical parquet schema that satisfies the given filter function.

    Apply a filter on the list of AddFile to only keep the files that have physical parquet schema that satisfies the given filter function.

    Note: Filtering happens on the executors: **any variable captured by filterFileFn must be Serializable**

    Attributes

    protected

    Definition Classes

    ReorgTableHelper

13.  def finalize(): Unit

    Attributes

    protected[lang]

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.Throwable])

14.  final def getClass(): Class[_ <: AnyRef]

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

15.  def hashCode(): Int

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

16.  final def isInstanceOf[T0]: Boolean

    Definition Classes

    Any

17.  final def ne(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

18.  final def notify(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

19.  final def notifyAll(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

20.  final def synchronized[T0](arg0: => T0): T0

    Definition Classes

    AnyRef

21.  def toString(): String

    Definition Classes

    AnyRef → Any

22.  final def wait(): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.InterruptedException])

23.  final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.InterruptedException])

24.  final def wait(arg0: Long): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.InterruptedException]) @native()