Value Members

1.  final def !=(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

2.  final def ##: Int

   Definition Classes

   AnyRef → Any

3.  final def ==(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

4.  final def asInstanceOf[T0]: T0

   Definition Classes

   Any

5.  def clone(): AnyRef

   Attributes

   protected[lang]

   Definition Classes

   AnyRef

   Annotations

   @throws(classOf[java.lang.CloneNotSupportedException]) @native()

6.  final def eq(arg0: AnyRef): Boolean

   Definition Classes

   AnyRef

7.  def equals(arg0: AnyRef): Boolean

   Definition Classes

   AnyRef → Any

8.  def fileHasDifferentTypes(fileSchema: StructType, tablePhysicalSchema: StructType): Boolean

   Determine whether fileSchema has any column that has a type that differs from tablePhysicalSchema.

   Determine whether fileSchema has any column that has a type that differs from tablePhysicalSchema.

   fileSchema

   the current parquet schema to be checked.

   tablePhysicalSchema

   the current table schema.

   returns

   whether the file has any column that has a different type from table column.

   Attributes

   protected

9.  def fileHasExtraColumns(fileSchema: StructType, tablePhysicalSchema: StructType, protocol: Protocol, metadata: Metadata): Boolean

   Determine whether fileSchema has any column that does not exist in the tablePhysicalSchema, this is possible by running ALTER TABLE commands, e.g., ALTER TABLE DROP COLUMN.

   Determine whether fileSchema has any column that does not exist in the tablePhysicalSchema, this is possible by running ALTER TABLE commands, e.g., ALTER TABLE DROP COLUMN.

   fileSchema

   the current parquet schema to be checked.

   tablePhysicalSchema

   the current table schema.

   protocol

   the protocol used to check row_id and row_commit_version.

   metadata

   the metadata used to check row_id and row_commit_version.

   returns

   whether the file has any dropped column.

   Attributes

   protected

10.  def filterParquetFiles(files: Seq[AddFile], dataPath: Path, configuration: Configuration, ignoreCorruptFiles: Boolean, assumeBinaryIsString: Boolean, assumeInt96IsTimestamp: Boolean)(filterFileFn: (StructType) => Boolean): Seq[AddFile]

    Attributes

    protected

11.  def filterParquetFilesOnExecutors(spark: SparkSession, files: Seq[AddFile], snapshot: Snapshot, ignoreCorruptFiles: Boolean)(filterFileFn: (StructType) => Boolean): Seq[AddFile]

    Apply a filter on the list of AddFile to only keep the files that have physical parquet schema that satisfies the given filter function.

    Apply a filter on the list of AddFile to only keep the files that have physical parquet schema that satisfies the given filter function.

    Note: Filtering happens on the executors: **any variable captured by filterFileFn must be Serializable**

    Attributes

    protected

12.  def finalize(): Unit

    Attributes

    protected[lang]

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.Throwable])

13.  final def getClass(): Class[_ <: AnyRef]

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

14.  def hashCode(): Int

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

15.  final def isInstanceOf[T0]: Boolean

    Definition Classes

    Any

16.  final def ne(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

17.  final def notify(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

18.  final def notifyAll(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

19.  final def synchronized[T0](arg0: => T0): T0

    Definition Classes

    AnyRef

20.  def toString(): String

    Definition Classes

    AnyRef → Any

21.  final def wait(): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.InterruptedException])

22.  final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.InterruptedException])

23.  final def wait(arg0: Long): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.InterruptedException]) @native()