Packages

o

org.apache.spark.sql.delta.uniform

ParquetIcebergCompatV2Utils

object ParquetIcebergCompatV2Utils

Contains utilities to check whether a specific parquet data file is considered IcebergCompatV2. See isParquetIcebergV2Compatible for details.

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ParquetIcebergCompatV2Utils
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  9. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  10. def getParquetFooter(parquetPath: String): ParquetMetadata

    Get the parquet footer based on the input parquetPath.

    Get the parquet footer based on the input parquetPath.

    parquetPath

    the absolute path to the parquet file.

    returns

    the corresponding parquet metadata/footer.

  11. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  13. def isParquetIcebergCompatV2(footer: ParquetMetadata): Boolean

    Check if the parquet file is IcebergCompatV2 by inspecting the provided parquet footer.

    Check if the parquet file is IcebergCompatV2 by inspecting the provided parquet footer.

    note: icebergV2-compatible check refer to the following two properties. 1. TIMESTAMP

    • If TIMESTAMP is stored as int96, it's considered incompatible since iceberg stores TIMESTAMP as int64 according to the iceberg spec. 2. field_id
    • field_id is needed for *every* field in a parquet footer, this includes the field of each column, and the potential nested fields for nested types like LIST, MAP or STRUCT. See https://github.com/apache/parquet-format/blob/master/LogicalTypes.md for details.
    • This is checked by inspecting whether the field_id for each column is null or not recursively.
    footer

    the parquet footer to be checked.

    returns

    whether the parquet file is considered IcebergCompatV2.

  14. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  15. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  16. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  17. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  18. def toString(): String
    Definition Classes
    AnyRef → Any
  19. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  20. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  21. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()

Inherited from AnyRef

Inherited from Any

Ungrouped