Packages

case class CatalogTable(identifier: TableIdentifier, tableType: CatalogTableType, storage: CatalogStorageFormat, schema: StructType, provider: Option[String] = None, partitionColumnNames: Seq[String] = Seq.empty, bucketSpec: Option[BucketSpec] = None, owner: String = "", createTime: Long = System.currentTimeMillis, lastAccessTime: Long = -1, createVersion: String = "", properties: Map[String, String] = Map.empty, stats: Option[CatalogStatistics] = None, viewText: Option[String] = None, comment: Option[String] = None, unsupportedFeatures: Seq[String] = Seq.empty, tracksPartitionsInCatalog: Boolean = false, schemaPreservesCase: Boolean = true, ignoredProperties: Map[String, String] = Map.empty, viewOriginalText: Option[String] = None) extends Product with Serializable

A table defined in the catalog.

Note that Hive's metastore also tracks skewed columns. We should consider adding that in the future once we have a better understanding of how we want to handle skewed columns.

provider

the name of the data source provider for this table, e.g. parquet, json, etc. Can be None if this table is a View, should be "hive" for hive serde tables.

createVersion

records the version of Spark that created this table metadata. The default is an empty string. We expect it will be read from the catalog or filled by ExternalCatalog.createTable. For temporary views, the value will be empty.

unsupportedFeatures

is a list of string descriptions of features that are used by the underlying table but not supported by Spark SQL yet.

tracksPartitionsInCatalog

whether this table's partition metadata is stored in the catalog. If false, it is inferred automatically based on file structure.

schemaPreservesCase

Whether or not the schema resolved for this table is case-sensitive. When using a Hive Metastore, this flag is set to false if a case- sensitive schema was unable to be read from the table properties. Used to trigger case-sensitive schema inference at query time, when configured.

ignoredProperties

is a list of table properties that are used by the underlying table but ignored by Spark SQL yet.

Linear Supertypes
Serializable, Product, Equals, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. CatalogTable
  2. Serializable
  3. Product
  4. Equals
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new CatalogTable(identifier: TableIdentifier, tableType: CatalogTableType, storage: CatalogStorageFormat, schema: StructType, provider: Option[String] = None, partitionColumnNames: Seq[String] = Seq.empty, bucketSpec: Option[BucketSpec] = None, owner: String = "", createTime: Long = System.currentTimeMillis, lastAccessTime: Long = -1, createVersion: String = "", properties: Map[String, String] = Map.empty, stats: Option[CatalogStatistics] = None, viewText: Option[String] = None, comment: Option[String] = None, unsupportedFeatures: Seq[String] = Seq.empty, tracksPartitionsInCatalog: Boolean = false, schemaPreservesCase: Boolean = true, ignoredProperties: Map[String, String] = Map.empty, viewOriginalText: Option[String] = None)

    provider

    the name of the data source provider for this table, e.g. parquet, json, etc. Can be None if this table is a View, should be "hive" for hive serde tables.

    createVersion

    records the version of Spark that created this table metadata. The default is an empty string. We expect it will be read from the catalog or filled by ExternalCatalog.createTable. For temporary views, the value will be empty.

    unsupportedFeatures

    is a list of string descriptions of features that are used by the underlying table but not supported by Spark SQL yet.

    tracksPartitionsInCatalog

    whether this table's partition metadata is stored in the catalog. If false, it is inferred automatically based on file structure.

    schemaPreservesCase

    Whether or not the schema resolved for this table is case-sensitive. When using a Hive Metastore, this flag is set to false if a case- sensitive schema was unable to be read from the table properties. Used to trigger case-sensitive schema inference at query time, when configured.

    ignoredProperties

    is a list of table properties that are used by the underlying table but ignored by Spark SQL yet.

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. val bucketSpec: Option[BucketSpec]
  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  7. val comment: Option[String]
  8. val createTime: Long
  9. val createVersion: String
  10. def dataSchema: StructType

    schema of this table's data columns

  11. def database: String

    Return the database this table was specified to belong to, assuming it exists.

  12. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  13. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  14. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  15. val identifier: TableIdentifier
  16. val ignoredProperties: Map[String, String]
  17. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  18. val lastAccessTime: Long
  19. def location: URI

    Return the table location, assuming it is specified.

  20. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  21. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  22. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  23. val owner: String
  24. val partitionColumnNames: Seq[String]
  25. def partitionSchema: StructType

    schema of this table's partition columns

  26. def productElementNames: Iterator[String]
    Definition Classes
    Product
  27. val properties: Map[String, String]
  28. val provider: Option[String]
  29. def qualifiedName: String

    Return the fully qualified name of this table, assuming the database was specified.

  30. val schema: StructType
  31. val schemaPreservesCase: Boolean
  32. def simpleString: String

    Readable string representation for the CatalogTable.

  33. val stats: Option[CatalogStatistics]
  34. val storage: CatalogStorageFormat
  35. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  36. val tableType: CatalogTableType
  37. def toLinkedHashMap: LinkedHashMap[String, String]
  38. def toString(): String
    Definition Classes
    CatalogTable → AnyRef → Any
  39. val tracksPartitionsInCatalog: Boolean
  40. val unsupportedFeatures: Seq[String]
  41. def viewCatalogAndNamespace: Seq[String]

    Return the current catalog and namespace (concatenated as a Seq[String]) of when the view was created.

  42. val viewOriginalText: Option[String]
  43. def viewQueryColumnNames: Seq[String]

    Return the output column names of the query that creates a view, the column names are used to resolve a view, should be empty if the CatalogTable is not a View or created by older versions of Spark(before 2.2.0).

  44. def viewReferredTempFunctionNames: Seq[String]

    Return temporary function names the current view was referred.

    Return temporary function names the current view was referred. should be empty if the CatalogTable is not a Temporary View or created by older versions of Spark(before 3.1.0).

  45. def viewReferredTempViewNames: Seq[Seq[String]]

    Return temporary view names the current view was referred.

    Return temporary view names the current view was referred. should be empty if the CatalogTable is not a Temporary View or created by older versions of Spark(before 3.1.0).

  46. def viewSQLConfigs: Map[String, String]

    Return the SQL configs of when the view was created, the configs are applied when parsing and analyzing the view, should be empty if the CatalogTable is not a View or created by older versions of Spark(before 3.1.0).

  47. val viewText: Option[String]
  48. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  49. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  50. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  51. def withNewStorage(locationUri: Option[URI] = storage.locationUri, inputFormat: Option[String] = storage.inputFormat, outputFormat: Option[String] = storage.outputFormat, compressed: Boolean = false, serde: Option[String] = storage.serde, properties: Map[String, String] = storage.properties): CatalogTable

    Syntactic sugar to update a field in storage.

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from AnyRef

Inherited from Any

Ungrouped