org.apache.spark.sql.catalyst.catalog

ExternalCatalog

abstract class ExternalCatalog extends ListenerBus[ExternalCatalogEventListener, ExternalCatalogEvent]

Interface for the system catalog (of functions, partitions, tables, and databases).

This is only used for non-temporary items, and implementations must be thread-safe as they can be accessed in multiple threads. This is an external catalog because it is expected to interact with external systems.

Implementations should throw NoSuchDatabaseException when databases don't exist.

Linear Supertypes
ListenerBus[ExternalCatalogEventListener, ExternalCatalogEvent], Logging, AnyRef, Any
Known Subclasses
InMemoryCatalog
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. ExternalCatalog
  2. ListenerBus
  3. Logging
  4. AnyRef
  5. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new ExternalCatalog()

Abstract Value Members

  1. abstract def alterDatabase(dbDefinition: CatalogDatabase): Unit

    Alter a database whose name matches the one specified in dbDefinition, assuming the database exists.

    Alter a database whose name matches the one specified in dbDefinition, assuming the database exists.

    Note: If the underlying implementation does not support altering a certain field, this becomes a no-op.

  2. abstract def alterPartitions(db: String, table: String, parts: Seq[CatalogTablePartition]): Unit

    Alter one or many table partitions whose specs that match those specified in parts, assuming the partitions exist.

    Alter one or many table partitions whose specs that match those specified in parts, assuming the partitions exist.

    Note: If the underlying implementation does not support altering a certain field, this becomes a no-op.

  3. abstract def alterTable(tableDefinition: CatalogTable): Unit

    Alter a table whose database and name match the ones specified in tableDefinition, assuming the table exists.

    Alter a table whose database and name match the ones specified in tableDefinition, assuming the table exists. Note that, even though we can specify database in tableDefinition, it's used to identify the table, not to alter the table's database, which is not allowed.

    Note: If the underlying implementation does not support altering a certain field, this becomes a no-op.

  4. abstract def alterTableDataSchema(db: String, table: String, newDataSchema: StructType): Unit

    Alter the data schema of a table identified by the provided database and table name.

    Alter the data schema of a table identified by the provided database and table name. The new data schema should not have conflict column names with the existing partition columns, and should still contain all the existing data columns.

    db

    Database that table to alter schema for exists in

    table

    Name of table to alter schema for

    newDataSchema

    Updated data schema to be used for the table.

  5. abstract def createPartitions(db: String, table: String, parts: Seq[CatalogTablePartition], ignoreIfExists: Boolean): Unit

  6. abstract def databaseExists(db: String): Boolean

  7. abstract def doCreateDatabase(dbDefinition: CatalogDatabase, ignoreIfExists: Boolean): Unit

    Attributes
    protected

  8. abstract def doCreateFunction(db: String, funcDefinition: CatalogFunction): Unit

    Attributes
    protected

  9. abstract def doCreateTable(tableDefinition: CatalogTable, ignoreIfExists: Boolean): Unit

    Attributes
    protected

  10. abstract def doDropDatabase(db: String, ignoreIfNotExists: Boolean, cascade: Boolean): Unit

    Attributes
    protected

  11. abstract def doDropFunction(db: String, funcName: String): Unit

    Attributes
    protected

  12. abstract def doDropTable(db: String, table: String, ignoreIfNotExists: Boolean, purge: Boolean): Unit

    Attributes
    protected

  13. abstract def doRenameFunction(db: String, oldName: String, newName: String): Unit

    Attributes
    protected

  14. abstract def doRenameTable(db: String, oldName: String, newName: String): Unit

    Attributes
    protected

  15. abstract def dropPartitions(db: String, table: String, parts: Seq[TablePartitionSpec], ignoreIfNotExists: Boolean, purge: Boolean, retainData: Boolean): Unit

  16. abstract def functionExists(db: String, funcName: String): Boolean

  17. abstract def getDatabase(db: String): CatalogDatabase

  18. abstract def getFunction(db: String, funcName: String): CatalogFunction

  19. abstract def getPartition(db: String, table: String, spec: TablePartitionSpec): CatalogTablePartition

  20. abstract def getPartitionOption(db: String, table: String, spec: TablePartitionSpec): Option[CatalogTablePartition]

    Returns the specified partition or None if it does not exist.

  21. abstract def getTable(db: String, table: String): CatalogTable

  22. abstract def listDatabases(pattern: String): Seq[String]

  23. abstract def listDatabases(): Seq[String]

  24. abstract def listFunctions(db: String, pattern: String): Seq[String]

  25. abstract def listPartitionNames(db: String, table: String, partialSpec: Option[TablePartitionSpec] = None): Seq[String]

    List the names of all partitions that belong to the specified table, assuming it exists.

    List the names of all partitions that belong to the specified table, assuming it exists.

    For a table with partition columns p1, p2, p3, each partition name is formatted as p1=v1/p2=v2/p3=v3. Each partition column name and value is an escaped path name, and can be decoded with the ExternalCatalogUtils.unescapePathName method.

    The returned sequence is sorted as strings.

    A partial partition spec may optionally be provided to filter the partitions returned, as described in the listPartitions method.

    db

    database name

    table

    table name

    partialSpec

    partition spec

  26. abstract def listPartitions(db: String, table: String, partialSpec: Option[TablePartitionSpec] = None): Seq[CatalogTablePartition]

    List the metadata of all partitions that belong to the specified table, assuming it exists.

    List the metadata of all partitions that belong to the specified table, assuming it exists.

    A partial partition spec may optionally be provided to filter the partitions returned. For instance, if there exist partitions (a='1', b='2'), (a='1', b='3') and (a='2', b='4'), then a partial spec of (a='1') will return the first two only.

    db

    database name

    table

    table name

    partialSpec

    partition spec

  27. abstract def listPartitionsByFilter(db: String, table: String, predicates: Seq[Expression], defaultTimeZoneId: String): Seq[CatalogTablePartition]

    List the metadata of partitions that belong to the specified table, assuming it exists, that satisfy the given partition-pruning predicate expressions.

    List the metadata of partitions that belong to the specified table, assuming it exists, that satisfy the given partition-pruning predicate expressions.

    db

    database name

    table

    table name

    predicates

    partition-pruning predicates

    defaultTimeZoneId

    default timezone id to parse partition values of TimestampType

  28. abstract def listTables(db: String, pattern: String): Seq[String]

  29. abstract def listTables(db: String): Seq[String]

  30. abstract def loadDynamicPartitions(db: String, table: String, loadPath: String, partition: TablePartitionSpec, replace: Boolean, numDP: Int): Unit

  31. abstract def loadPartition(db: String, table: String, loadPath: String, partition: TablePartitionSpec, isOverwrite: Boolean, inheritTableSpecs: Boolean, isSrcLocal: Boolean): Unit

    Loads data into a partition.

    Loads data into a partition.

    isSrcLocal

    Whether the source data is local, as defined by the "LOAD DATA LOCAL" HiveQL command.

  32. abstract def loadTable(db: String, table: String, loadPath: String, isOverwrite: Boolean, isSrcLocal: Boolean): Unit

    Loads data into a table.

    Loads data into a table.

    isSrcLocal

    Whether the source data is local, as defined by the "LOAD DATA LOCAL" HiveQL command.

  33. abstract def renamePartitions(db: String, table: String, specs: Seq[TablePartitionSpec], newSpecs: Seq[TablePartitionSpec]): Unit

    Override the specs of one or many existing table partitions, assuming they exist.

    Override the specs of one or many existing table partitions, assuming they exist. This assumes index i of specs corresponds to index i of newSpecs.

  34. abstract def setCurrentDatabase(db: String): Unit

  35. abstract def tableExists(db: String, table: String): Boolean

Concrete Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any

  3. final def ##(): Int

    Definition Classes
    AnyRef → Any

  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any

  6. final def addListener(listener: ExternalCatalogEventListener): Unit

    Definition Classes
    ListenerBus

  7. final def asInstanceOf[T0]: T0

    Definition Classes
    Any

  8. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  9. final def createDatabase(dbDefinition: CatalogDatabase, ignoreIfExists: Boolean): Unit

  10. final def createFunction(db: String, funcDefinition: CatalogFunction): Unit

  11. final def createTable(tableDefinition: CatalogTable, ignoreIfExists: Boolean): Unit

  12. def doPostEvent(listener: ExternalCatalogEventListener, event: ExternalCatalogEvent): Unit

    Attributes
    protected
    Definition Classes
    ExternalCatalog → ListenerBus

  13. final def dropDatabase(db: String, ignoreIfNotExists: Boolean, cascade: Boolean): Unit

  14. final def dropFunction(db: String, funcName: String): Unit

  15. final def dropTable(db: String, table: String, ignoreIfNotExists: Boolean, purge: Boolean): Unit

  16. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  17. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any

  18. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  19. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any

  20. def hashCode(): Int

    Definition Classes
    AnyRef → Any

  21. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Attributes
    protected
    Definition Classes
    Logging

  22. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any

  23. def isTraceEnabled(): Boolean

    Attributes
    protected
    Definition Classes
    Logging

  24. def log: Logger

    Attributes
    protected
    Definition Classes
    Logging

  25. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  26. def logDebug(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  27. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  28. def logError(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  29. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  30. def logInfo(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  31. def logName: String

    Attributes
    protected
    Definition Classes
    Logging

  32. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  33. def logTrace(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  34. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  35. def logWarning(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  36. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  37. final def notify(): Unit

    Definition Classes
    AnyRef

  38. final def notifyAll(): Unit

    Definition Classes
    AnyRef

  39. def postToAll(event: ExternalCatalogEvent): Unit

    Definition Classes
    ListenerBus

  40. final def removeListener(listener: ExternalCatalogEventListener): Unit

    Definition Classes
    ListenerBus

  41. final def renameFunction(db: String, oldName: String, newName: String): Unit

  42. final def renameTable(db: String, oldName: String, newName: String): Unit

  43. def requireDbExists(db: String): Unit

    Attributes
    protected

  44. def requireFunctionExists(db: String, funcName: String): Unit

    Attributes
    protected

  45. def requireFunctionNotExists(db: String, funcName: String): Unit

    Attributes
    protected

  46. def requireTableExists(db: String, table: String): Unit

    Attributes
    protected

  47. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef

  48. def toString(): String

    Definition Classes
    AnyRef → Any

  49. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  50. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  51. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from ListenerBus[ExternalCatalogEventListener, ExternalCatalogEvent]

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped