SessionCatalog

Companion object SessionCatalog

class SessionCatalog extends SQLConfHelper with Logging

An internal catalog that is used by a Spark Session. This internal catalog serves as a proxy to the underlying metastore (e.g. Hive Metastore) and it also manages temporary views and functions of the Spark Session that it belongs to.

This class must be thread-safe.

Linear Supertypes

Logging, SQLConfHelper, AnyRef, Any

Ordering

Alphabetic
By Inheritance

Inherited

SessionCatalog
Logging
SQLConfHelper
AnyRef
Any

Hide All
Show All

Visibility

Public
All

Instance Constructors

new SessionCatalog(externalCatalog: ExternalCatalog)
new SessionCatalog(externalCatalog: ExternalCatalog, functionRegistry: FunctionRegistry)
new SessionCatalog(externalCatalog: ExternalCatalog, functionRegistry: FunctionRegistry, tableFunctionRegistry: TableFunctionRegistry)
new SessionCatalog(externalCatalog: ExternalCatalog, functionRegistry: FunctionRegistry, conf: SQLConf)
new SessionCatalog(externalCatalog: ExternalCatalog, functionRegistry: FunctionRegistry, tableFunctionRegistry: TableFunctionRegistry, conf: SQLConf)
new SessionCatalog(externalCatalogBuilder: () ⇒ ExternalCatalog, globalTempViewManagerBuilder: () ⇒ GlobalTempViewManager, functionRegistry: FunctionRegistry, tableFunctionRegistry: TableFunctionRegistry, hadoopConf: Configuration, parser: ParserInterface, functionResourceLoader: FunctionResourceLoader, functionExpressionBuilder: FunctionExpressionBuilder, cacheSize: Int = SQLConf.get.tableRelationCacheSize, cacheTTL: Long = SQLConf.get.metadataCacheTTL, defaultDatabase: String = SQLConf.get.defaultDatabase)

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def alterDatabase(dbDefinition: CatalogDatabase): Unit
def alterFunction(funcDefinition: CatalogFunction): Unit
overwrite a metastore function in the database specified in funcDefinition..
overwrite a metastore function in the database specified in funcDefinition.. If no database is specified, assume the function is in the current database.
def alterPartitions(tableName: TableIdentifier, parts: Seq[CatalogTablePartition]): Unit
Alter one or many table partitions whose specs that match those specified in parts, assuming the partitions exist.
Alter one or many table partitions whose specs that match those specified in parts, assuming the partitions exist.
If no database is specified, assume the table is in the current database.
Note: If the underlying implementation does not support altering a certain field, this becomes a no-op.
def alterTable(tableDefinition: CatalogTable): Unit
Alter the metadata of an existing metastore table identified by tableDefinition.
Alter the metadata of an existing metastore table identified by tableDefinition.
If no database is specified in tableDefinition, assume the table is in the current database.
Note: If the underlying implementation does not support altering a certain field, this becomes a no-op.
def alterTableDataSchema(identifier: TableIdentifier, newDataSchema: StructType): Unit
Alter the data schema of a table identified by the provided table identifier.
Alter the data schema of a table identified by the provided table identifier. The new data schema should not have conflict column names with the existing partition columns, and should still contain all the existing data columns.
identifier
TableIdentifier
newDataSchema
Updated data schema to be used for the table
def alterTableStats(identifier: TableIdentifier, newStats: Option[CatalogStatistics]): Unit
Alter Spark's statistics of an existing metastore table identified by the provided table identifier.
def alterTempViewDefinition(name: TableIdentifier, viewDefinition: TemporaryViewRelation): Boolean
Alter the definition of a local/global temp view matching the given name, returns true if a temp view is matched and altered, false otherwise.
final def asInstanceOf[T0]: T0

Definition Classes
Any
def cacheTable(t: QualifiedTableName, l: LogicalPlan): Unit
This method provides a way to cache a plan.
def clearTempTables(): Unit
Drop all existing temporary views.
Drop all existing temporary views. For testing only.
def clone(): AnyRef

Attributes
protected[lang]
Definition Classes
AnyRef
Annotations
@throws( ... ) @native()
def conf: SQLConf
The active config object within the current scope.
The active config object within the current scope. See SQLConf.get for more information.

Definition Classes
SQLConfHelper
def createDatabase(dbDefinition: CatalogDatabase, ignoreIfExists: Boolean): Unit
def createFunction(funcDefinition: CatalogFunction, ignoreIfExists: Boolean): Unit
Create a function in the database specified in funcDefinition.
Create a function in the database specified in funcDefinition. If no such database is specified, create it in the current database.
def createGlobalTempView(name: String, viewDefinition: TemporaryViewRelation, overrideIfExists: Boolean): Unit
Create a global temporary view.
def createPartitions(tableName: TableIdentifier, parts: Seq[CatalogTablePartition], ignoreIfExists: Boolean): Unit
Create partitions in an existing table, assuming it exists.
Create partitions in an existing table, assuming it exists. If no database is specified, assume the table is in the current database.
def createTable(tableDefinition: CatalogTable, ignoreIfExists: Boolean, validateLocation: Boolean = true): Unit
Create a metastore table in the database specified in tableDefinition.
Create a metastore table in the database specified in tableDefinition. If no such database is specified, create it in the current database.
def createTempView(name: String, viewDefinition: TemporaryViewRelation, overrideIfExists: Boolean): Unit
Create a local temporary view.
var currentDb: String

Attributes
protected
def databaseExists(db: String): Boolean
def defaultTablePath(tableIdent: TableIdentifier): URI
def dropDatabase(db: String, ignoreIfNotExists: Boolean, cascade: Boolean): Unit
def dropFunction(name: FunctionIdentifier, ignoreIfNotExists: Boolean): Unit
Drop a metastore function.
Drop a metastore function. If no database is specified, assume the function is in the current database.
def dropGlobalTempView(name: String): Boolean
Drop a global temporary view.
Drop a global temporary view.
Returns true if this view is dropped successfully, false otherwise.
def dropPartitions(tableName: TableIdentifier, specs: Seq[TablePartitionSpec], ignoreIfNotExists: Boolean, purge: Boolean, retainData: Boolean): Unit
Drop partitions from a table, assuming they exist.
Drop partitions from a table, assuming they exist. If no database is specified, assume the table is in the current database.
def dropTable(name: TableIdentifier, ignoreIfNotExists: Boolean, purge: Boolean): Unit
Drop a table.
Drop a table.
If a database is specified in name, this will drop the table from that database. If no database is specified, this will first attempt to drop a temporary view with the same name, then, if that does not exist, drop the table from the current database.
def dropTempFunction(name: String, ignoreIfNotExists: Boolean): Unit
Drop a temporary function.
def dropTempView(name: String): Boolean
Drop a local temporary view.
Drop a local temporary view.
Returns true if this view is dropped successfully, false otherwise.
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
lazy val externalCatalog: ExternalCatalog
def failFunctionLookup(name: FunctionIdentifier): Nothing

Attributes
protected[sql]
def finalize(): Unit

Attributes
protected[lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
def format(name: String): String
Formats object names, taking into account case sensitivity.
Formats object names, taking into account case sensitivity.

Attributes
protected
def functionExists(name: FunctionIdentifier): Boolean
Check if the function with the specified name exists
def getCachedPlan(t: QualifiedTableName, c: Callable[LogicalPlan]): LogicalPlan
This method provides a way to get a cached plan.
def getCachedTable(key: QualifiedTableName): LogicalPlan
This method provides a way to get a cached plan if the key exists.
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
Annotations
@native()
def getCurrentDatabase: String
def getDatabaseMetadata(db: String): CatalogDatabase
def getDefaultDBPath(db: String): URI
Get the path for creating a non-default database when database location is not provided by users.
def getFunctionMetadata(name: FunctionIdentifier): CatalogFunction
Retrieve the metadata of a metastore function.
Retrieve the metadata of a metastore function.
If a database is specified in name, this will return the function in that database. If no database is specified, this will return the function in the current database.
def getGlobalTempView(name: String): Option[View]
Generate a View operator from the global temporary view stored.
def getLocalOrGlobalTempView(name: TableIdentifier): Option[View]
Generate a View operator from the local or global temporary view stored.
def getPartition(tableName: TableIdentifier, spec: TablePartitionSpec): CatalogTablePartition
Retrieve the metadata of a table partition, assuming it exists.
Retrieve the metadata of a table partition, assuming it exists. If no database is specified, assume the table is in the current database.
def getRawGlobalTempView(name: String): Option[TemporaryViewRelation]
Return a global temporary view exactly as it was stored.
def getRawLocalOrGlobalTempView(name: Seq[String]): Option[TemporaryViewRelation]
Return the raw logical plan of a temporary local or global view for the given name.
def getRawTempView(name: String): Option[TemporaryViewRelation]
Return a local temporary view exactly as it was stored.
def getRelation(metadata: CatalogTable, options: CaseInsensitiveStringMap = CaseInsensitiveStringMap.empty()): LogicalPlan
def getTableMetadata(name: TableIdentifier): CatalogTable
Retrieve the metadata of an existing permanent table/view.
Retrieve the metadata of an existing permanent table/view. If no database is specified, assume the table/view is in the current database. We replace char/varchar with "annotated" string type in the table schema, as the query engine doesn't support char/varchar yet.

Annotations
@throws( ... ) @throws( ... )
def getTableRawMetadata(name: TableIdentifier): CatalogTable
Retrieve the metadata of an existing permanent table/view.
Retrieve the metadata of an existing permanent table/view. If no database is specified, assume the table/view is in the current database.

Annotations
@throws( ... ) @throws( ... )
def getTablesByName(names: Seq[TableIdentifier]): Seq[CatalogTable]
Retrieve all metadata of existing permanent tables/views.
Retrieve all metadata of existing permanent tables/views. If no database is specified, assume the table/view is in the current database. Only the tables/views belong to the same database that can be retrieved are returned. For example, if none of the requested tables could be retrieved, an empty list is returned. There is no guarantee of ordering of the returned tables.

Annotations
@throws( ... )
def getTempView(name: String): Option[View]
Generate a View operator from the temporary view stored.
def getTempViewNames(): Seq[String]
def getTempViewOrPermanentTableMetadata(name: TableIdentifier): CatalogTable
Retrieve the metadata of an existing temporary view or permanent table/view.
Retrieve the metadata of an existing temporary view or permanent table/view.
If a database is specified in name, this will return the metadata of table/view in that database. If no database is specified, this will first attempt to get the metadata of a temporary view with the same name, then, if that does not exist, return the metadata of table/view in the current database.
def getTempViewRelation(viewInfo: TemporaryViewRelation): SubqueryAlias
Generates a SubqueryAlias operator from the stored temporary view.
lazy val globalTempViewManager: GlobalTempViewManager
def hashCode(): Int

Definition Classes
AnyRef → Any
Annotations
@native()
def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

Attributes
protected
Definition Classes
Logging
def initializeLogIfNecessary(isInterpreter: Boolean): Unit

Attributes
protected
Definition Classes
Logging
def invalidateAllCachedTables(): Unit
This method provides a way to invalidate all the cached plans.
def invalidateCachedTable(name: TableIdentifier): Unit
This method discards any cached table relation plans for the given table identifier.
def invalidateCachedTable(key: QualifiedTableName): Unit
This method provides a way to invalidate a cached plan.
def isBuiltinFunction(name: FunctionIdentifier): Boolean
Returns whether it is a built-in function.
def isGlobalTempViewDB(dbName: String): Boolean
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
def isPersistentFunction(name: FunctionIdentifier): Boolean
Returns whether it is a persistent function.
Returns whether it is a persistent function. If not existed, returns false.
def isRegisteredFunction(name: FunctionIdentifier): Boolean
Return whether this function has been registered in the function registry of the current session.
Return whether this function has been registered in the function registry of the current session. If not existed, return false.
def isTempView(name: TableIdentifier): Boolean
Return whether a table with the specified name is a temporary view.
def isTempView(nameParts: Seq[String]): Boolean
Return whether the given name parts belong to a temporary or global temporary view.
def isTemporaryFunction(name: FunctionIdentifier): Boolean
Returns whether it is a temporary function.
Returns whether it is a temporary function. If not existed, returns false.
def isTraceEnabled(): Boolean

Attributes
protected
Definition Classes
Logging
def isView(nameParts: Seq[String]): Boolean
def listDatabases(pattern: String): Seq[String]
def listDatabases(): Seq[String]
def listFunctions(db: String, pattern: String): Seq[(FunctionIdentifier, String)]
List all matching functions in the specified database, including temporary functions.
List all matching functions in the specified database, including temporary functions. This returns the function identifier and the scope in which it was defined (system or user defined).
def listFunctions(db: String): Seq[(FunctionIdentifier, String)]
List all functions in the specified database, including temporary functions.
List all functions in the specified database, including temporary functions. This returns the function identifier and the scope in which it was defined (system or user defined).
def listLocalTempViews(pattern: String): Seq[TableIdentifier]
List all matching local temporary views.
def listPartitionNames(tableName: TableIdentifier, partialSpec: Option[TablePartitionSpec] = None): Seq[String]
List the names of all partitions that belong to the specified table, assuming it exists.
List the names of all partitions that belong to the specified table, assuming it exists.
A partial partition spec may optionally be provided to filter the partitions returned. For instance, if there exist partitions (a='1', b='2'), (a='1', b='3') and (a='2', b='4'), then a partial spec of (a='1') will return the first two only.
def listPartitions(tableName: TableIdentifier, partialSpec: Option[TablePartitionSpec] = None): Seq[CatalogTablePartition]
List the metadata of all partitions that belong to the specified table, assuming it exists.
List the metadata of all partitions that belong to the specified table, assuming it exists.
A partial partition spec may optionally be provided to filter the partitions returned. For instance, if there exist partitions (a='1', b='2'), (a='1', b='3') and (a='2', b='4'), then a partial spec of (a='1') will return the first two only.
def listPartitionsByFilter(tableName: TableIdentifier, predicates: Seq[Expression]): Seq[CatalogTablePartition]
List the metadata of partitions that belong to the specified table, assuming it exists, that satisfy the given partition-pruning predicate expressions.
def listTables(db: String, pattern: String, includeLocalTempViews: Boolean): Seq[TableIdentifier]
List all matching tables in the specified database, including local temporary views if includeLocalTempViews is enabled.
List all matching tables in the specified database, including local temporary views if includeLocalTempViews is enabled.
Note that, if the specified database is global temporary view database, we will list global temporary views.
def listTables(db: String, pattern: String): Seq[TableIdentifier]
List all matching tables in the specified database, including local temporary views.
List all matching tables in the specified database, including local temporary views.
Note that, if the specified database is global temporary view database, we will list global temporary views.
def listTables(db: String): Seq[TableIdentifier]
List all tables in the specified database, including local temporary views.
List all tables in the specified database, including local temporary views.
Note that, if the specified database is global temporary view database, we will list global temporary views.
def listTemporaryFunctions(): Seq[FunctionIdentifier]
List all temporary functions.
def listViews(db: String, pattern: String): Seq[TableIdentifier]
List all matching views in the specified database, including local temporary views.
def loadFunctionResources(resources: Seq[FunctionResource]): Unit
Loads resources such as JARs and Files for a function.
Loads resources such as JARs and Files for a function. Every resource is represented by a tuple (resource type, resource uri).
def loadPartition(name: TableIdentifier, loadPath: String, spec: TablePartitionSpec, isOverwrite: Boolean, inheritTableSpecs: Boolean, isSrcLocal: Boolean): Unit
Load files stored in given path into the partition of an existing metastore table.
Load files stored in given path into the partition of an existing metastore table. If no database is specified, assume the table is in the current database. If the specified table is not found in the database then a NoSuchTableException is thrown.
def loadTable(name: TableIdentifier, loadPath: String, isOverwrite: Boolean, isSrcLocal: Boolean): Unit
Load files stored in given path into an existing metastore table.
Load files stored in given path into an existing metastore table. If no database is specified, assume the table is in the current database. If the specified table is not found in the database then a NoSuchTableException is thrown.
def log: Logger

Attributes
protected
Definition Classes
Logging
def logDebug(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logDebug(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logError(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logError(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logInfo(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logInfo(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logName: String

Attributes
protected
Definition Classes
Logging
def logTrace(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logTrace(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logWarning(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logWarning(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def lookupBuiltinOrTempFunction(name: String): Option[ExpressionInfo]
Look up the ExpressionInfo of the given function by name if it's a built-in or temp function.
Look up the ExpressionInfo of the given function by name if it's a built-in or temp function. This only supports scalar functions.
def lookupBuiltinOrTempTableFunction(name: String): Option[ExpressionInfo]
Look up the ExpressionInfo of the given function by name if it's a built-in or temp table function.
def lookupFunction(name: FunctionIdentifier, children: Seq[Expression]): Expression
def lookupFunctionInfo(name: FunctionIdentifier): ExpressionInfo
Look up the ExpressionInfo associated with the specified function, assuming it exists.
def lookupPersistentFunction(name: FunctionIdentifier): ExpressionInfo
Look up the ExpressionInfo of the given function by name if it's a persistent function.
Look up the ExpressionInfo of the given function by name if it's a persistent function. This supports both scalar and table functions.
def lookupRelation(name: TableIdentifier): LogicalPlan
Return a LogicalPlan that represents the given table or view.
Return a LogicalPlan that represents the given table or view.
If a database is specified in name, this will return the table/view from that database. If no database is specified, this will first attempt to return a temporary view with the same name, then, if that does not exist, return the table/view from the current database.
Note that, the global temp view database is also valid here, this will return the global temp view matching the given name.
If the relation is a view, we generate a View operator from the view description, and wrap the logical plan in a SubqueryAlias which will track the name of the view. SubqueryAlias will also keep track of the name and database(optional) of the table/view
name
The name of the table/view that we look up.
def lookupTableFunction(name: FunctionIdentifier, children: Seq[Expression]): LogicalPlan
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
Annotations
@native()
final def notifyAll(): Unit

Definition Classes
AnyRef
Annotations
@native()
def qualifyIdentifier(ident: FunctionIdentifier): FunctionIdentifier
Qualifies the function identifier with the current database if not specified, and normalize all the names.
def qualifyIdentifier(ident: TableIdentifier): TableIdentifier
Qualifies the table identifier with the current database if not specified, and normalize all the names.
def refreshTable(name: TableIdentifier): Unit
Refresh table entries in structures maintained by the session catalog such as:
Refresh table entries in structures maintained by the session catalog such as:
- The map of temporary or global temporary view names to their logical plans
- The relation cache which maps table identifiers to their logical plans
For temp views, it refreshes their logical plans, and as a consequence of that it can refresh the file indexes of the base relations (HadoopFsRelation for instance) used in the views. The method still keeps the views in the internal lists of session catalog.
For tables/views, it removes their entries from the relation cache.
The method is supposed to use in the following situations:
1. The logical plan of a table/view was changed, and cached table/view data is cleared explicitly. For example, like in AlterTableRenameCommand which re-caches the table itself. Otherwise if you need to refresh cached data, consider using of CatalogImpl.refreshTable(). 2. A table/view doesn't exist, and need to only remove its entry in the relation cache since the cached data is invalidated explicitly like in DropTableCommand which uncaches table/view data itself. 3. Meta-data (such as file indexes) of any relation used in a temporary view should be updated.
def registerFunction(funcDefinition: CatalogFunction, overrideIfExists: Boolean, functionBuilder: Option[FunctionBuilder] = None): Unit
Registers a temporary or permanent scalar function into a session-specific FunctionRegistry
def renamePartitions(tableName: TableIdentifier, specs: Seq[TablePartitionSpec], newSpecs: Seq[TablePartitionSpec]): Unit
Override the specs of one or many existing table partitions, assuming they exist.
Override the specs of one or many existing table partitions, assuming they exist.
This assumes index i of specs corresponds to index i of newSpecs. If no database is specified, assume the table is in the current database.
def renameTable(oldName: TableIdentifier, newName: TableIdentifier): Unit
Rename a table.
Rename a table.
If a database is specified in oldName, this will rename the table in that database. If no database is specified, this will first attempt to rename a temporary view with the same name, then, if that does not exist, rename the table in the current database.
This assumes the database specified in newName matches the one in oldName.
def reset(): Unit
Drop all existing databases (except "default"), tables, partitions and functions, and set the current database to "default".
Drop all existing databases (except "default"), tables, partitions and functions, and set the current database to "default".
This is mainly used for tests.
def resolveBuiltinOrTempFunction(name: String, arguments: Seq[Expression]): Option[Expression]
Look up a built-in or temp scalar function by name and resolves it to an Expression if such a function exists.
def resolveBuiltinOrTempTableFunction(name: String, arguments: Seq[Expression]): Option[LogicalPlan]
Look up a built-in or temp table function by name and resolves it to a LogicalPlan if such a function exists.
def resolvePersistentFunction(name: FunctionIdentifier, arguments: Seq[Expression]): Expression
Look up a persistent scalar function by name and resolves it to an Expression.
def resolvePersistentTableFunction(name: FunctionIdentifier, arguments: Seq[Expression]): LogicalPlan
Look up a persistent table function by name and resolves it to a LogicalPlan.
def setCurrentDatabase(db: String): Unit
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def tableExists(name: TableIdentifier): Boolean
Return whether a table/view with the specified name exists.
Return whether a table/view with the specified name exists. If no database is specified, check with current database.
val tempViews: HashMap[String, TemporaryViewRelation]
List of temporary views, mapping from table name to their logical plan.
List of temporary views, mapping from table name to their logical plan.

Attributes
protected
def toString(): String

Definition Classes
AnyRef → Any
def unregisterFunction(name: FunctionIdentifier): Boolean
Unregister a temporary or permanent function from a session-specific FunctionRegistry or TableFunctionRegistry.
Unregister a temporary or permanent function from a session-specific FunctionRegistry or TableFunctionRegistry. Return true if function exists.
def validateTableLocation(table: CatalogTable): Unit
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... ) @native()

Packages

SessionCatalog

Companion object SessionCatalog

class SessionCatalog extends SQLConfHelper with Logging

Instance Constructors

Value Members

Inherited from Logging

Inherited from SQLConfHelper

Inherited from AnyRef

Inherited from Any

Ungrouped

Packages

SessionCatalog 

Companion object SessionCatalog

class SessionCatalog extends SQLConfHelper with Logging

Instance Constructors

Value Members

Inherited from Logging

Inherited from SQLConfHelper

Inherited from AnyRef

Inherited from Any

Ungrouped

SessionCatalog