Packages

class CatalogImpl extends Catalog

Internal implementation of the user-facing Catalog.

Linear Supertypes
Catalog, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. CatalogImpl
  2. Catalog
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new CatalogImpl(sparkSession: SparkSession)

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def cacheTable(tableName: String, storageLevel: StorageLevel): Unit

    Caches the specified table or view with the given storage level.

    Caches the specified table or view with the given storage level.

    tableName

    is either a qualified or unqualified name that designates a table/view. If no database identifier is provided, it refers to a temporary view or a table/view in the current database.

    storageLevel

    storage level to cache table.

    Definition Classes
    CatalogImpl → Catalog
    Since

    2.3.0

  6. def cacheTable(tableName: String): Unit

    Caches the specified table or view in-memory.

    Caches the specified table or view in-memory.

    tableName

    is either a qualified or unqualified name that designates a table/view. If no database identifier is provided, it refers to a temporary view or a table/view in the current database.

    Definition Classes
    CatalogImpl → Catalog
    Since

    2.0.0

  7. def clearCache(): Unit

    Removes all cached tables or views from the in-memory cache.

    Removes all cached tables or views from the in-memory cache.

    Definition Classes
    CatalogImpl → Catalog
    Since

    2.0.0

  8. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  9. def createTable(tableName: String, source: String, schema: StructType, description: String, options: Map[String, String]): DataFrame

    (Scala-specific) Creates a table based on the dataset in a data source, a schema and a set of options.

    (Scala-specific) Creates a table based on the dataset in a data source, a schema and a set of options. Then, returns the corresponding DataFrame.

    tableName

    is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.

    Definition Classes
    CatalogImpl → Catalog
    Since

    3.1.0

  10. def createTable(tableName: String, source: String, schema: StructType, options: Map[String, String]): DataFrame

    (Scala-specific) Creates a table based on the dataset in a data source, a schema and a set of options.

    (Scala-specific) Creates a table based on the dataset in a data source, a schema and a set of options. Then, returns the corresponding DataFrame.

    tableName

    is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.

    Definition Classes
    CatalogImpl → Catalog
    Since

    2.2.0

  11. def createTable(tableName: String, source: String, description: String, options: Map[String, String]): DataFrame

    (Scala-specific) Creates a table based on the dataset in a data source and a set of options.

    (Scala-specific) Creates a table based on the dataset in a data source and a set of options. Then, returns the corresponding DataFrame.

    tableName

    is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.

    Definition Classes
    CatalogImpl → Catalog
    Since

    3.1.0

  12. def createTable(tableName: String, source: String, options: Map[String, String]): DataFrame

    (Scala-specific) Creates a table based on the dataset in a data source and a set of options.

    (Scala-specific) Creates a table based on the dataset in a data source and a set of options. Then, returns the corresponding DataFrame.

    tableName

    is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.

    Definition Classes
    CatalogImpl → Catalog
    Since

    2.2.0

  13. def createTable(tableName: String, path: String, source: String): DataFrame

    Creates a table from the given path and returns the corresponding DataFrame.

    Creates a table from the given path and returns the corresponding DataFrame.

    tableName

    is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.

    Definition Classes
    CatalogImpl → Catalog
    Since

    2.2.0

  14. def createTable(tableName: String, path: String): DataFrame

    Creates a table from the given path and returns the corresponding DataFrame.

    Creates a table from the given path and returns the corresponding DataFrame. It will use the default data source configured by spark.sql.sources.default.

    tableName

    is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.

    Definition Classes
    CatalogImpl → Catalog
    Since

    2.2.0

  15. def createTable(tableName: String, source: String, schema: StructType, description: String, options: Map[String, String]): DataFrame

    Create a table based on the dataset in a data source, a schema and a set of options.

    Create a table based on the dataset in a data source, a schema and a set of options. Then, returns the corresponding DataFrame.

    tableName

    is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.

    Definition Classes
    Catalog
    Since

    3.1.0

  16. def createTable(tableName: String, source: String, schema: StructType, options: Map[String, String]): DataFrame

    Create a table based on the dataset in a data source, a schema and a set of options.

    Create a table based on the dataset in a data source, a schema and a set of options. Then, returns the corresponding DataFrame.

    tableName

    is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.

    Definition Classes
    Catalog
    Since

    2.2.0

  17. def createTable(tableName: String, source: String, description: String, options: Map[String, String]): DataFrame

    Creates a table based on the dataset in a data source and a set of options.

    Creates a table based on the dataset in a data source and a set of options. Then, returns the corresponding DataFrame.

    tableName

    is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.

    Definition Classes
    Catalog
    Since

    3.1.0

  18. def createTable(tableName: String, source: String, options: Map[String, String]): DataFrame

    Creates a table based on the dataset in a data source and a set of options.

    Creates a table based on the dataset in a data source and a set of options. Then, returns the corresponding DataFrame.

    tableName

    is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.

    Definition Classes
    Catalog
    Since

    2.2.0

  19. def currentDatabase: String

    Returns the current default database in this session.

    Returns the current default database in this session.

    Definition Classes
    CatalogImpl → Catalog
  20. def databaseExists(dbName: String): Boolean

    Checks if the database with the specified name exists.

    Checks if the database with the specified name exists.

    Definition Classes
    CatalogImpl → Catalog
  21. def dropGlobalTempView(viewName: String): Boolean

    Drops the global temporary view with the given view name in the catalog.

    Drops the global temporary view with the given view name in the catalog. If the view has been cached/persisted before, it's also unpersisted.

    viewName

    the identifier of the global temporary view to be dropped.

    returns

    true if the view is dropped successfully, false otherwise.

    Definition Classes
    CatalogImpl → Catalog
    Since

    2.1.0

  22. def dropTempView(viewName: String): Boolean

    Drops the local temporary view with the given view name in the catalog.

    Drops the local temporary view with the given view name in the catalog. If the view has been cached/persisted before, it's also unpersisted.

    viewName

    the identifier of the temporary view to be dropped.

    returns

    true if the view is dropped successfully, false otherwise.

    Definition Classes
    CatalogImpl → Catalog
    Since

    2.0.0

  23. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  24. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  25. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  26. def functionExists(dbName: String, functionName: String): Boolean

    Checks if the function with the specified name exists in the specified database.

    Checks if the function with the specified name exists in the specified database.

    dbName

    is a name that designates a database.

    functionName

    is an unqualified name that designates a function.

    Definition Classes
    CatalogImpl → Catalog
  27. def functionExists(functionName: String): Boolean

    Checks if the function with the specified name exists.

    Checks if the function with the specified name exists. This can either be a temporary function or a function.

    functionName

    is either a qualified or unqualified name that designates a function. If no database identifier is provided, it refers to a function in the current database.

    Definition Classes
    CatalogImpl → Catalog
  28. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  29. def getDatabase(dbName: String): Database

    Gets the database with the specified name.

    Gets the database with the specified name. This throws an AnalysisException when no Database can be found.

    Definition Classes
    CatalogImpl → Catalog
  30. def getFunction(dbName: String, functionName: String): Function

    Gets the function with the specified name.

    Gets the function with the specified name. This returns None when no Function can be found.

    dbName

    is a name that designates a database.

    functionName

    is an unqualified name that designates a function in the specified database

    Definition Classes
    CatalogImpl → Catalog
  31. def getFunction(functionName: String): Function

    Gets the function with the specified name.

    Gets the function with the specified name. This function can be a temporary function or a function. This throws an AnalysisException when no Function can be found.

    functionName

    is either a qualified or unqualified name that designates a function. If no database identifier is provided, it refers to a temporary function or a function in the current database.

    Definition Classes
    CatalogImpl → Catalog
  32. def getTable(dbName: String, tableName: String): Table

    Gets the table or view with the specified name in the specified database.

    Gets the table or view with the specified name in the specified database. This throws an AnalysisException when no Table can be found.

    Definition Classes
    CatalogImpl → Catalog
  33. def getTable(tableName: String): Table

    Gets the table or view with the specified name.

    Gets the table or view with the specified name. This table can be a temporary view or a table/view. This throws an AnalysisException when no Table can be found.

    tableName

    is either a qualified or unqualified name that designates a table/view. If no database identifier is provided, it refers to a table/view in the current database.

    Definition Classes
    CatalogImpl → Catalog
  34. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  35. def isCached(qName: Dataset[_]): Boolean

    Returns true if the Dataset is currently cached in-memory.

    Returns true if the Dataset is currently cached in-memory.

    Attributes
    protected[sql]
    Since

    2.0.0

  36. def isCached(tableName: String): Boolean

    Returns true if the table or view is currently cached in-memory.

    Returns true if the table or view is currently cached in-memory.

    tableName

    is either a qualified or unqualified name that designates a table/view. If no database identifier is provided, it refers to a temporary view or a table/view in the current database.

    Definition Classes
    CatalogImpl → Catalog
    Since

    2.0.0

  37. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  38. def listColumns(dbName: String, tableName: String): Dataset[catalog.Column]

    Returns a list of columns for the given table/view or temporary view in the specified database.

    Returns a list of columns for the given table/view or temporary view in the specified database.

    dbName

    is a name that designates a database.

    tableName

    is an unqualified name that designates a table/view.

    Definition Classes
    CatalogImpl → Catalog
    Annotations
    @throws("database or table does not exist")
  39. def listColumns(tableName: String): Dataset[catalog.Column]

    Returns a list of columns for the given table/view or temporary view.

    Returns a list of columns for the given table/view or temporary view.

    tableName

    is either a qualified or unqualified name that designates a table/view. If no database identifier is provided, it refers to a temporary view or a table/view in the current database.

    Definition Classes
    CatalogImpl → Catalog
    Annotations
    @throws("table does not exist")
  40. def listDatabases(): Dataset[Database]

    Returns a list of databases available across all sessions.

    Returns a list of databases available across all sessions.

    Definition Classes
    CatalogImpl → Catalog
  41. def listFunctions(dbName: String): Dataset[Function]

    Returns a list of functions registered in the specified database.

    Returns a list of functions registered in the specified database. This includes all temporary functions

    Definition Classes
    CatalogImpl → Catalog
    Annotations
    @throws("database does not exist")
  42. def listFunctions(): Dataset[Function]

    Returns a list of functions registered in the current database.

    Returns a list of functions registered in the current database. This includes all temporary functions

    Definition Classes
    CatalogImpl → Catalog
  43. def listTables(dbName: String): Dataset[Table]

    Returns a list of tables in the specified database.

    Returns a list of tables in the specified database. This includes all temporary tables.

    Definition Classes
    CatalogImpl → Catalog
    Annotations
    @throws("database does not exist")
  44. def listTables(): Dataset[Table]

    Returns a list of tables in the current database.

    Returns a list of tables in the current database. This includes all temporary tables.

    Definition Classes
    CatalogImpl → Catalog
  45. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  46. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  47. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  48. def recoverPartitions(tableName: String): Unit

    Recovers all the partitions in the directory of a table and update the catalog.

    Recovers all the partitions in the directory of a table and update the catalog. Only works with a partitioned table, and not a temporary view.

    tableName

    is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.

    Definition Classes
    CatalogImpl → Catalog
    Since

    2.1.1

  49. def refreshByPath(resourcePath: String): Unit

    Refreshes the cache entry and the associated metadata for all Dataset (if any), that contain the given data source path.

    Refreshes the cache entry and the associated metadata for all Dataset (if any), that contain the given data source path. Path matching is by prefix, i.e. "/" would invalidate everything that is cached.

    Definition Classes
    CatalogImpl → Catalog
    Since

    2.0.0

  50. def refreshTable(tableName: String): Unit

    The method fully refreshes a table or view with the given name including:

    The method fully refreshes a table or view with the given name including:

    1. The relation cache in the session catalog. The method removes table entry from the cache. 2. The file indexes of all relations used by the given view. 3. Table/View schema in the Hive Metastore if the SQL config spark.sql.hive.caseSensitiveInferenceMode is set to INFER_AND_SAVE. 4. Cached data of the given table or view, and all its dependents that refer to it. Existing cached data will be cleared and the cache will be lazily filled when the next time the table/view or the dependents are accessed.

    The method does not do:

    • schema inference for file source tables
    • statistics update

    The method is supposed to be used in all cases when need to refresh table/view data and meta-data.

    tableName

    is either a qualified or unqualified name that designates a table/view. If no database identifier is provided, it refers to a temporary view or a table/view in the current database.

    Definition Classes
    CatalogImpl → Catalog
    Since

    2.0.0

  51. def setCurrentDatabase(dbName: String): Unit

    Sets the current default database in this session.

    Sets the current default database in this session.

    Definition Classes
    CatalogImpl → Catalog
    Annotations
    @throws("database does not exist")
  52. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  53. def tableExists(dbName: String, tableName: String): Boolean

    Checks if the table or view with the specified name exists in the specified database.

    Checks if the table or view with the specified name exists in the specified database.

    dbName

    is a name that designates a database.

    tableName

    is an unqualified name that designates a table.

    Definition Classes
    CatalogImpl → Catalog
  54. def tableExists(tableName: String): Boolean

    Checks if the table or view with the specified name exists.

    Checks if the table or view with the specified name exists. This can either be a temporary view or a table/view.

    tableName

    is either a qualified or unqualified name that designates a table/view. If no database identifier is provided, it refers to a table/view in the current database.

    Definition Classes
    CatalogImpl → Catalog
  55. def toString(): String
    Definition Classes
    AnyRef → Any
  56. def uncacheTable(tableName: String): Unit

    Removes the specified table or view from the in-memory cache.

    Removes the specified table or view from the in-memory cache.

    tableName

    is either a qualified or unqualified name that designates a table/view. If no database identifier is provided, it refers to a temporary view or a table/view in the current database.

    Definition Classes
    CatalogImpl → Catalog
    Since

    2.0.0

  57. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  58. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  59. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()

Deprecated Value Members

  1. def createExternalTable(tableName: String, source: String, schema: StructType, options: Map[String, String]): DataFrame

    (Scala-specific) Create a table from the given path based on a data source, a schema and a set of options.

    (Scala-specific) Create a table from the given path based on a data source, a schema and a set of options. Then, returns the corresponding DataFrame.

    tableName

    is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.

    Definition Classes
    Catalog
    Annotations
    @deprecated
    Deprecated

    (Since version 2.2.0) use createTable instead.

    Since

    2.0.0

  2. def createExternalTable(tableName: String, source: String, schema: StructType, options: Map[String, String]): DataFrame

    Create a table from the given path based on a data source, a schema and a set of options.

    Create a table from the given path based on a data source, a schema and a set of options. Then, returns the corresponding DataFrame.

    tableName

    is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.

    Definition Classes
    Catalog
    Annotations
    @deprecated
    Deprecated

    (Since version 2.2.0) use createTable instead.

    Since

    2.0.0

  3. def createExternalTable(tableName: String, source: String, options: Map[String, String]): DataFrame

    (Scala-specific) Creates a table from the given path based on a data source and a set of options.

    (Scala-specific) Creates a table from the given path based on a data source and a set of options. Then, returns the corresponding DataFrame.

    tableName

    is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.

    Definition Classes
    Catalog
    Annotations
    @deprecated
    Deprecated

    (Since version 2.2.0) use createTable instead.

    Since

    2.0.0

  4. def createExternalTable(tableName: String, source: String, options: Map[String, String]): DataFrame

    Creates a table from the given path based on a data source and a set of options.

    Creates a table from the given path based on a data source and a set of options. Then, returns the corresponding DataFrame.

    tableName

    is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.

    Definition Classes
    Catalog
    Annotations
    @deprecated
    Deprecated

    (Since version 2.2.0) use createTable instead.

    Since

    2.0.0

  5. def createExternalTable(tableName: String, path: String, source: String): DataFrame

    Creates a table from the given path based on a data source and returns the corresponding DataFrame.

    Creates a table from the given path based on a data source and returns the corresponding DataFrame.

    tableName

    is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.

    Definition Classes
    Catalog
    Annotations
    @deprecated
    Deprecated

    (Since version 2.2.0) use createTable instead.

    Since

    2.0.0

  6. def createExternalTable(tableName: String, path: String): DataFrame

    Creates a table from the given path and returns the corresponding DataFrame.

    Creates a table from the given path and returns the corresponding DataFrame. It will use the default data source configured by spark.sql.sources.default.

    tableName

    is either a qualified or unqualified name that designates a table. If no database identifier is provided, it refers to a table in the current database.

    Definition Classes
    Catalog
    Annotations
    @deprecated
    Deprecated

    (Since version 2.2.0) use createTable instead.

    Since

    2.0.0

Inherited from Catalog

Inherited from AnyRef

Inherited from Any

cachemgmt

ddl_ops

Ungrouped