org.apache.hadoop.hive.metastore

Class Warehouse

java.lang.Object

org.apache.hadoop.hive.metastore.Warehouse

public class Warehouse
extends Object

This class represents a warehouse where data of Hive tables is stored

Field Summary

Fields

Modifier and Type	Field and Description
static String	DATABASE_WAREHOUSE_SUFFIX
static String	DEFAULT_CATALOG_COMMENT
static String	DEFAULT_CATALOG_NAME
static String	DEFAULT_DATABASE_COMMENT
static String	DEFAULT_DATABASE_NAME
static String	DEFAULT_SERIALIZATION_FORMAT
static org.slf4j.Logger	LOG

Constructor Summary

Constructors

Constructor and Description
Warehouse(org.apache.hadoop.conf.Configuration conf)

Method Summary

Modifier and Type	Method and Description
boolean	copyDir(org.apache.hadoop.fs.Path sourcePath, org.apache.hadoop.fs.Path destPath, boolean needCmRecycle)
boolean	deleteDir(org.apache.hadoop.fs.Path f, boolean recursive, boolean ifPurge, boolean needCmRecycle)
boolean	deleteDir(org.apache.hadoop.fs.Path f, boolean recursive, boolean ifPurge, Database db)
boolean	deleteDir(org.apache.hadoop.fs.Path f, boolean recursive, Database db)
org.apache.hadoop.fs.Path	determineDatabasePath(Catalog cat, Database db) Build the database path based on catalog name and database name.
static String	escapePathName(String path)
static String	getCatalogQualifiedTableName(Table table) Get table name in cat.db.table format.
org.apache.hadoop.fs.Path	getDatabaseExternalPath(Database db) Get the external tables path specified by the database.
org.apache.hadoop.fs.Path	getDatabaseManagedPath(Database db) Get the managed tables path specified by the database.
org.apache.hadoop.fs.Path	getDatabasePath(Database db) Get the managed tables path specified by the database.
org.apache.hadoop.fs.Path	getDefaultDatabasePath(String dbName)
org.apache.hadoop.fs.Path	getDefaultDatabasePath(String dbName, boolean inExternalWH)
org.apache.hadoop.fs.Path	getDefaultExternalDatabasePath(String dbName)
org.apache.hadoop.fs.Path	getDefaultManagedTablePath(Database db, String tableName)
org.apache.hadoop.fs.Path	getDefaultPartitionPath(Database db, Table table, Map<String,String> pm) Returns the default partition path of a table within a given database and partition key value pairs.
org.apache.hadoop.fs.Path	getDefaultTablePath(Database db, String tableName) Deprecated.
org.apache.hadoop.fs.Path	getDefaultTablePath(Database db, String tableName, boolean isExternal)
org.apache.hadoop.fs.Path	getDefaultTablePath(Database db, Table table)
org.apache.hadoop.fs.Path	getDefaultTablePath(String dbName, String tableName, boolean isExternal)
org.apache.hadoop.fs.Path	getDnsPath(org.apache.hadoop.fs.Path path)
static org.apache.hadoop.fs.Path	getDnsPath(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf) Hadoop File System reverse lookups paths with raw ip addresses The File System URI always contains the canonical DNS name of the Namenode.
List<org.apache.hadoop.fs.FileStatus>	getFileStatusesForLocation(String location)
List<org.apache.hadoop.fs.FileStatus>	getFileStatusesForSD(StorageDescriptor desc)
List<org.apache.hadoop.fs.FileStatus>	getFileStatusesForUnpartitionedTable(Database db, Table table)
org.apache.hadoop.fs.FileSystem	getFs(org.apache.hadoop.fs.Path f)
static org.apache.hadoop.fs.FileSystem	getFs(org.apache.hadoop.fs.Path f, org.apache.hadoop.conf.Configuration conf) Helper functions to convert IOException to MetaException
org.apache.hadoop.fs.Path	getPartitionPath(Database db, Table table, List<String> vals) Given a database, a table and the partition key value pairs this method returns the Path object corresponding to the partition key value pairs.
org.apache.hadoop.fs.Path	getPartitionPath(org.apache.hadoop.fs.Path tblPath, Map<String,String> pm) Returns the path object for the given partition key-value pairs and the base location
static List<String>	getPartValuesFromPartName(String partName)
static String	getQualifiedName(Partition partition)
static String	getQualifiedName(String dbName, String tableName) Deprecated.
static String	getQualifiedName(Table table) Deprecated.
org.apache.hadoop.fs.Path	getWhRoot() Resolve the configured warehouse root dir with respect to the configuration This involves opening the FileSystem corresponding to the warehouse root dir (but that should be ok given that this is only called during DDL statements for non-external tables).
org.apache.hadoop.fs.Path	getWhRootExternal()
boolean	isDir(org.apache.hadoop.fs.Path f)
boolean	isEmptyDir(org.apache.hadoop.fs.Path path)
boolean	isEmptyDir(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.PathFilter pathFilter)
boolean	isWritable(org.apache.hadoop.fs.Path path)
static String	makeDynamicPartName(Map<String,String> spec) Given a dynamic partition specification, return the path corresponding to the static part of partition specification.
static String	makeDynamicPartNameNoTrailingSeperator(Map<String,String> spec) Given a dynamic partition specification, return the path corresponding to the static part of partition specification.
static Map<String,String>	makeEscSpecFromName(String name)
static String	makePartName(List<FieldSchema> partCols, List<String> vals)
static String	makePartName(List<FieldSchema> partCols, List<String> vals, String defaultStr) Makes a valid partition name.
static String	makePartName(Map<String,String> spec, boolean addTrailingSeperator) Makes a partition name from a specification
static String	makePartNameUtil(Map<String,String> spec, boolean addTrailingSeperator, boolean dynamic) Makes a partition name from a specification
static String	makePartPath(Map<String,String> spec) Given a partition specification, return the path corresponding to the partition spec.
static boolean	makeSpecFromName(Map<String,String> partSpec, org.apache.hadoop.fs.Path currPath, Set<String> requiredKeys)
static LinkedHashMap<String,String>	makeSpecFromName(String name)
static Map<String,String>	makeSpecFromValues(List<FieldSchema> partCols, List<String> values)
static AbstractList<String>	makeValsFromName(String name, AbstractList<String> result) Extracts values from partition name without the column names.
boolean	mkdirs(org.apache.hadoop.fs.Path f)
void	recycleDirToCmPath(org.apache.hadoop.fs.Path f, boolean ifPurge)
boolean	renameDir(org.apache.hadoop.fs.Path sourcePath, org.apache.hadoop.fs.Path destPath, boolean needCmRecycle)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_CATALOG_NAME

public static final String DEFAULT_CATALOG_NAME

See Also:

Constant Field Values

DEFAULT_CATALOG_COMMENT

public static final String DEFAULT_CATALOG_COMMENT

See Also:

Constant Field Values

DEFAULT_DATABASE_NAME

public static final String DEFAULT_DATABASE_NAME

See Also:

Constant Field Values

DEFAULT_DATABASE_COMMENT

public static final String DEFAULT_DATABASE_COMMENT

See Also:

Constant Field Values

DEFAULT_SERIALIZATION_FORMAT

public static final String DEFAULT_SERIALIZATION_FORMAT

See Also:

Constant Field Values

DATABASE_WAREHOUSE_SUFFIX

public static final String DATABASE_WAREHOUSE_SUFFIX

See Also:

Constant Field Values

LOG

public static final org.slf4j.Logger LOG

Constructor Detail

Warehouse

public Warehouse(org.apache.hadoop.conf.Configuration conf)
          throws MetaException

Throws:

MetaException

Method Detail

getFs

public static org.apache.hadoop.fs.FileSystem getFs(org.apache.hadoop.fs.Path f,
                                                    org.apache.hadoop.conf.Configuration conf)
                                             throws MetaException

Helper functions to convert IOException to MetaException

Throws:

MetaException

getFs

public org.apache.hadoop.fs.FileSystem getFs(org.apache.hadoop.fs.Path f)
                                      throws MetaException

Throws:

MetaException

getDnsPath

public static org.apache.hadoop.fs.Path getDnsPath(org.apache.hadoop.fs.Path path,
                                                   org.apache.hadoop.conf.Configuration conf)
                                            throws MetaException

Hadoop File System reverse lookups paths with raw ip addresses The File System URI always contains the canonical DNS name of the Namenode. Subsequently, operations on paths with raw ip addresses cause an exception since they don't match the file system URI. This routine solves this problem by replacing the scheme and authority of a path with the scheme and authority of the FileSystem that it maps to. Since creating a new file system object is expensive, this method mimics getFileSystem() without creating an actual FileSystem object. When the input path lacks a scheme or an authority this is added from the default URI.

Parameters:

path - Path to be canonicalized

Returns:

Path with canonical scheme and authority

Throws:

MetaException

getDnsPath

public org.apache.hadoop.fs.Path getDnsPath(org.apache.hadoop.fs.Path path)
                                     throws MetaException

Throws:

MetaException

getWhRoot

public org.apache.hadoop.fs.Path getWhRoot()
                                    throws MetaException

Resolve the configured warehouse root dir with respect to the configuration This involves opening the FileSystem corresponding to the warehouse root dir (but that should be ok given that this is only called during DDL statements for non-external tables).

Throws:

MetaException

getWhRootExternal

public org.apache.hadoop.fs.Path getWhRootExternal()
                                            throws MetaException

Throws:

MetaException

determineDatabasePath

public org.apache.hadoop.fs.Path determineDatabasePath(Catalog cat,
                                                       Database db)
                                                throws MetaException

Build the database path based on catalog name and database name. This should only be used when a database is being created or altered. If you just want to find out the path a database is already using call getDatabasePath(Database). If the passed in database already has a path set that will be used. If not the location will be built using catalog's path and the database name.

Parameters:

cat - catalog the database is in

db - database object

Returns:

Path representing the directory for the database

Throws:

MetaException - when the file path cannot be properly determined from the configured file system.

getDatabasePath

public org.apache.hadoop.fs.Path getDatabasePath(Database db)
                                          throws MetaException

Get the managed tables path specified by the database. In the case of the default database the root of the warehouse is returned.

Parameters:

db - database to get the path of

Returns:

path to the database directory

Throws:

MetaException - when the file path cannot be properly determined from the configured file system.

getDatabaseExternalPath

public org.apache.hadoop.fs.Path getDatabaseExternalPath(Database db)
                                                  throws MetaException

Get the external tables path specified by the database. In the case of the default database the root of the warehouse is returned.

Parameters:

db - database to get the path of

Returns:

path to the database directory

Throws:

MetaException - when the file path cannot be properly determined from the configured file system.

getDatabaseManagedPath

public org.apache.hadoop.fs.Path getDatabaseManagedPath(Database db)
                                                 throws MetaException

Get the managed tables path specified by the database. In the case of the default database the root of the warehouse is returned.

Parameters:

db - database to get the path of

Returns:

path to the database directory

Throws:

MetaException - when the file path cannot be properly determined from the configured file system.

getDefaultDatabasePath

public org.apache.hadoop.fs.Path getDefaultDatabasePath(String dbName)
                                                 throws MetaException

Throws:

MetaException

getDefaultExternalDatabasePath

public org.apache.hadoop.fs.Path getDefaultExternalDatabasePath(String dbName)
                                                         throws MetaException

Throws:

MetaException

getDefaultDatabasePath

public org.apache.hadoop.fs.Path getDefaultDatabasePath(String dbName,
                                                        boolean inExternalWH)
                                                 throws MetaException

Throws:

MetaException

getDefaultTablePath

@Deprecated
public org.apache.hadoop.fs.Path getDefaultTablePath(Database db,
                                                                 String tableName)
                                                          throws MetaException

Deprecated.

Returns the default location of the table path using the parent database's location

Parameters:

db - Database where the table is created

tableName - table name

Returns:

Throws:

MetaException

getDefaultTablePath

public org.apache.hadoop.fs.Path getDefaultTablePath(Database db,
                                                     String tableName,
                                                     boolean isExternal)
                                              throws MetaException

Throws:

MetaException

getDefaultManagedTablePath

public org.apache.hadoop.fs.Path getDefaultManagedTablePath(Database db,
                                                            String tableName)
                                                     throws MetaException

Throws:

MetaException

getDefaultTablePath

public org.apache.hadoop.fs.Path getDefaultTablePath(String dbName,
                                                     String tableName,
                                                     boolean isExternal)
                                              throws MetaException

Throws:

MetaException

getDefaultTablePath

public org.apache.hadoop.fs.Path getDefaultTablePath(Database db,
                                                     Table table)
                                              throws MetaException

Throws:

MetaException

getQualifiedName

@Deprecated
public static String getQualifiedName(Table table)

Deprecated.

getQualifiedName

@Deprecated
public static String getQualifiedName(String dbName,
                                                  String tableName)

Deprecated.

getQualifiedName

public static String getQualifiedName(Partition partition)

getCatalogQualifiedTableName

public static String getCatalogQualifiedTableName(Table table)

Get table name in cat.db.table format.

Parameters:

table - table object

Returns:

fully qualified name.

mkdirs

public boolean mkdirs(org.apache.hadoop.fs.Path f)
               throws MetaException

Throws:

MetaException

renameDir

public boolean renameDir(org.apache.hadoop.fs.Path sourcePath,
                         org.apache.hadoop.fs.Path destPath,
                         boolean needCmRecycle)
                  throws MetaException

Throws:

MetaException

copyDir

public boolean copyDir(org.apache.hadoop.fs.Path sourcePath,
                       org.apache.hadoop.fs.Path destPath,
                       boolean needCmRecycle)
                throws MetaException

Throws:

MetaException

deleteDir

public boolean deleteDir(org.apache.hadoop.fs.Path f,
                         boolean recursive,
                         Database db)
                  throws MetaException

Throws:

MetaException

deleteDir

public boolean deleteDir(org.apache.hadoop.fs.Path f,
                         boolean recursive,
                         boolean ifPurge,
                         Database db)
                  throws MetaException

Throws:

MetaException

deleteDir

public boolean deleteDir(org.apache.hadoop.fs.Path f,
                         boolean recursive,
                         boolean ifPurge,
                         boolean needCmRecycle)
                  throws MetaException

Throws:

MetaException

recycleDirToCmPath

public void recycleDirToCmPath(org.apache.hadoop.fs.Path f,
                               boolean ifPurge)
                        throws MetaException

Throws:

MetaException

isEmptyDir

public boolean isEmptyDir(org.apache.hadoop.fs.Path path)
                   throws IOException,
                          MetaException

Throws:

IOException

MetaException

isEmptyDir

public boolean isEmptyDir(org.apache.hadoop.fs.Path path,
                          org.apache.hadoop.fs.PathFilter pathFilter)
                   throws IOException,
                          MetaException

Throws:

IOException

MetaException

isWritable

public boolean isWritable(org.apache.hadoop.fs.Path path)
                   throws IOException

Throws:

IOException

escapePathName

public static String escapePathName(String path)

makePartPath

public static String makePartPath(Map<String,String> spec)
                           throws MetaException

Given a partition specification, return the path corresponding to the partition spec. By default, the specification does not include dynamic partitions.

Parameters:

spec -

Returns:

string representation of the partition specification.

Throws:

MetaException

makePartNameUtil

public static String makePartNameUtil(Map<String,String> spec,
                                      boolean addTrailingSeperator,
                                      boolean dynamic)
                               throws MetaException

Makes a partition name from a specification

Parameters:

spec - The partition specification, key and value pairs.

addTrailingSeperator - If true, adds a trailing separator e.g. 'ds=1/'.

dynamic - If true, create a dynamic partition name.

Returns:

partition name

Throws:

MetaException

makePartName

public static String makePartName(Map<String,String> spec,
                                  boolean addTrailingSeperator)
                           throws MetaException

Makes a partition name from a specification

Parameters:

spec -

addTrailingSeperator - if true, adds a trailing separator e.g. 'ds=1/'

Returns:

partition name

Throws:

MetaException

makeDynamicPartName

public static String makeDynamicPartName(Map<String,String> spec)

Given a dynamic partition specification, return the path corresponding to the static part of partition specification. This is basically similar to makePartName but we get rid of MetaException since it is not serializable.

Parameters:

spec -

Returns:

string representation of the static part of the partition specification.

makeDynamicPartNameNoTrailingSeperator

public static String makeDynamicPartNameNoTrailingSeperator(Map<String,String> spec)

Given a dynamic partition specification, return the path corresponding to the static part of partition specification. This is basically similar to makePartName but we get rid of MetaException since it is not serializable. This method skips the trailing path seperator also.

Parameters:

spec -

Returns:

string representation of the static part of the partition specification.

makeValsFromName

public static AbstractList<String> makeValsFromName(String name,
                                                    AbstractList<String> result)
                                             throws MetaException

Extracts values from partition name without the column names.

Parameters:

name - Partition name.

result - The result. Must be pre-sized to the expected number of columns.

Throws:

MetaException

makeSpecFromName

public static LinkedHashMap<String,String> makeSpecFromName(String name)
                                                     throws MetaException

Throws:

MetaException

makeSpecFromName

public static boolean makeSpecFromName(Map<String,String> partSpec,
                                       org.apache.hadoop.fs.Path currPath,
                                       Set<String> requiredKeys)

makeEscSpecFromName

public static Map<String,String> makeEscSpecFromName(String name)
                                              throws MetaException

Throws:

MetaException

getDefaultPartitionPath

public org.apache.hadoop.fs.Path getDefaultPartitionPath(Database db,
                                                         Table table,
                                                         Map<String,String> pm)
                                                  throws MetaException

Returns the default partition path of a table within a given database and partition key value pairs. It uses the database location and appends it the table name and the partition key,value pairs to create the Path for the partition directory

Parameters:

db - - parent database which is used to get the base location of the partition directory

table - - table for the partitions

pm - - Partition key value pairs

Returns:

Throws:

MetaException

getPartitionPath

public org.apache.hadoop.fs.Path getPartitionPath(org.apache.hadoop.fs.Path tblPath,
                                                  Map<String,String> pm)
                                           throws MetaException

Returns the path object for the given partition key-value pairs and the base location

Parameters:

tblPath - - the base location for the partitions. Typically the table location

pm - - Partition key value pairs

Returns:

Throws:

MetaException

getPartitionPath

public org.apache.hadoop.fs.Path getPartitionPath(Database db,
                                                  Table table,
                                                  List<String> vals)
                                           throws MetaException

Given a database, a table and the partition key value pairs this method returns the Path object corresponding to the partition key value pairs. It uses the table location if available else uses the database location for constructing the path corresponding to the partition key-value pairs

Parameters:

db - - Parent database of the given table

table - - Table for which the partition key-values are given

vals - - List of values for the partition keys

Returns:

Path corresponding to the partition key-value pairs

Throws:

MetaException

isDir

public boolean isDir(org.apache.hadoop.fs.Path f)
              throws MetaException

Throws:

MetaException

makePartName

public static String makePartName(List<FieldSchema> partCols,
                                  List<String> vals)
                           throws MetaException

Throws:

MetaException

getFileStatusesForSD

public List<org.apache.hadoop.fs.FileStatus> getFileStatusesForSD(StorageDescriptor desc)
                                                           throws MetaException

Parameters:

desc -

Returns:

array of FileStatus objects corresponding to the files making up the passed storage description

Throws:

MetaException

getFileStatusesForLocation

public List<org.apache.hadoop.fs.FileStatus> getFileStatusesForLocation(String location)
                                                                 throws MetaException

Parameters:

location -

Returns:

array of FileStatus objects corresponding to the files making up the passed storage description

Throws:

MetaException

getFileStatusesForUnpartitionedTable

public List<org.apache.hadoop.fs.FileStatus> getFileStatusesForUnpartitionedTable(Database db,
                                                                                  Table table)
                                                                           throws MetaException

Parameters:

db - database

table - table

Returns:

array of FileStatus objects corresponding to the files making up the passed unpartitioned table

Throws:

MetaException

makePartName

public static String makePartName(List<FieldSchema> partCols,
                                  List<String> vals,
                                  String defaultStr)
                           throws MetaException

Makes a valid partition name.

Parameters:

partCols - The partition columns

vals - The partition values

defaultStr - The default name given to a partition value if the respective value is empty or null.

Returns:

An escaped, valid partition name.

Throws:

MetaException

getPartValuesFromPartName

public static List<String> getPartValuesFromPartName(String partName)
                                              throws MetaException

Throws:

MetaException

makeSpecFromValues

public static Map<String,String> makeSpecFromValues(List<FieldSchema> partCols,
                                                    List<String> values)