org.apache.hadoop.hive.metastore.utils

Class MetaStoreUtils

java.lang.Object

org.apache.hadoop.hive.metastore.utils.MetaStoreUtils

public class MetaStoreUtils
extends Object

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	MetaStoreUtils.FailoverEndpoint

Field Summary

Fields

Modifier and Type	Field and Description
static int	CAT_NAME Position in the array returned by parseDbName(java.lang.String, org.apache.hadoop.conf.Configuration) that has the catalog name.
static String	CATALOG_DB_SEPARATOR This String is used to separate the catalog name from the database name.
static char	CATALOG_DB_THRIFT_NAME_MARKER This character is used to mark a database name as having a catalog name prepended.
static String	DB_EMPTY_MARKER Mark a database as being empty (as distinct from null).
static int	DB_NAME Position in the array returned by parseDbName(java.lang.String, org.apache.hadoop.conf.Configuration) that has the database name.
static String	EXTERNAL_TABLE_PURGE
static String	NO_VAL
static String	TYPE_FROM_DESERIALIZER
static String	USER_NAME_HTTP_HEADER

Constructor Summary

Constructors

Constructor and Description
MetaStoreUtils()

Method Summary

Modifier and Type	Method and Description
static void	addPartitonSpecsToList(PartitionsSpecByExprResult r, List<PartitionSpec> result)
static ClassLoader	addToClassPath(ClassLoader cloader, String[] newPaths) Add new elements to the classpath.
static boolean	checkIfDbNeedsToBeSkipped(Database db)
static boolean	compareFieldColumns(List<FieldSchema> schema1, List<FieldSchema> schema2)
static List<Predicate<String>>	compilePatternsToPredicates(List<String> patterns) Compile a list of regex patterns and collect them as Predicates.
static String	convertDateToString(Date date) Converts java.sql.Date to String format date.
static Date	convertStringToDate(String date) Converts string format date to java.sql.Date.
static Timestamp	convertStringToTimestamp(String timestamp) Converts timestamp string format to java.sql.Timestamp.
static String	convertTimestampToString(Timestamp timestamp) Converts java.sql.Timestamp to string format timestamp.
static GetPartitionsByNamesRequest	convertToGetPartitionsByNamesRequest(String dbName, String tblName, List<String> partNames)
static GetPartitionsByNamesRequest	convertToGetPartitionsByNamesRequest(String dbName, String tblName, List<String> partNames, boolean getColStats, String engine, String validWriteIdList, Long tableId)
static <T> T	createThriftPartitionsReq(Class<T> clazz, org.apache.hadoop.conf.Configuration conf, T... deepCopy)
static String	encodeTableName(String name)
static <T> void	filterMapkeys(Map<String,T> map, List<Predicate<String>> predicates) filters a given map with list of predicates.
static <T> void	filterMapKeys(Map<String,T> map, Predicate<String> predicate) filters a given map with predicate provided.
static int	getArchivingLevel(Partition part)
static String	getColumnCommentsFromFieldSchema(List<FieldSchema> fieldSchemas)
static String	getColumnNameDelimiter(List<FieldSchema> fieldSchemas)
static List<String>	getColumnNames(List<FieldSchema> schema)
static List<String>	getColumnNamesForTable(Table table)
static String	getColumnNamesFromFieldSchema(List<FieldSchema> fieldSchemas) Convert FieldSchemas to columnNames.
static String	getColumnTypesFromFieldSchema(List<FieldSchema> fieldSchemas)
static String	getColumnTypesFromFieldSchema(List<FieldSchema> fieldSchemas, String delimiter) Convert FieldSchemas to columnTypes.
static String	getDbNameFromReplPolicy(String replPolicy)
static String	getDDLFromFieldSchema(String structName, List<FieldSchema> fieldSchemas) Convert FieldSchemas to Thrift DDL.
static String	getDefaultCatalog(org.apache.hadoop.conf.Configuration conf)
static String	getHostFromId(String id)
static String	getHttpPath(String httpPath) The config parameter can be like "path", "/path", "/path/", "path/*", "/path1/path2/*" and so on.
static Map<String,String>	getMetaStoreSaslProperties(org.apache.hadoop.conf.Configuration conf, boolean useSSL) Read and return the meta store Sasl configuration.
static String	getNoAutoCompact(Map<String,String> parameters) Get no_auto_compact property by checking in both lower and upper cases
static org.apache.hadoop.fs.Path	getOriginalLocation(Partition part)
static Properties	getPartitionMetadata(Partition partition, Table table)
static Properties	getPartSchemaFromTableSchema(StorageDescriptor sd, Map<String,String> parameters, Properties tblSchema) Get partition level schema from table level schema.
static List<String>	getPvals(List<FieldSchema> partCols, Map<String,String> partSpec) Given a list of partition columns and a partial mapping from some partition columns to values the function returns the values for the column.
static List<String>	getReplicationDbProps()
static Properties	getSchema(Partition part, Table table)
static Properties	getSchema(StorageDescriptor sd, StorageDescriptor tblsd, Map<String,String> parameters, String databaseName, String tableName, List<FieldSchema> partitionKeys)
static Properties	getSchemaWithoutCols(StorageDescriptor sd, Map<String,String> parameters, String databaseName, String tableName, List<FieldSchema> partitionKeys)
static Properties	getTableMetadata(Table table)
static TableName	getTableNameFor(Table table)
static String	getThreadIdFromId(String id)
static boolean	hasUnknownPartitions(PartitionsSpecByExprResult r)
static boolean	isArchived(Partition part)
static boolean	isBackgroundThreadsEnabledForRepl(Database db)
static boolean	isDbBeingPlannedFailedOver(Database db)
static boolean	isDbBeingPlannedFailedOverAtEndpoint(Database db, MetaStoreUtils.FailoverEndpoint endPoint)
static boolean	isDbReplIncompatible(Database db)
static boolean	isExternal(Map<String,String> tableParams)
static boolean	isExternalTable(Table table) Determines whether a table is an external table.
static boolean	isExternalTablePurge(Table table) Determines whether an table needs to be purged or not.
static boolean	isIcebergTable(Map<String,String> params)
static boolean	isInsertOnlyTableParam(Map<String,String> params) Duplicates AcidUtils; used in a couple places in metastore.
static boolean	isMaterializedViewTable(Table table)
static boolean	isNoAutoCompactSet(Map<String,String> dbParameters, Map<String,String> tblParameters) Because TABLE_NO_AUTO_COMPACT was originally assumed to be NO_AUTO_COMPACT and then was moved to no_auto_compact, we need to check it in both cases.
static boolean	isNoCleanUpSet(Map<String,String> parameters)
static boolean	isNonNativeTable(Map<String,String> tblProps)
static boolean	isNonNativeTable(Table table)
static boolean	isPropertyTrue(Map<String,String> tableParams, String prop)
static boolean	isSkipTrash(Map<String,String> tableParams) Determines whether an table needs to be deleted completely or moved to trash directory.
static boolean	isTargetOfReplication(Database db)
static boolean	isTranslatedToExternalTable(Table table)
static boolean	isValidSchedulingPolicy(String str)
static boolean	isView(Table table)
static List<Object[]>	makeOrderSpecs(String order) Get order specs from a represented string.
static String	makePartNameMatcher(Table table, List<String> partVals, String defaultStr)
static MetaException	newMetaException(Exception e) convert Exception to MetaException, which sets the cause to such exception
static MetaException	newMetaException(String errorMessage, Exception e) convert Exception to MetaException, which sets the cause to such exception
static String	normalizeDate(String date) Converts the string format date without a time-zone to a time-zone based string format date
static String[]	parseDbName(String dbName, org.apache.hadoop.conf.Configuration conf) Parse the catalog name out of the database name.
static WMPoolSchedulingPolicy	parseSchedulingPolicy(String schedulingPolicy)
static String	prependCatalogToDbName(String dbName, org.apache.hadoop.conf.Configuration conf) Prepend the default 'hive' catalog onto the database name.
static String	prependCatalogToDbName(String catalogName, String dbName, org.apache.hadoop.conf.Configuration conf) Given a catalog name and database name cram them together into one string.
static String	prependNotNullCatToDbName(String catalogName, String dbName) Given a catalog name and database name, cram them together into one string.
static void	throwMetaException(Exception e) Catches exceptions that cannot be handled and wraps them in MetaException.
static boolean	validateName(String name, org.apache.hadoop.conf.Configuration conf) validateName Checks the name conforms to our standars which are: "[a-zA-z_0-9]+".
static boolean	validateTblStorage(StorageDescriptor sd)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TYPE_FROM_DESERIALIZER

public static final String TYPE_FROM_DESERIALIZER

See Also:

Constant Field Values

CATALOG_DB_THRIFT_NAME_MARKER

public static final char CATALOG_DB_THRIFT_NAME_MARKER

This character is used to mark a database name as having a catalog name prepended. This marker should be placed first in the String to make it easy to determine that this has both a catalog and a database name. @ is chosen as it is not used in regular expressions. This is only intended for use when making old Thrift calls that do not support catalog names.

See Also:

Constant Field Values

CATALOG_DB_SEPARATOR

public static final String CATALOG_DB_SEPARATOR

This String is used to separate the catalog name from the database name. This should only be used in Strings that are prepended with CATALOG_DB_THRIFT_NAME_MARKER. # is chosen because it is not used in regular expressions. this is only intended for use when making old Thrift calls that do not support catalog names.

See Also:

Constant Field Values

DB_EMPTY_MARKER

public static final String DB_EMPTY_MARKER

Mark a database as being empty (as distinct from null).

See Also:

Constant Field Values

EXTERNAL_TABLE_PURGE

public static final String EXTERNAL_TABLE_PURGE

See Also:

Constant Field Values

NO_VAL

public static final String NO_VAL

See Also:

Constant Field Values

USER_NAME_HTTP_HEADER

public static final String USER_NAME_HTTP_HEADER

See Also:

Constant Field Values

CAT_NAME

public static final int CAT_NAME

Position in the array returned by parseDbName(java.lang.String, org.apache.hadoop.conf.Configuration) that has the catalog name.

See Also:

Constant Field Values

DB_NAME

public static final int DB_NAME

Position in the array returned by parseDbName(java.lang.String, org.apache.hadoop.conf.Configuration) that has the database name.

See Also:

Constant Field Values

Constructor Detail

MetaStoreUtils

public MetaStoreUtils()

Method Detail

convertDateToString

public static String convertDateToString(Date date)

Converts java.sql.Date to String format date.

Parameters:

date - - java.sql.Date object.

Returns:

Date in string format.

convertStringToDate

public static Date convertStringToDate(String date)

Converts string format date to java.sql.Date.

Parameters:

date - Date in string format.

Returns:

java.sql.Date object.

normalizeDate

public static String normalizeDate(String date)

Converts the string format date without a time-zone to a time-zone based string format date

Parameters:

date - the date without a time-zone

Returns:

time-zone based string format date

convertTimestampToString

public static String convertTimestampToString(Timestamp timestamp)

Converts java.sql.Timestamp to string format timestamp.

Parameters:

timestamp - java.sql.Timestamp object.

Returns:

Timestamp in string format.

convertStringToTimestamp

public static Timestamp convertStringToTimestamp(String timestamp)

Converts timestamp string format to java.sql.Timestamp.

Parameters:

timestamp - Timestamp in string format.

Returns:

java.sql.Timestamp object.

throwMetaException

public static void throwMetaException(Exception e)
                               throws MetaException

Catches exceptions that cannot be handled and wraps them in MetaException.

Parameters:

e - exception to wrap.

Throws:

MetaException - wrapper for the exception

encodeTableName

public static String encodeTableName(String name)

newMetaException

public static MetaException newMetaException(Exception e)

convert Exception to MetaException, which sets the cause to such exception

Parameters:

e - cause of the exception

Returns:

the MetaException with the specified exception as the cause

newMetaException

public static MetaException newMetaException(String errorMessage,
                                             Exception e)

convert Exception to MetaException, which sets the cause to such exception

Parameters:

errorMessage - the error message for this MetaException

e - cause of the exception

Returns:

the MetaException with the specified exception as the cause

getColumnNamesForTable

public static List<String> getColumnNamesForTable(Table table)

validateTblStorage

public static boolean validateTblStorage(StorageDescriptor sd)

validateName

public static boolean validateName(String name,
                                   org.apache.hadoop.conf.Configuration conf)

validateName Checks the name conforms to our standars which are: "[a-zA-z_0-9]+". checks this is just characters and numbers and _

Parameters:

name - the name to validate

conf - hive configuration

Returns:

true or false depending on conformance if it doesn't match the pattern.

isExternalTable

public static boolean isExternalTable(Table table)

Determines whether a table is an external table.

Parameters:

table - table of interest

Returns:

true if external

isIcebergTable

public static boolean isIcebergTable(Map<String,String> params)

isTranslatedToExternalTable

public static boolean isTranslatedToExternalTable(Table table)

getDbNameFromReplPolicy

public static String getDbNameFromReplPolicy(String replPolicy)

isDbReplIncompatible

public static boolean isDbReplIncompatible(Database db)

isDbBeingPlannedFailedOver

public static boolean isDbBeingPlannedFailedOver(Database db)

isDbBeingPlannedFailedOverAtEndpoint

public static boolean isDbBeingPlannedFailedOverAtEndpoint(Database db,
                                                           MetaStoreUtils.FailoverEndpoint endPoint)

isTargetOfReplication

public static boolean isTargetOfReplication(Database db)

isBackgroundThreadsEnabledForRepl

public static boolean isBackgroundThreadsEnabledForRepl(Database db)

checkIfDbNeedsToBeSkipped

public static boolean checkIfDbNeedsToBeSkipped(Database db)

getReplicationDbProps

public static List<String> getReplicationDbProps()

isExternalTablePurge

public static boolean isExternalTablePurge(Table table)

Determines whether an table needs to be purged or not.

Parameters:

table - table of interest

Returns:

true if external table needs to be purged

isExternal

public static boolean isExternal(Map<String,String> tableParams)

isPropertyTrue

public static boolean isPropertyTrue(Map<String,String> tableParams,
                                     String prop)

isSkipTrash

public static boolean isSkipTrash(Map<String,String> tableParams)

Determines whether an table needs to be deleted completely or moved to trash directory.

Parameters:

tableParams - parameters of the table

Returns:

true if the table needs to be deleted rather than moved to trash directory

isInsertOnlyTableParam

public static boolean isInsertOnlyTableParam(Map<String,String> params)

Duplicates AcidUtils; used in a couple places in metastore.

isNonNativeTable

public static boolean isNonNativeTable(Table table)

isNonNativeTable

public static boolean isNonNativeTable(Map<String,String> tblProps)

getPvals

public static List<String> getPvals(List<FieldSchema> partCols,
                                    Map<String,String> partSpec)

Given a list of partition columns and a partial mapping from some partition columns to values the function returns the values for the column.

Parameters:

partCols - the list of table partition columns

partSpec - the partial mapping from partition column to values

Returns:

list of values of for given partition columns, any missing values in partSpec is replaced by an empty string

makePartNameMatcher

public static String makePartNameMatcher(Table table,
                                         List<String> partVals,
                                         String defaultStr)
                                  throws MetaException

Throws:

MetaException

compareFieldColumns

public static boolean compareFieldColumns(List<FieldSchema> schema1,
                                          List<FieldSchema> schema2)

Parameters:

schema1: - The first schema to be compared

schema2: - The second schema to be compared

Returns:

true if the two schemas are the same else false for comparing a field we ignore the comment it has

isArchived

public static boolean isArchived(Partition part)

getOriginalLocation

public static org.apache.hadoop.fs.Path getOriginalLocation(Partition part)

getArchivingLevel

public static int getArchivingLevel(Partition part)
                             throws MetaException

Throws:

MetaException

getMetaStoreSaslProperties

public static Map<String,String> getMetaStoreSaslProperties(org.apache.hadoop.conf.Configuration conf,
                                                            boolean useSSL)

Read and return the meta store Sasl configuration. Currently it uses the default Hadoop SASL configuration and can be configured using "hadoop.rpc.protection" HADOOP-10211, made a backward incompatible change due to which this call doesn't work with Hadoop 2.4.0 and later.

Parameters:

conf -

Returns:

The SASL configuration

addToClassPath

public static ClassLoader addToClassPath(ClassLoader cloader,
                                         String[] newPaths)
                                  throws Exception

Add new elements to the classpath.

Parameters:

newPaths - Array of classpath elements

Throws:

Exception

getDDLFromFieldSchema

public static String getDDLFromFieldSchema(String structName,
                                           List<FieldSchema> fieldSchemas)

Convert FieldSchemas to Thrift DDL.

getTableMetadata

public static Properties getTableMetadata(Table table)

getPartitionMetadata

public static Properties getPartitionMetadata(Partition partition,
                                              Table table)

getSchema

public static Properties getSchema(Partition part,
                                   Table table)

getPartSchemaFromTableSchema

public static Properties getPartSchemaFromTableSchema(StorageDescriptor sd,
                                                      Map<String,String> parameters,
                                                      Properties tblSchema)

Get partition level schema from table level schema. This function will use the same column names, column types and partition keys for each partition Properties. Their values are copied from the table Properties. This is mainly to save CPU and memory. CPU is saved because the first time the StorageDescriptor column names are accessed, JDO needs to execute a SQL query to retrieve the data. If we know the data will be the same as the table level schema and they are immutable, we should just reuse the table level schema objects.

Parameters:

sd - The Partition level Storage Descriptor.

parameters - partition level parameters

tblSchema - The table level schema from which this partition should be copied.

Returns:

the properties

getSchemaWithoutCols

public static Properties getSchemaWithoutCols(StorageDescriptor sd,
                                              Map<String,String> parameters,
                                              String databaseName,
                                              String tableName,
                                              List<FieldSchema> partitionKeys)

getSchema

public static Properties getSchema(StorageDescriptor sd,
                                   StorageDescriptor tblsd,
                                   Map<String,String> parameters,
                                   String databaseName,
                                   String tableName,
                                   List<FieldSchema> partitionKeys)

getColumnNameDelimiter

public static String getColumnNameDelimiter(List<FieldSchema> fieldSchemas)

getColumnNamesFromFieldSchema

public static String getColumnNamesFromFieldSchema(List<FieldSchema> fieldSchemas)

Convert FieldSchemas to columnNames.

getColumnTypesFromFieldSchema

public static String getColumnTypesFromFieldSchema(List<FieldSchema> fieldSchemas,
                                                   String delimiter)

Convert FieldSchemas to columnTypes.

getColumnTypesFromFieldSchema

public static String getColumnTypesFromFieldSchema(List<FieldSchema> fieldSchemas)

getColumnCommentsFromFieldSchema

public static String getColumnCommentsFromFieldSchema(List<FieldSchema> fieldSchemas)

isMaterializedViewTable

public static boolean isMaterializedViewTable(Table table)

getColumnNames

public static List<String> getColumnNames(List<FieldSchema> schema)

isValidSchedulingPolicy

public static boolean isValidSchedulingPolicy(String str)

parseSchedulingPolicy

public static WMPoolSchedulingPolicy parseSchedulingPolicy(String schedulingPolicy)

prependCatalogToDbName

public static String prependCatalogToDbName(@Nullable
                                            String catalogName,
                                            @Nullable
                                            String dbName,
                                            org.apache.hadoop.conf.Configuration conf)

Given a catalog name and database name cram them together into one string. This method can be used if you do not know the catalog name, in which case the default catalog will be retrieved from the conf object. The resulting string can be parsed apart again via parseDbName(String, Configuration).

Parameters:

catalogName - catalog name, can be null if no known.

dbName - database name, can be null or empty.

conf - configuration object, used to determine default catalog if catalogName is null

Returns:

one string that contains both.

prependNotNullCatToDbName

public static String prependNotNullCatToDbName(String catalogName,
                                               String dbName)

Given a catalog name and database name, cram them together into one string. These can be parsed apart again via parseDbName(String, Configuration).

Parameters:

catalogName - catalog name. This cannot be null. If this might be null use prependCatalogToDbName(String, String, Configuration) instead.

dbName - database name.

Returns:

one string that contains both.

prependCatalogToDbName

public static String prependCatalogToDbName(String dbName,
                                            org.apache.hadoop.conf.Configuration conf)

Prepend the default 'hive' catalog onto the database name.

Parameters:

dbName - database name

conf - configuration object, used to determine default catalog

Returns:

one string with the 'hive' catalog name prepended.

parseDbName

public static String[] parseDbName(String dbName,
                                   org.apache.hadoop.conf.Configuration conf)
                            throws MetaException

Parse the catalog name out of the database name. If no catalog name is present then the default catalog (as set in configuration file) will be assumed.

Parameters:

dbName - name of the database. This may or may not contain the catalog name.

conf - configuration object, used to determine the default catalog if it is not present in the database name.

Returns:

an array of two elements, the first being the catalog name, the second the database name.

Throws:

MetaException - if the name is not either just a database name or a catalog plus database name with the proper delimiters.

getDefaultCatalog

public static String getDefaultCatalog(org.apache.hadoop.conf.Configuration conf)

isView

public static boolean isView(Table table)

filterMapKeys

public static <T> void filterMapKeys(Map<String,T> map,
                                     Predicate<String> predicate)

filters a given map with predicate provided. All entries of map whose key matches with predicate will be removed. Expects map to be modifiable and does the operation on actual map, so does not return a copy of filtered map.

Parameters:

map - A map of String key-value pairs

predicate - Predicate with pattern to filter the map

filterMapkeys

public static <T> void filterMapkeys(Map<String,T> map,
                                     List<Predicate<String>> predicates)

filters a given map with list of predicates. All entries of map whose key matches with any predicate will be removed. Expects map to be modifiable and does the operation on actual map, so does not return a copy of filtered map.

Parameters:

map - A map of String key-value pairs

predicates - List of predicates with patterns to filter the map

compilePatternsToPredicates

public static List<Predicate<String>> compilePatternsToPredicates(List<String> patterns)

Compile a list of regex patterns and collect them as Predicates.

Parameters:

patterns - List of regex patterns to be compiled

Returns:

a List of Predicate created by compiling the regex patterns

makeOrderSpecs

public static List<Object[]> makeOrderSpecs(String order)

Get order specs from a represented string.

Parameters:

order - specified in partColIndex[,partColIndex]*:[-|\+]+ pattern

Returns:

the order specs

addPartitonSpecsToList

public static void addPartitonSpecsToList(PartitionsSpecByExprResult r,
                                          List<PartitionSpec> result)

hasUnknownPartitions

public static boolean hasUnknownPartitions(PartitionsSpecByExprResult r)

getTableNameFor

public static TableName getTableNameFor(Table table)

isNoAutoCompactSet

public static boolean isNoAutoCompactSet(Map<String,String> dbParameters,
                                         Map<String,String> tblParameters)

Because TABLE_NO_AUTO_COMPACT was originally assumed to be NO_AUTO_COMPACT and then was moved to no_auto_compact, we need to check it in both cases. Check the database level no_auto_compact , if present it is given priority else table level no_auto_compact is considered.

getNoAutoCompact

public static String getNoAutoCompact(Map<String,String> parameters)

Get no_auto_compact property by checking in both lower and upper cases

Parameters:

parameters -

Returns:

true/false if set, null if there is no NO_AUTO_COMPACT set in database level config,

getHostFromId

public static String getHostFromId(String id)

getThreadIdFromId

public static String getThreadIdFromId(String id)

isNoCleanUpSet

public static boolean isNoCleanUpSet(Map<String,String> parameters)

convertToGetPartitionsByNamesRequest

public static GetPartitionsByNamesRequest convertToGetPartitionsByNamesRequest(String dbName,
                                                                               String tblName,
                                                                               List<String> partNames)

convertToGetPartitionsByNamesRequest

public static GetPartitionsByNamesRequest convertToGetPartitionsByNamesRequest(String dbName,
                                                                               String tblName,
                                                                               List<String> partNames,
                                                                               boolean getColStats,
                                                                               String engine,
                                                                               String validWriteIdList,
                                                                               Long tableId)

createThriftPartitionsReq

public static <T> T createThriftPartitionsReq(Class<T> clazz,
                                              org.apache.hadoop.conf.Configuration conf,
                                              T... deepCopy)

getHttpPath

public static String getHttpPath(String httpPath)

The config parameter can be like "path", "/path", "/path/", "path/*", "/path1/path2/*" and so on. httpPath should end up as "/*", "/path/*" or "/path1/../pathN/*"

Parameters:

httpPath -

Returns: