org.apache.hadoop.hive.ql.metadata

Interface HiveStorageHandler

All Superinterfaces:

org.apache.hadoop.conf.Configurable

All Known Implementing Classes:

DefaultStorageHandler, SyslogStorageHandler

@InterfaceAudience.Public
 @InterfaceStability.Stable
public interface HiveStorageHandler
extends org.apache.hadoop.conf.Configurable

HiveStorageHandler defines a pluggable interface for adding new storage handlers to Hive. A storage handler consists of a bundle of the following:

• input format
• output format
• serde
• metadata hooks for keeping an external catalog in sync with Hive's metastore
• rules for setting up the configuration properties on map/reduce jobs which access tables stored by this handler

Storage handler classes are plugged in using the STORED BY 'classname' clause in CREATE TABLE.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	HiveStorageHandler.AcidSupportType

Field Summary

Fields

Modifier and Type	Field and Description
static List<AlterTableType>	DEFAULT_ALLOWED_ALTER_OPS

Method Summary

Modifier and Type	Method and Description
default List<org.apache.hadoop.hive.metastore.api.FieldSchema>	acidSelectColumns(Table table, Context.Operation operation) UpdateDeleteSemanticAnalyzer rewrites DELETE/UPDATE queries into INSERT queries.
default List<org.apache.hadoop.hive.metastore.api.FieldSchema>	acidSortColumns(Table table, Context.Operation operation) UpdateDeleteSemanticAnalyzer rewrites DELETE/UPDATE queries into INSERT queries.
default List<VirtualColumn>	acidVirtualColumns() Specifies which additional virtual columns should be added to the virtual column registry during compilation for tables that support ACID operations.
default boolean	addDynamicSplitPruningEdge(Table table, ExprNodeDesc syntheticFilterPredicate) Test if the storage handler allows the push-down of join filter predicate to prune further the splits.
default boolean	alwaysUnpartitioned() Check if partition columns should be removed and added to the list of regular columns in HMS.
default boolean	areSnapshotsSupported() Gets whether this storage handler supports snapshots.
default boolean	canProvideBasicStatistics() Check if the storage handler can provide basic statistics.
default boolean	commitInMoveTask() Checks if we should keep the MoveTask and use the storageHandlerCommit(Properties, boolean) method for committing inserts instead of DefaultHiveMetaHook.commitInsertTable(Table, boolean).
void	configureInputJobCredentials(TableDesc tableDesc, Map<String,String> secrets) This method is called to allow the StorageHandlers the chance to populate secret keys into the job's credentials.
void	configureInputJobProperties(TableDesc tableDesc, Map<String,String> jobProperties) This method is called to allow the StorageHandlers the chance to populate the JobContext.getConfiguration() with properties that maybe be needed by the handler's bundled artifacts (ie InputFormat, SerDe, etc).
void	configureJobConf(TableDesc tableDesc, org.apache.hadoop.mapred.JobConf jobConf) Called just before submitting MapReduce job.
void	configureOutputJobProperties(TableDesc tableDesc, Map<String,String> jobProperties) This method is called to allow the StorageHandlers the chance to populate the JobContext.getConfiguration() with properties that maybe be needed by the handler's bundled artifacts (ie InputFormat, SerDe, etc).
void	configureTableJobProperties(TableDesc tableDesc, Map<String,String> jobProperties) Deprecated.
default DynamicPartitionCtx	createDPContext(HiveConf conf, Table table, Context.Operation writeOperation) Creates a DynamicPartitionCtx instance that will be set up by the storage handler itself.
default boolean	directInsertCTAS() Check if CTAS operations should behave in a direct-insert manner (i.e.
default void	executeOperation(Table table, AlterTableExecuteSpec executeSpec) Execute an operation on storage handler level
HiveAuthorizationProvider	getAuthorizationProvider() Returns the implementation specific authorization provider
default Map<String,String>	getBasicStatistics(Partish partish) Return some basic statistics (numRows, numFiles, totalSize) calculated by the underlying storage handler implementation.
default SnapshotContext	getCurrentSnapshotContext(Table table) Query the most recent unique snapshot's context of the passed table.
default String	getFileFormatPropertyKey() Get file format property key, if the file format is configured through a table property.
Class<? extends org.apache.hadoop.mapred.InputFormat>	getInputFormatClass()
default org.apache.hadoop.hive.metastore.api.LockType	getLockType(WriteEntity writeEntity)
HiveMetaHook	getMetaHook()
default Map<String,String>	getOperatorDescProperties(OperatorDesc operatorDesc, Map<String,String> initialProps) Used to add additional operator specific information from storage handler during DESCRIBE EXTENDED statement.
Class<? extends org.apache.hadoop.mapred.OutputFormat>	getOutputFormatClass()
default List<TransformSpec>	getPartitionTransformSpec(Table table) Return a list of partition transform specifications.
Class<? extends AbstractSerDe>	getSerDeClass()
default StorageHandlerInfo	getStorageHandlerInfo(org.apache.hadoop.hive.metastore.api.Table table) Used to fetch runtime information about storage handler during DESCRIBE EXTENDED statement
default URI	getURIForAuth(org.apache.hadoop.hive.metastore.api.Table table) Constructs a URI for authorization purposes using the HMS table object
default boolean	isAllowedAlterOperation(AlterTableType opType) Checks whether a certain ALTER TABLE operation is supported by the storage handler implementation.
default boolean	isMetadataTableSupported()
default boolean	isTimeTravelAllowed() Should return true if the StorageHandler is able to handle time travel.
default boolean	isValidMetadataTable(String metaTableName)
default void	prepareAlterTableEnvironmentContext(AbstractAlterTableDesc alterTableDesc, org.apache.hadoop.hive.metastore.api.EnvironmentContext environmentContext) Alter table operations can rely on this to customize the EnvironmentContext to be used during the alter table invocation (both on client and server side of HMS)
default List<org.apache.hadoop.hive.metastore.api.FieldSchema>	sortColumns(Table table) Collect the columns that are used to sort the content of the data files
default void	storageHandlerCommit(Properties commitProperties, boolean overwrite) Commits the inserts for the non-native tables.
default HiveStorageHandler.AcidSupportType	supportsAcidOperations(Table table) Specifies whether the table supports ACID operations or not (DELETE, UPDATE and MERGE statements).
default boolean	supportsPartitionTransform() Check if the underlying storage handler implementation support partition transformations.
default boolean	supportsSortColumns() Check if the underlying storage handler implementation supports sort columns.
default boolean	supportsTruncateOnNonNativeTables() Check if the underlying storage handler implementation supports truncate operation for non native tables.
default void	validateSinkDesc(FileSinkDesc sinkDesc) Validates whether the sink operation is permitted for the specific storage handler, based on information contained in the sinkDesc.

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Field Detail

DEFAULT_ALLOWED_ALTER_OPS

static final List<AlterTableType> DEFAULT_ALLOWED_ALTER_OPS

Method Detail

getInputFormatClass

Class<? extends org.apache.hadoop.mapred.InputFormat> getInputFormatClass()

Returns:

Class providing an implementation of InputFormat

getOutputFormatClass

Class<? extends org.apache.hadoop.mapred.OutputFormat> getOutputFormatClass()

Returns:

Class providing an implementation of OutputFormat

getSerDeClass

Class<? extends AbstractSerDe> getSerDeClass()

Returns:

Class providing an implementation of AbstractSerDe

getMetaHook

HiveMetaHook getMetaHook()

Returns:

metadata hook implementation, or null if this storage handler does not need any metadata notifications

getAuthorizationProvider

HiveAuthorizationProvider getAuthorizationProvider()
                                            throws HiveException

Returns the implementation specific authorization provider

Returns:

authorization provider

Throws:

HiveException

configureInputJobProperties

void configureInputJobProperties(TableDesc tableDesc,
                                 Map<String,String> jobProperties)

This method is called to allow the StorageHandlers the chance to populate the JobContext.getConfiguration() with properties that maybe be needed by the handler's bundled artifacts (ie InputFormat, SerDe, etc). Key value pairs passed into jobProperties are guaranteed to be set in the job's configuration object. User's can retrieve "context" information from tableDesc. User's should avoid mutating tableDesc and only make changes in jobProperties. This method is expected to be idempotent such that a job called with the same tableDesc values should return the same key-value pairs in jobProperties. Any external state set by this method should remain the same if this method is called again. It is up to the user to determine how best guarantee this invariant. This method in particular is to create a configuration for input.

Parameters:

tableDesc - descriptor for the table being accessed

jobProperties - receives properties copied or transformed from the table properties

configureInputJobCredentials

void configureInputJobCredentials(TableDesc tableDesc,
                                  Map<String,String> secrets)

This method is called to allow the StorageHandlers the chance to populate secret keys into the job's credentials.

configureOutputJobProperties

void configureOutputJobProperties(TableDesc tableDesc,
                                  Map<String,String> jobProperties)

This method is called to allow the StorageHandlers the chance to populate the JobContext.getConfiguration() with properties that maybe be needed by the handler's bundled artifacts (ie InputFormat, SerDe, etc). Key value pairs passed into jobProperties are guaranteed to be set in the job's configuration object. User's can retrieve "context" information from tableDesc. User's should avoid mutating tableDesc and only make changes in jobProperties. This method is expected to be idempotent such that a job called with the same tableDesc values should return the same key-value pairs in jobProperties. Any external state set by this method should remain the same if this method is called again. It is up to the user to determine how best guarantee this invariant. This method in particular is to create a configuration for output.

Parameters:

tableDesc - descriptor for the table being accessed

jobProperties - receives properties copied or transformed from the table properties

configureTableJobProperties

@Deprecated
void configureTableJobProperties(TableDesc tableDesc,
                                             Map<String,String> jobProperties)

Deprecated.

Deprecated use configureInputJobProperties/configureOutputJobProperties methods instead. Configures properties for a job based on the definition of the source or target table it accesses.

Parameters:

tableDesc - descriptor for the table being accessed

jobProperties - receives properties copied or transformed from the table properties

configureJobConf

void configureJobConf(TableDesc tableDesc,
                      org.apache.hadoop.mapred.JobConf jobConf)

Called just before submitting MapReduce job.

Parameters:

tableDesc - descriptor for the table being accessed

jobConf - jobConf for MapReduce job

getStorageHandlerInfo

default StorageHandlerInfo getStorageHandlerInfo(org.apache.hadoop.hive.metastore.api.Table table)
                                          throws org.apache.hadoop.hive.metastore.api.MetaException

Used to fetch runtime information about storage handler during DESCRIBE EXTENDED statement

Parameters:

table - table definition

Returns:

StorageHandlerInfo containing runtime information about storage handler OR `null` if the storage handler choose to not provide any runtime information.

Throws:

org.apache.hadoop.hive.metastore.api.MetaException

getLockType

default org.apache.hadoop.hive.metastore.api.LockType getLockType(WriteEntity writeEntity)

addDynamicSplitPruningEdge

default boolean addDynamicSplitPruningEdge(Table table,
                                           ExprNodeDesc syntheticFilterPredicate)

Test if the storage handler allows the push-down of join filter predicate to prune further the splits.

Parameters:

table - The table to filter.

syntheticFilterPredicate - Join filter predicate.

Returns:

true if supports dynamic split pruning for the given predicate.

getOperatorDescProperties

default Map<String,String> getOperatorDescProperties(OperatorDesc operatorDesc,
                                                     Map<String,String> initialProps)

Used to add additional operator specific information from storage handler during DESCRIBE EXTENDED statement.

Parameters:

operatorDesc - operatorDesc

initialProps - Map containing initial operator properties

Returns:

Map<String, String> containing additional operator specific information from storage handler OR `initialProps` if the storage handler choose to not provide any such information.

getBasicStatistics

default Map<String,String> getBasicStatistics(Partish partish)

Return some basic statistics (numRows, numFiles, totalSize) calculated by the underlying storage handler implementation.

Parameters:

partish - a partish wrapper class

Returns:

map of basic statistics, can be null

canProvideBasicStatistics

default boolean canProvideBasicStatistics()

Check if the storage handler can provide basic statistics.

Returns:

true if the storage handler can supply the basic statistics

directInsertCTAS

default boolean directInsertCTAS()

Check if CTAS operations should behave in a direct-insert manner (i.e. no move task). If true, the compiler will not include the table creation task and move task into the execution plan. Instead, it's the responsibility of storage handler/serde to create the table during the compilation phase. Please note that the atomicity of the operation will suffer in this case, i.e. the created table might become exposed, depending on the implementation, before the CTAS operations finishes. Rollback (e.g. dropping the table) is also the responsibility of the storage handler in case of failures.

Returns:

whether direct insert CTAS is required

alwaysUnpartitioned

default boolean alwaysUnpartitioned()

Check if partition columns should be removed and added to the list of regular columns in HMS. This can be useful for non-native tables where the table format/layout differs from the standard Hive table layout, e.g. Iceberg tables. For these table formats, the partition column values are stored in the data files along with regular column values, therefore the object inspectors should include the partition columns as well. Any partitioning scheme provided via the standard HiveQL syntax will be honored but stored in someplace other than HMS, depending on the storage handler implementation.

Returns:

whether table should always be unpartitioned from the perspective of HMS

supportsAcidOperations

default HiveStorageHandler.AcidSupportType supportsAcidOperations(Table table)

Specifies whether the table supports ACID operations or not (DELETE, UPDATE and MERGE statements). Possible return values:

• AcidSupportType.NONE - ACID operations are not supported
• AcidSupportType.WITH_TRANSACTIONS - ACID operations are supported, and must use a valid HiveTxnManager to wrap the operation in a transaction, like in the case of standard Hive ACID tables
• AcidSupportType.WITHOUT_TRANSACTIONS - ACID operations are supported, and there is no need for a HiveTxnManager to open/close transactions for the operation, i.e. org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager can be used

Returns:

the table's ACID support type

acidVirtualColumns

default List<VirtualColumn> acidVirtualColumns()

Specifies which additional virtual columns should be added to the virtual column registry during compilation for tables that support ACID operations. Should only return a non-empty list if supportsAcidOperations(org.apache.hadoop.hive.ql.metadata.Table) ()} returns something other NONE.

Returns:

the list of ACID virtual columns

acidSelectColumns

default List<org.apache.hadoop.hive.metastore.api.FieldSchema> acidSelectColumns(Table table,
                                                                                 Context.Operation operation)

UpdateDeleteSemanticAnalyzer rewrites DELETE/UPDATE queries into INSERT queries. - DELETE FROM T WHERE A = 32 is rewritten into INSERT INTO T SELECT <selectCols> FROM T WHERE A = 32 SORT BY <sortCols>. - UPDATE T SET B=12 WHERE A = 32 is rewritten into INSERT INTO T SELECT <selectCols>, <newValues> FROM T WHERE A = 32 SORT BY <sortCols>. This method specifies which columns should be injected into the <selectCols> part of the rewritten query. Should only return a non-empty list if supportsAcidOperations(org.apache.hadoop.hive.ql.metadata.Table) returns something other NONE.

Parameters:

table - the table which is being deleted/updated/merged into

operation - the operation type we are executing

Returns:

the list of columns that should be projected in the rewritten ACID query

acidSortColumns

default List<org.apache.hadoop.hive.metastore.api.FieldSchema> acidSortColumns(Table table,
                                                                               Context.Operation operation)

UpdateDeleteSemanticAnalyzer rewrites DELETE/UPDATE queries into INSERT queries. E.g. DELETE FROM T WHERE A = 32 is rewritten into INSERT INTO T SELECT <selectCols> FROM T WHERE A = 32 SORT BY <sortCols>. This method specifies which columns should be injected into the <sortCols> part of the rewritten query. Should only return a non-empty list if supportsAcidOperations(org.apache.hadoop.hive.ql.metadata.Table) returns something other NONE.

Parameters:

table - the table which is being deleted/updated/merged into

operation - the operation type we are executing

Returns:

the list of columns that should be used as sort columns in the rewritten ACID query

supportsSortColumns

default boolean supportsSortColumns()

Check if the underlying storage handler implementation supports sort columns.

Returns:

true if the storage handler can support it

sortColumns

default List<org.apache.hadoop.hive.metastore.api.FieldSchema> sortColumns(Table table)

Collect the columns that are used to sort the content of the data files

Parameters:

table - the table which is being sorted

Returns:

the list of columns that are used during data sorting

supportsPartitionTransform

default boolean supportsPartitionTransform()

Check if the underlying storage handler implementation support partition transformations.

Returns:

true if the storage handler can support it

getPartitionTransformSpec

default List<TransformSpec> getPartitionTransformSpec(Table table)

Return a list of partition transform specifications. This method should be overwritten in case supportsPartitionTransform() returns true.

Parameters:

table - the HMS table, must be non-null

Returns:

partition transform specification, can be null.

createDPContext

default DynamicPartitionCtx createDPContext(HiveConf conf,
                                            Table table,
                                            Context.Operation writeOperation)
                                     throws SemanticException

Creates a DynamicPartitionCtx instance that will be set up by the storage handler itself. Useful for non-native tables where partitions are not handled by Hive, and sorting is required in a custom way before writing the table.

Parameters:

conf - job conf

table - the HMS table

Returns:

the created DP context object, null if DP context / sorting is not required

Throws:

SemanticException

getFileFormatPropertyKey

default String getFileFormatPropertyKey()

Get file format property key, if the file format is configured through a table property.

Returns:

table property key, can be null

commitInMoveTask

default boolean commitInMoveTask()

Checks if we should keep the MoveTask and use the storageHandlerCommit(Properties, boolean) method for committing inserts instead of DefaultHiveMetaHook.commitInsertTable(Table, boolean).

Returns:

Returns true if we should use the storageHandlerCommit(Properties, boolean) method

storageHandlerCommit

default void storageHandlerCommit(Properties commitProperties,
                                  boolean overwrite)
                           throws HiveException

Commits the inserts for the non-native tables. Used in the MoveTask.

Parameters:

commitProperties - Commit properties which are needed for the handler based commit

overwrite - If this is an INSERT OVERWRITE then it is true

Throws:

HiveException - If there is an error during commit

isAllowedAlterOperation

default boolean isAllowedAlterOperation(AlterTableType opType)

Checks whether a certain ALTER TABLE operation is supported by the storage handler implementation.

Parameters:

opType - The alter operation type (e.g. RENAME_COLUMNS)

Returns:

whether the operation is supported by the storage handler

supportsTruncateOnNonNativeTables

default boolean supportsTruncateOnNonNativeTables()

Check if the underlying storage handler implementation supports truncate operation for non native tables.

Returns:

true if the storage handler can support it

isTimeTravelAllowed

default boolean isTimeTravelAllowed()

Should return true if the StorageHandler is able to handle time travel.

Returns:

True if time travel is allowed

isMetadataTableSupported

default boolean isMetadataTableSupported()

isValidMetadataTable

default boolean isValidMetadataTable(String metaTableName)

getURIForAuth

default URI getURIForAuth(org.apache.hadoop.hive.metastore.api.Table table)
                   throws URISyntaxException

Constructs a URI for authorization purposes using the HMS table object

Parameters:

table - The HMS table object

Returns:

the URI for authorization

Throws:

URISyntaxException

validateSinkDesc

default void validateSinkDesc(FileSinkDesc sinkDesc)
                       throws SemanticException

Validates whether the sink operation is permitted for the specific storage handler, based on information contained in the sinkDesc.

Parameters:

sinkDesc - The sink descriptor

Throws:

SemanticException - if the sink operation is not allowed

executeOperation

default void executeOperation(Table table,
                              AlterTableExecuteSpec executeSpec)

Execute an operation on storage handler level

Parameters:

executeSpec - operation specification

areSnapshotsSupported

default boolean areSnapshotsSupported()

Gets whether this storage handler supports snapshots.

Returns:

true means snapshots are supported false otherwise

getCurrentSnapshotContext

default SnapshotContext getCurrentSnapshotContext(Table table)

Query the most recent unique snapshot's context of the passed table.

Parameters:

table - - Table which snapshot context should be returned.

Returns:

SnapshotContext wraps the snapshotId or null if no snapshot present.

prepareAlterTableEnvironmentContext

default void prepareAlterTableEnvironmentContext(AbstractAlterTableDesc alterTableDesc,
                                                 org.apache.hadoop.hive.metastore.api.EnvironmentContext environmentContext)

Alter table operations can rely on this to customize the EnvironmentContext to be used during the alter table invocation (both on client and server side of HMS)

Parameters:

alterTableDesc - the alter table desc (e.g.: AlterTableSetPropertiesDesc) containing the work to do

environmentContext - an existing EnvironmentContext created prior, now to be filled/amended