org.apache.hadoop.hive.ql.parse

Class RewriteSemanticAnalyzer

java.lang.Object

org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer

org.apache.hadoop.hive.ql.parse.SemanticAnalyzer

org.apache.hadoop.hive.ql.parse.CalcitePlanner

org.apache.hadoop.hive.ql.parse.RewriteSemanticAnalyzer

Direct Known Subclasses:

AcidExportSemanticAnalyzer, MergeSemanticAnalyzer, SplitUpdateSemanticAnalyzer, UpdateDeleteSemanticAnalyzer

public abstract class RewriteSemanticAnalyzer
extends CalcitePlanner

A subclass of the SemanticAnalyzer that just handles update, delete and merge statements. It works by rewriting the updates and deletes into insert statements (since they are actually inserts) and then doing some patch up to make them work as updates and deletes instead.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	RewriteSemanticAnalyzer.ColumnAppender
protected static class	RewriteSemanticAnalyzer.NativeAcidColumnAppender
protected static class	RewriteSemanticAnalyzer.NonNativeAcidColumnAppender
protected static class	RewriteSemanticAnalyzer.ReparseResult

Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.parse.CalcitePlanner

CalcitePlanner.ASTSearcher, CalcitePlanner.CalcitePlannerAction, CalcitePlanner.InputContext, CalcitePlanner.PreCboCtx

Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.parse.SemanticAnalyzer

SemanticAnalyzer.GenericUDAFInfo, SemanticAnalyzer.MaterializationRebuildMode

Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer

BaseSemanticAnalyzer.AnalyzeRewriteContext, BaseSemanticAnalyzer.TableSpec

Field Summary

Fields

Modifier and Type	Field and Description
static String	DELETE_PREFIX
protected static String	INDENT
protected static org.slf4j.Logger	LOG
static String	SUB_QUERY_ALIAS
protected boolean	useSuper

Fields inherited from class org.apache.hadoop.hive.ql.parse.SemanticAnalyzer

aliasToCTEs, analyzeRewrite, ast, autogenColAliasPrfxLbl, createVwDesc, defaultJoinMerge, disableJoinMerge, DUMMY_DATABASE, DUMMY_TABLE, forViewCreation, mvRebuildDbName, mvRebuildMode, mvRebuildName, opParseCtx, originalResultSchema, prunedPartitions, resultSchema, SUBQUERY_TAG_1, SUBQUERY_TAG_2, topOps, topOpsCopy, unparseTranslator, VALUES_TMP_TABLE_NAME_PREFIX, viewSelect

Fields inherited from class org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer

acidFileSinks, cacheUsage, cContext, columnAccessInfo, conf, console, ctx, db, fetchTask, idToTableNameMap, inputs, linfo, outputs, prepareQuery, queryProperties, queryState, rootTasks, STATIC_LOG, tableAccessInfo, transactionalInQuery, txnManager, updateColumnAccessInfo

Method Summary

Modifier and Type	Method and Description
protected void	addColsToSelect(List<org.apache.hadoop.hive.metastore.api.FieldSchema> partCols, StringBuilder rewrittenQueryStr) Append list of partition columns to Insert statement, i.e.
protected void	addColsToSelect(List<org.apache.hadoop.hive.metastore.api.FieldSchema> cols, StringBuilder rewrittenQueryStr, ASTNode target) Append list of columns to rewritten statement.
protected void	addColsToSelect(List<org.apache.hadoop.hive.metastore.api.FieldSchema> cols, StringBuilder rewrittenQueryStr, String alias) Append list of columns to rewritten statement.
protected void	addPartitionColsAsValues(List<org.apache.hadoop.hive.metastore.api.FieldSchema> partCols, String alias, List<String> values)
protected abstract void	analyze(ASTNode tree, Table table, ASTNode tableName)
void	analyzeInternal(ASTNode tree)
void	analyzeRewrittenTree(ASTNode rewrittenTree, Context rewrittenCtx)
protected void	appendInsertBranch(StringBuilder rewrittenQueryStr, String hintStr, List<String> values)
protected void	appendSortBy(StringBuilder rewrittenQueryStr, List<String> keys)
protected void	appendTarget(StringBuilder rewrittenQueryStr, ASTNode target, String targetName)
protected void	checkValidSetClauseTarget(ASTNode colName, Table targetTable) Assert that we are not asked to update a bucketing column or partition column.
protected Map<String,ASTNode>	collectSetColumnsAndExpressions(ASTNode setClause, Set<String> setRCols, Table targetTable)
protected StringBuilder	createRewrittenQueryStrBuilder()
protected ASTNode	findLHSofAssignment(ASTNode assignment)
protected RewriteSemanticAnalyzer.ColumnAppender	getColumnAppender(String subQueryAlias)
protected String	getMatchedText(ASTNode n) This allows us to take an arbitrary ASTNode and turn it back into SQL that produced it without needing to understand what it is (except for QuotedIdentifiers).
protected String	getSimpleTableName(ASTNode n) Returns the table name to use in the generated query preserving original quotes/escapes if any.
protected static Table	getTable(ASTNode tabRef, Hive db, boolean throwException)
protected abstract ASTNode	getTargetTableNode(ASTNode tree)
protected boolean	isAliased(ASTNode n)
protected void	markReadEntityForUpdate() Walk through all our inputs and set them to note that this read is part of an update or a delete.
protected RewriteSemanticAnalyzer.ReparseResult	parseRewrittenQuery(StringBuilder rewrittenQueryStr, String originalQuery) Parse the newly generated SQL statement to get a new AST.
protected void	patchProjectionForUpdate(ASTNode insertBranch, Map<Integer,ASTNode> setColExprs)
protected void	setUpAccessControlInfoForUpdate(Table mTable, Map<String,ASTNode> setCols) For updates, we need to set the column access info so that it contains information on the columns we are updating.
protected void	updateOutputs(Table targetTable) SemanticAnalyzer will generate a WriteEntity for the target table since it doesn't know/check if the read and write are of the same table in "insert ...
protected void	validateTargetTable(Table mTable) Assert it supports Acid write.

Methods inherited from class org.apache.hadoop.hive.ql.parse.CalcitePlanner

createPlanner, createPlannerAction, fixUpAfterCbo, genLogicalPlan, getOptimizedSql, getTableObjectByName, resetCalciteConfiguration, warmup

Methods inherited from class org.apache.hadoop.hive.ql.parse.SemanticAnalyzer

addPartitionColsToInsert, addPartitionColsToInsert, allowOutputMultipleTimes, analyzeCreateView, compilePlan, enableColumnStatsCollecting, executeUnparseTranlations, forViewCreation, generateErrorMessage, genExprNodeDesc, genFileSinkPlan, getAcidAnalyzeTable, getAliasId, getAllInputs, getAllOutputs, getAllRootTasks, getColNameToDefaultValueMap, getColumnInternalName, getDefaultConstraints, getDummyTable, getFullTableNameForSQL, getGenericUDAFEvaluator, getGenericUDAFEvaluator2, getGenericUDAFInfo, getGenericUDAFInfo2, getInvalidAutomaticRewritingMaterializationReason, getInvalidResultCacheReason, getLoadFileWork, getLoadTableWork, getOpContext, getOriginalResultSchema, getParseContext, getQB, getResultSchema, getSinkOp, getTableObjectByName, getTargetTable, getTopOps, getViewAliasToInput, groupByDescModeToUDAFMode, hasGroupBySibling, init, isConstantParameterInAggregationParameters, isInsertInto, isSelectDistinct, isValidAutomaticRewritingMaterialization, isValidQueryCaching, parseSelect, processNoScanCommand, processTableColumnNames, reset, rewriteGroupingFunctionAST, saveViewDefinition, setAST, setBit, setInvalidAutomaticRewritingMaterializationReason, setInvalidResultCacheReason, setLoadFileWork, setLoadTableWork, startAnalysis, translateFieldDesc, updating, validate

Methods inherited from class org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer

addPropertyReadEntry, analyze, analyzeStoredAdDirs, charSetString, checkColumnName, constructListBucketingCtx, createFetchTask, createHiveDB, endAnalysis, escapeSQLString, findTabRefIdxs, getAcidDdlDesc, getAcidFileSinks, getAutoCommitValue, getCacheUsage, getCboInfo, getColumnAccessInfo, getColumnNames, getColumnNamesOrder, getColumns, getColumns, getColumns, getCtx, getDatabase, getDatabase, getDataConnector, getDataConnector, getDb, getDbTableNamePair, getFetchTask, getIdToTableNameMap, getInputs, getLineageInfo, getOutputs, getPartitionSpecs, getPartSpec, getProps, getQualifiedTableName, getQualifiedTableName, getQueryProperties, getQueryState, getRootTasks, getSimpleTableNameBase, getTable, getTable, getTable, getTable, getTable, getTable, getTableAccessInfo, getTableAlias, getTxnMgr, getTypeStringFromAST, getUnescapedName, getUnescapedName, getUnescapedUnqualifiedTableName, getUpdateColumnAccessInfo, getValidatedPartSpec, hasTransactionalInQuery, initCtx, isPrepareQuery, isValidPrefixSpec, raiseWrongType, readProps, setAutoCommitValue, setCacheUsage, setColumnAccessInfo, setFetchTask, setLineageInfo, setTableAccessInfo, setUpdateColumnAccessInfo, skipAuthorization, stripQuotes, toReadEntity, toReadEntity, toReadEntity, toWriteEntity, toWriteEntity, toWriteEntity, tryQualifyPath, unescapeIdentifier, unescapeSQLString, validatePartColumnType, validatePartSpec, validateUnsupportedPartitionClause

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

protected static final org.slf4j.Logger LOG

useSuper

protected boolean useSuper

INDENT

protected static final String INDENT

See Also:

Constant Field Values

DELETE_PREFIX

public static final String DELETE_PREFIX

See Also:

Constant Field Values

SUB_QUERY_ALIAS

public static final String SUB_QUERY_ALIAS

See Also:

Constant Field Values

Method Detail

analyzeInternal

public void analyzeInternal(ASTNode tree)
                     throws SemanticException

Overrides:

analyzeInternal in class CalcitePlanner

Throws:

SemanticException

getTargetTableNode

protected abstract ASTNode getTargetTableNode(ASTNode tree)

analyze

protected abstract void analyze(ASTNode tree,
                                Table table,
                                ASTNode tableName)
                         throws SemanticException

Throws:

SemanticException

analyzeRewrittenTree

public void analyzeRewrittenTree(ASTNode rewrittenTree,
                                 Context rewrittenCtx)
                          throws SemanticException

Throws:

SemanticException

addColsToSelect

protected void addColsToSelect(List<org.apache.hadoop.hive.metastore.api.FieldSchema> partCols,
                               StringBuilder rewrittenQueryStr)
                        throws SemanticException

Append list of partition columns to Insert statement, i.e. the 2nd set of partCol1,partCol2 INSERT INTO T PARTITION(partCol1,partCol2...) SELECT col1, ... partCol1,partCol2...

Throws:

SemanticException

addColsToSelect

protected void addColsToSelect(List<org.apache.hadoop.hive.metastore.api.FieldSchema> cols,
                               StringBuilder rewrittenQueryStr,
                               ASTNode target)
                        throws SemanticException

Append list of columns to rewritten statement.

Throws:

SemanticException

addColsToSelect

protected void addColsToSelect(List<org.apache.hadoop.hive.metastore.api.FieldSchema> cols,
                               StringBuilder rewrittenQueryStr,
                               String alias)

Append list of columns to rewritten statement. Column names are qualified with the specified alias and quoted.

addPartitionColsAsValues

protected void addPartitionColsAsValues(List<org.apache.hadoop.hive.metastore.api.FieldSchema> partCols,
                                        String alias,
                                        List<String> values)

checkValidSetClauseTarget

protected void checkValidSetClauseTarget(ASTNode colName,
                                         Table targetTable)
                                  throws SemanticException

Assert that we are not asked to update a bucketing column or partition column.

Parameters:

colName - it's the A in "SET A = B"

Throws:

SemanticException

findLHSofAssignment

protected ASTNode findLHSofAssignment(ASTNode assignment)

collectSetColumnsAndExpressions

protected Map<String,ASTNode> collectSetColumnsAndExpressions(ASTNode setClause,
                                                              Set<String> setRCols,
                                                              Table targetTable)
                                                       throws SemanticException

Throws:

SemanticException

getTable

protected static Table getTable(ASTNode tabRef,
                                Hive db,
                                boolean throwException)
                         throws SemanticException

Parameters:

throwException - if false, return null if table doesn't exist, else throw

Throws:

SemanticException

markReadEntityForUpdate

protected void markReadEntityForUpdate()

Walk through all our inputs and set them to note that this read is part of an update or a delete.

setUpAccessControlInfoForUpdate

protected void setUpAccessControlInfoForUpdate(Table mTable,
                                               Map<String,ASTNode> setCols)

For updates, we need to set the column access info so that it contains information on the columns we are updating. (But not all the columns of the target table even though the rewritten query writes all columns of target table since that is an implementation detail).

parseRewrittenQuery

protected RewriteSemanticAnalyzer.ReparseResult parseRewrittenQuery(StringBuilder rewrittenQueryStr,
                                                                    String originalQuery)
                                                             throws SemanticException

Parse the newly generated SQL statement to get a new AST.

Throws:

SemanticException

validateTargetTable

protected void validateTargetTable(Table mTable)
                            throws SemanticException

Assert it supports Acid write.

Throws:

SemanticException

updateOutputs

protected void updateOutputs(Table targetTable)

SemanticAnalyzer will generate a WriteEntity for the target table since it doesn't know/check if the read and write are of the same table in "insert ... select ....". Since DbTxnManager uses Read/WriteEntity objects to decide which locks to acquire, we get more concurrency if we have change the table WriteEntity to a set of partition WriteEntity objects based on ReadEntity objects computed for this table.

getSimpleTableName

protected String getSimpleTableName(ASTNode n)
                             throws SemanticException

Returns the table name to use in the generated query preserving original quotes/escapes if any.

Throws:

SemanticException

See Also:

SemanticAnalyzer.getFullTableNameForSQL(ASTNode)

patchProjectionForUpdate

protected void patchProjectionForUpdate(ASTNode insertBranch,
                                        Map<Integer,ASTNode> setColExprs)

createRewrittenQueryStrBuilder

protected StringBuilder createRewrittenQueryStrBuilder()

appendTarget

protected void appendTarget(StringBuilder rewrittenQueryStr,
                            ASTNode target,
                            String targetName)

isAliased

protected boolean isAliased(ASTNode n)

appendInsertBranch

protected void appendInsertBranch(StringBuilder rewrittenQueryStr,
                                  String hintStr,
                                  List<String> values)

appendSortBy

protected void appendSortBy(StringBuilder rewrittenQueryStr,
                            List<String> keys)

getMatchedText

protected String getMatchedText(ASTNode n)

This allows us to take an arbitrary ASTNode and turn it back into SQL that produced it without needing to understand what it is (except for QuotedIdentifiers).

getColumnAppender

protected RewriteSemanticAnalyzer.ColumnAppender getColumnAppender(String subQueryAlias)