org.apache.hadoop.hive.ql.parse

Class SemanticAnalyzer

java.lang.Object

org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer

org.apache.hadoop.hive.ql.parse.SemanticAnalyzer

Direct Known Subclasses:

CalcitePlanner, ColumnStatsSemanticAnalyzer, ExecuteStatementAnalyzer, LoadSemanticAnalyzer

public class SemanticAnalyzer
extends BaseSemanticAnalyzer

Implementation of the semantic analyzer. It generates the query plan. There are other specific semantic analyzers for some hive operations such as various analyzers for DDL commands.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SemanticAnalyzer.GenericUDAFInfo Class to store GenericUDAF related information.
protected static class	SemanticAnalyzer.MaterializationRebuildMode

Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer

BaseSemanticAnalyzer.AnalyzeRewriteContext, BaseSemanticAnalyzer.TableSpec

Field Summary

Fields

Modifier and Type	Field and Description
protected Map<String,org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.CTEClause>	aliasToCTEs
protected BaseSemanticAnalyzer.AnalyzeRewriteContext	analyzeRewrite
protected ASTNode	ast
protected String	autogenColAliasPrfxLbl
protected CreateMaterializedViewDesc	createVwDesc
protected boolean	defaultJoinMerge
protected boolean	disableJoinMerge
static String	DUMMY_DATABASE
static String	DUMMY_TABLE
protected boolean	forViewCreation
protected SemanticAnalyzer.MaterializationRebuildMode	mvRebuildMode
protected Map<Operator<? extends OperatorDesc>,OpParseContext>	opParseCtx
protected List<org.apache.hadoop.hive.metastore.api.FieldSchema>	originalResultSchema
protected Map<String,PrunedPartitionList>	prunedPartitions
protected List<org.apache.hadoop.hive.metastore.api.FieldSchema>	resultSchema
static String	SUBQUERY_TAG_1
static String	SUBQUERY_TAG_2
protected Map<String,TableScanOperator>	topOps
protected Map<String,TableScanOperator>	topOpsCopy This is required by prepare/execute statement Original operator tree { @link topOps} shape is changed when going through transformations and task generation, as a result original operator tree can not be used later to e.g.
protected UnparseTranslator	unparseTranslator
static String	VALUES_TMP_TABLE_NAME_PREFIX
protected ASTNode	viewSelect

Fields inherited from class org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer

acidFileSinks, cacheUsage, cContext, columnAccessInfo, conf, console, ctx, db, fetchTask, idToTableNameMap, inputs, linfo, LOG, outputs, prepareQuery, queryProperties, queryState, rootTasks, STATIC_LOG, tableAccessInfo, transactionalInQuery, txnManager, updateColumnAccessInfo, userSuppliedFunctions

Constructor Summary

Constructors

Constructor and Description
SemanticAnalyzer(QueryState queryState)

Method Summary

Modifier and Type	Method and Description
protected void	addPartitionColsToInsert(List<org.apache.hadoop.hive.metastore.api.FieldSchema> partCols, Map<String,String> partSpec, StringBuilder rewrittenQueryStr) Append list of partition columns to Insert statement.
protected void	addPartitionColsToInsert(List<org.apache.hadoop.hive.metastore.api.FieldSchema> partCols, StringBuilder rewrittenQueryStr) Append list of partition columns to Insert statement, i.e.
protected boolean	allowOutputMultipleTimes()
protected ASTNode	analyzeCreateView(ASTNode ast, QB qb, org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.PlannerContext plannerCtx)
void	analyzeInternal(ASTNode ast)
protected void	compilePlan(ParseContext pCtx)
protected boolean	enableColumnStatsCollecting()
void	executeUnParseTranslations() Unparses the analyzed statement
void	forViewCreation(String fqViewName)
static String	generateErrorMessage(ASTNode ast, String message)
ExprNodeDesc	genExprNodeDesc(ASTNode expr, RowResolver input) Generates an expression node descriptor for the expression with TypeCheckCtx.
protected Operator	genFileSinkPlan(String dest, QB qb, Operator input)
WriteEntity	getAcidAnalyzeTable()
protected String	getAliasId(String alias, QB qb)
Set<ReadEntity>	getAllInputs()
Set<WriteEntity>	getAllOutputs()
List<Task<?>>	getAllRootTasks()
protected Map<String,String>	getColNameToDefaultValueMap(Table tbl)
static String	getColumnInternalName(int pos)
protected List<String>	getDefaultConstraints(Table tbl, List<String> targetSchema) This method creates a list of default constraints which corresponds to given schema (targetSchema) or target table's column schema (if targetSchema is null)
Table	getDummyTable()
static GenericUDAFEvaluator	getGenericUDAFEvaluator(String aggName, List<ExprNodeDesc> aggParameters, ASTNode aggTree, boolean isDistinct, boolean isAllColumns) Returns the GenericUDAFEvaluator for the aggregation.
static GenericUDAFEvaluator	getGenericUDAFEvaluator2(String aggName, List<ObjectInspector> aggParameterOIs, ASTNode aggTree, boolean isDistinct, boolean isAllColumns)
static SemanticAnalyzer.GenericUDAFInfo	getGenericUDAFInfo(GenericUDAFEvaluator evaluator, GenericUDAFEvaluator.Mode emode, List<ExprNodeDesc> aggParameters) Returns the GenericUDAFInfo struct for the aggregation.
static SemanticAnalyzer.GenericUDAFInfo	getGenericUDAFInfo2(GenericUDAFEvaluator evaluator, GenericUDAFEvaluator.Mode emode, List<ObjectInspector> aggOIs)
String	getInvalidResultCacheReason()
List<LoadFileDesc>	getLoadFileWork()
List<LoadTableDesc>	getLoadTableWork()
MaterializationValidationResult	getMaterializationValidationResult()
CompilationOpContext	getOpContext()
List<org.apache.hadoop.hive.metastore.api.FieldSchema>	getOriginalResultSchema()
ParseContext	getParseContext()
QB	getQB()
List<org.apache.hadoop.hive.metastore.api.FieldSchema>	getResultSchema()
Operator	getSinkOp()
Table	getTableObjectByName(String tableName)
protected Table	getTableObjectByName(String tableName, boolean throwException)
protected Table	getTargetTable(QB qb, String dest)
Map<String,TableScanOperator>	getTopOps()
Map<String,ReadEntity>	getViewAliasToInput()
static GenericUDAFEvaluator.Mode	groupByDescModeToUDAFMode(GroupByDesc.Mode mode, boolean isDistinct)
protected boolean	hasGroupBySibling(ASTNode selectExpr)
void	init(boolean clearPartsCache)
static ExprNodeDesc	isConstantParameterInAggregationParameters(String internalName, List<ExprNodeDesc> reduceValues) Check if the given internalName represents a constant parameter in aggregation parameters of an aggregation tree.
protected boolean	isInsertInto(QBParseInfo qbp, String dest) This method figures out if current AST is for INSERT INTO
protected boolean	isSelectDistinct(ASTNode expr)
boolean	isValidQueryCaching()
static List<WindowingSpec.WindowExpressionSpec>	parseSelect(String selectExprStr)
protected void	processNoScanCommand(ASTNode tree) process analyze ...
protected List<String>	processTableColumnNames(ASTNode tabColName, String tableName)
protected void	reset(boolean clearCache)
protected ASTNode	rewriteGroupingFunctionAST(List<ASTNode> grpByAstExprs, ASTNode targetNode, boolean noneSet)
protected void	saveViewDefinition()
protected void	setAST(ASTNode newAST)
static long	setBit(long bitmap, int bitIdx)
void	setInvalidResultCacheReason(String invalidQueryMaterializationReason)
void	setLoadFileWork(List<LoadFileDesc> loadFileWork)
void	setLoadTableWork(List<LoadTableDesc> tblWork)
void	setMaterializationValidationResult(MaterializationValidationResult materializationValidationResult)
void	startAnalysis() Called when we start analysis of a query.
static Map<ASTNode,String>	translateFieldDesc(ASTNode node, HiveConf conf)
protected boolean	updating(String destination)
void	validate()

Methods inherited from class org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer

addPropertyReadEntry, analyze, analyzeStoredAdDirs, charSetString, checkColumnName, constructListBucketingCtx, createFetchTask, createHiveDB, endAnalysis, escapeSQLString, findTabRefIdxs, getAcidDdlDesc, getAcidFileSinks, getAutoCommitValue, getCacheUsage, getCboInfo, getColumnAccessInfo, getColumnNames, getColumnNamesOrder, getColumns, getColumns, getColumns, getCtx, getDatabase, getDatabase, getDataConnector, getDataConnector, getDb, getDbTableNamePair, getFetchTask, getFullTableNameForSQL, getIdToTableNameMap, getInputs, getLineageInfo, getOutputs, getPartitionSpecs, getPartSpec, getProps, getQualifiedTableName, getQualifiedTableName, getQueryProperties, getQueryState, getRootTasks, getSimpleTableNameBase, getTable, getTable, getTable, getTable, getTable, getTable, getTableAccessInfo, getTableAlias, getTxnMgr, getTypeStringFromAST, getUnescapedName, getUnescapedName, getUnescapedUnqualifiedTableName, getUpdateColumnAccessInfo, getUserSuppliedFunctions, getValidatedPartSpec, hasTransactionalInQuery, initCtx, isPrepareQuery, isValidPrefixSpec, processOrderSpec, raiseWrongType, readProps, setAutoCommitValue, setCacheUsage, setColumnAccessInfo, setFetchTask, setLineageInfo, setTableAccessInfo, setUpdateColumnAccessInfo, skipAuthorization, stripQuotes, toReadEntity, toReadEntity, toReadEntity, toWriteEntity, toWriteEntity, toWriteEntity, tryQualifyPath, unescapeIdentifier, unescapeSQLString, validatePartColumnType, validatePartSpec, validateUnsupportedPartitionClause

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DUMMY_DATABASE

public static final String DUMMY_DATABASE

See Also:

Constant Field Values

DUMMY_TABLE

public static final String DUMMY_TABLE

See Also:

Constant Field Values

SUBQUERY_TAG_1

public static final String SUBQUERY_TAG_1

See Also:

Constant Field Values

SUBQUERY_TAG_2

public static final String SUBQUERY_TAG_2

See Also:

Constant Field Values

VALUES_TMP_TABLE_NAME_PREFIX

public static final String VALUES_TMP_TABLE_NAME_PREFIX

See Also:

Constant Field Values

topOps

protected Map<String,TableScanOperator> topOps

opParseCtx

protected Map<Operator<? extends OperatorDesc>,OpParseContext> opParseCtx

ast

protected ASTNode ast

prunedPartitions

protected Map<String,PrunedPartitionList> prunedPartitions

resultSchema

protected List<org.apache.hadoop.hive.metastore.api.FieldSchema> resultSchema

originalResultSchema

protected List<org.apache.hadoop.hive.metastore.api.FieldSchema> originalResultSchema

createVwDesc

protected CreateMaterializedViewDesc createVwDesc

viewSelect

protected ASTNode viewSelect

unparseTranslator

protected final UnparseTranslator unparseTranslator

autogenColAliasPrfxLbl

protected final String autogenColAliasPrfxLbl

forViewCreation

protected boolean forViewCreation

mvRebuildMode

protected SemanticAnalyzer.MaterializationRebuildMode mvRebuildMode

disableJoinMerge

protected volatile boolean disableJoinMerge

defaultJoinMerge

protected final boolean defaultJoinMerge

topOpsCopy

protected Map<String,TableScanOperator> topOpsCopy

This is required by prepare/execute statement Original operator tree { @link topOps} shape is changed when going through transformations and task generation, as a result original operator tree can not be used later to e.g. regenerate tasks or re-running physical transformations. Therefore we need to make a copy and cache it after operator tree is generated.

aliasToCTEs

protected final Map<String,org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.CTEClause> aliasToCTEs

analyzeRewrite

protected BaseSemanticAnalyzer.AnalyzeRewriteContext analyzeRewrite

Constructor Detail

SemanticAnalyzer

public SemanticAnalyzer(QueryState queryState)
                 throws SemanticException

Throws:

SemanticException

Method Detail

reset

protected void reset(boolean clearCache)

Overrides:

reset in class BaseSemanticAnalyzer

getParseContext

public ParseContext getParseContext()

Overrides:

getParseContext in class BaseSemanticAnalyzer

getOpContext

public CompilationOpContext getOpContext()

isInsertInto

protected boolean isInsertInto(QBParseInfo qbp,
                               String dest)

This method figures out if current AST is for INSERT INTO

Parameters:

qbp - qbParseInfo

dest - destination clause

Returns:

true or false

getDefaultConstraints

protected List<String> getDefaultConstraints(Table tbl,
                                             List<String> targetSchema)
                                      throws SemanticException

This method creates a list of default constraints which corresponds to given schema (targetSchema) or target table's column schema (if targetSchema is null)

Parameters:

tbl -

targetSchema -

Returns:

List of default constraints (including NULL if there is no default)

Throws:

SemanticException

getColNameToDefaultValueMap

protected Map<String,String> getColNameToDefaultValueMap(Table tbl)
                                                  throws SemanticException

Throws:

SemanticException

generateErrorMessage

public static String generateErrorMessage(ASTNode ast,
                                          String message)

setAST

protected void setAST(ASTNode newAST)

getAllRootTasks

public List<Task<?>> getAllRootTasks()

Overrides:

getAllRootTasks in class BaseSemanticAnalyzer

getAllInputs

public Set<ReadEntity> getAllInputs()

Overrides:

getAllInputs in class BaseSemanticAnalyzer

getAllOutputs

public Set<WriteEntity> getAllOutputs()

Overrides:

getAllOutputs in class BaseSemanticAnalyzer

processTableColumnNames

protected List<String> processTableColumnNames(ASTNode tabColName,
                                               String tableName)
                                        throws SemanticException

Throws:

SemanticException

rewriteGroupingFunctionAST

protected ASTNode rewriteGroupingFunctionAST(List<ASTNode> grpByAstExprs,
                                             ASTNode targetNode,
                                             boolean noneSet)

getColumnInternalName

public static String getColumnInternalName(int pos)

setBit

public static long setBit(long bitmap,
                          int bitIdx)

hasGroupBySibling

protected boolean hasGroupBySibling(ASTNode selectExpr)

isSelectDistinct

protected boolean isSelectDistinct(ASTNode expr)

getGenericUDAFEvaluator

public static GenericUDAFEvaluator getGenericUDAFEvaluator(String aggName,
                                                           List<ExprNodeDesc> aggParameters,
                                                           ASTNode aggTree,
                                                           boolean isDistinct,
                                                           boolean isAllColumns)
                                                    throws SemanticException

Returns the GenericUDAFEvaluator for the aggregation. This is called once for each GroupBy aggregation.

Throws:

SemanticException

getGenericUDAFEvaluator2

public static GenericUDAFEvaluator getGenericUDAFEvaluator2(String aggName,
                                                            List<ObjectInspector> aggParameterOIs,
                                                            ASTNode aggTree,
                                                            boolean isDistinct,
                                                            boolean isAllColumns)
                                                     throws SemanticException

Throws:

SemanticException

getGenericUDAFInfo

public static SemanticAnalyzer.GenericUDAFInfo getGenericUDAFInfo(GenericUDAFEvaluator evaluator,
                                                                  GenericUDAFEvaluator.Mode emode,
                                                                  List<ExprNodeDesc> aggParameters)
                                                           throws SemanticException

Returns the GenericUDAFInfo struct for the aggregation.

Parameters:

evaluator -

emode -

aggParameters - The exprNodeDesc of the original parameters

Returns:

GenericUDAFInfo

Throws:

SemanticException - when the UDAF is not found or has problems.

getGenericUDAFInfo2

public static SemanticAnalyzer.GenericUDAFInfo getGenericUDAFInfo2(GenericUDAFEvaluator evaluator,
                                                                   GenericUDAFEvaluator.Mode emode,
                                                                   List<ObjectInspector> aggOIs)
                                                            throws SemanticException

Throws:

SemanticException

groupByDescModeToUDAFMode

public static GenericUDAFEvaluator.Mode groupByDescModeToUDAFMode(GroupByDesc.Mode mode,
                                                                  boolean isDistinct)

isConstantParameterInAggregationParameters

public static ExprNodeDesc isConstantParameterInAggregationParameters(String internalName,
                                                                      List<ExprNodeDesc> reduceValues)

Check if the given internalName represents a constant parameter in aggregation parameters of an aggregation tree. This method is only invoked when map-side aggregation is not involved. In this case, every parameter in every aggregation tree should already have a corresponding ColumnInfo, which is generated when the corresponding ReduceSinkOperator of the GroupByOperator being generating is generated. If we find that this parameter is a constant parameter, we will return the corresponding ExprNodeDesc in reduceValues, and we will not need to use a new ExprNodeColumnDesc, which can not be treated as a constant parameter, for this parameter (since the writableObjectInspector of a ExprNodeColumnDesc will not be a instance of ConstantObjectInspector).

Parameters:

reduceValues - value columns of the corresponding ReduceSinkOperator

internalName - the internal name of this parameter

Returns:

the ExprNodeDesc of the constant parameter if the given internalName represents a constant parameter; otherwise, return null

getTargetTable

protected Table getTargetTable(QB qb,
                               String dest)
                        throws SemanticException

Throws:

SemanticException

genFileSinkPlan

protected Operator genFileSinkPlan(String dest,
                                   QB qb,
                                   Operator input)
                            throws SemanticException

Throws:

SemanticException

enableColumnStatsCollecting

protected boolean enableColumnStatsCollecting()

allowOutputMultipleTimes

protected boolean allowOutputMultipleTimes()

getAliasId

protected String getAliasId(String alias,
                            QB qb)

getDummyTable

public Table getDummyTable()
                    throws SemanticException

Throws:

SemanticException

init

public void init(boolean clearPartsCache)

Overrides:

init in class BaseSemanticAnalyzer

analyzeInternal

public void analyzeInternal(ASTNode ast)
                     throws SemanticException

Specified by:

analyzeInternal in class BaseSemanticAnalyzer

Throws:

SemanticException

getTableObjectByName

protected Table getTableObjectByName(String tableName,
                                     boolean throwException)
                              throws HiveException

Throws:

HiveException

getTableObjectByName

public Table getTableObjectByName(String tableName)
                           throws HiveException

Throws:

HiveException

compilePlan

protected void compilePlan(ParseContext pCtx)
                    throws SemanticException

Throws:

SemanticException

getResultSchema

public List<org.apache.hadoop.hive.metastore.api.FieldSchema> getResultSchema()

Overrides:

getResultSchema in class BaseSemanticAnalyzer

Returns:

the schema for the fields which will be produced when the statement is executed, or null if not known

getOriginalResultSchema

public List<org.apache.hadoop.hive.metastore.api.FieldSchema> getOriginalResultSchema()

saveViewDefinition

protected void saveViewDefinition()
                           throws SemanticException

Throws:

SemanticException

genExprNodeDesc

public ExprNodeDesc genExprNodeDesc(ASTNode expr,
                                    RowResolver input)
                             throws SemanticException

Generates an expression node descriptor for the expression with TypeCheckCtx.

Throws:

SemanticException

translateFieldDesc

public static final Map<ASTNode,String> translateFieldDesc(ASTNode node,
                                                           HiveConf conf)

validate

public void validate()
              throws SemanticException

Overrides:

validate in class BaseSemanticAnalyzer

Throws:

SemanticException

analyzeCreateView

protected ASTNode analyzeCreateView(ASTNode ast,
                                    QB qb,
                                    org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.PlannerContext plannerCtx)
                             throws SemanticException

Throws:

SemanticException

processNoScanCommand

protected void processNoScanCommand(ASTNode tree)
                             throws SemanticException

process analyze ... noscan command

Parameters:

tree -

Throws:

SemanticException

getQB

public QB getQB()

parseSelect

public static List<WindowingSpec.WindowExpressionSpec> parseSelect(String selectExprStr)
                                                            throws SemanticException

Throws:

SemanticException

updating

protected boolean updating(String destination)

getLoadFileWork

public List<LoadFileDesc> getLoadFileWork()

getLoadTableWork

public List<LoadTableDesc> getLoadTableWork()

setLoadFileWork

public void setLoadFileWork(List<LoadFileDesc> loadFileWork)

setLoadTableWork

public void setLoadTableWork(List<LoadTableDesc> tblWork)

getMaterializationValidationResult

public MaterializationValidationResult getMaterializationValidationResult()

setMaterializationValidationResult

public void setMaterializationValidationResult(MaterializationValidationResult materializationValidationResult)

getInvalidResultCacheReason

public String getInvalidResultCacheReason()

setInvalidResultCacheReason

public void setInvalidResultCacheReason(String invalidQueryMaterializationReason)

isValidQueryCaching

public boolean isValidQueryCaching()

forViewCreation

public void forViewCreation(String fqViewName)

getTopOps

public Map<String,TableScanOperator> getTopOps()

getViewAliasToInput

public Map<String,ReadEntity> getViewAliasToInput()

getSinkOp

public Operator getSinkOp()

addPartitionColsToInsert

protected void addPartitionColsToInsert(List<org.apache.hadoop.hive.metastore.api.FieldSchema> partCols,
                                        StringBuilder rewrittenQueryStr)

Append list of partition columns to Insert statement, i.e. the 1st set of partCol1,partCol2 INSERT INTO T PARTITION(partCol1,partCol2...) SELECT col1, ... partCol1,partCol2...

addPartitionColsToInsert

protected void addPartitionColsToInsert(List<org.apache.hadoop.hive.metastore.api.FieldSchema> partCols,
                                        Map<String,String> partSpec,
                                        StringBuilder rewrittenQueryStr)

Append list of partition columns to Insert statement. If user specified partition spec, then use it to get/set the value for partition column else use dynamic partition mode with no value. Static partition mode: INSERT INTO T PARTITION(partCol1=val1,partCol2...) SELECT col1, ... partCol1,partCol2... Dynamic partition mode: INSERT INTO T PARTITION(partCol1,partCol2...) SELECT col1, ... partCol1,partCol2...

getAcidAnalyzeTable

public WriteEntity getAcidAnalyzeTable()

Overrides:

getAcidAnalyzeTable in class BaseSemanticAnalyzer

executeUnParseTranslations

public void executeUnParseTranslations()

Description copied from class: BaseSemanticAnalyzer

Unparses the analyzed statement

Overrides:

executeUnParseTranslations in class BaseSemanticAnalyzer

startAnalysis

public void startAnalysis()

Description copied from class: BaseSemanticAnalyzer

Called when we start analysis of a query.

Overrides:

startAnalysis in class BaseSemanticAnalyzer