org.apache.hadoop.hive.ql.ddl.view.materialized.alter.rebuild

Class AlterMaterializedViewRebuildAnalyzer

java.lang.Object

org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer

org.apache.hadoop.hive.ql.parse.SemanticAnalyzer

org.apache.hadoop.hive.ql.parse.CalcitePlanner

org.apache.hadoop.hive.ql.ddl.view.materialized.alter.rebuild.AlterMaterializedViewRebuildAnalyzer

public class AlterMaterializedViewRebuildAnalyzer
extends CalcitePlanner

Semantic analyzer for alter materialized view rebuild commands. This subclass of SemanticAnalyzer generates a plan which is derived from the materialized view definition query plan.
Steps:

• Take the Calcite plan of the materialized view definition query.
• Using the snapshot data in materialized view metadata insert A HiveFilter operator on top of each HiveTableScan operator. The condition has a predicate like ROW_ID.writeid <= high_watermark This step is done by HiveAugmentMaterializationRule or HiveAugmentSnapshotMaterializationRule. The resulting plan should produce the current result of the materialized view, the one which was created at last rebuild.
• Transform the original view definition query plan using Union rewrite or Union rewrite with aggregate and the augmented plan. The result plan has a HiveUnion operator on top with two branches
  • Scan the materialize view for existing records
  • A plan which is derived from the augmented materialized view definition query plan. This produces the newly inserted records
• Transform the plan into incremental rebuild plan if possible:
  • The materialized view definition query has aggregate and base tables has insert operations only. HiveAggregateInsertIncrementalRewritingRule
  • The materialized view definition query hasn't got aggregate and base tables has insert operations only. HiveJoinInsertIncrementalRewritingRule
  • The materialized view definition query has aggregate and any base tables has delete operations. HiveAggregateInsertDeleteIncrementalRewritingRule
  • The materialized view definition query hasn't got aggregate and any base tables has delete operations. Incremental rebuild is not possible because all records from all source tables need a unique identifier to join it with the corresponding record exists in the view. ROW__ID can not be used because it's writedId component is changed at delete and unique and primary key constraints are not enforced in Hive.
  When any base tables has delete operations the HiveTableScan operators are fetching the deleted rows too and HiveRowIsDeletedPropagator ensures that extra filter conditions are added to address these.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected class	AlterMaterializedViewRebuildAnalyzer.MVRebuildCalcitePlannerAction

Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.parse.CalcitePlanner

CalcitePlanner.ASTSearcher, CalcitePlanner.CalcitePlannerAction, CalcitePlanner.InputContext, CalcitePlanner.PreCboCtx

Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.parse.SemanticAnalyzer

SemanticAnalyzer.GenericUDAFInfo, SemanticAnalyzer.MaterializationRebuildMode

Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer

BaseSemanticAnalyzer.AnalyzeRewriteContext, BaseSemanticAnalyzer.TableSpec

Field Summary

Fields

Modifier and Type	Field and Description
protected Table	mvTable

Fields inherited from class org.apache.hadoop.hive.ql.parse.SemanticAnalyzer

aliasToCTEs, analyzeRewrite, ast, autogenColAliasPrfxLbl, createVwDesc, defaultJoinMerge, disableJoinMerge, DUMMY_DATABASE, DUMMY_TABLE, forViewCreation, mvRebuildMode, opParseCtx, originalResultSchema, prunedPartitions, resultSchema, SUBQUERY_TAG_1, SUBQUERY_TAG_2, topOps, topOpsCopy, unparseTranslator, VALUES_TMP_TABLE_NAME_PREFIX, viewSelect

Fields inherited from class org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer

acidFileSinks, cacheUsage, cContext, columnAccessInfo, conf, console, ctx, db, fetchTask, idToTableNameMap, inputs, linfo, outputs, prepareQuery, queryProperties, queryState, rootTasks, STATIC_LOG, tableAccessInfo, transactionalInQuery, txnManager, updateColumnAccessInfo, userSuppliedFunctions

Constructor Summary

Constructors

Constructor and Description
AlterMaterializedViewRebuildAnalyzer(QueryState queryState)

Method Summary

Modifier and Type	Method and Description
protected boolean	allowOutputMultipleTimes()
void	analyzeInternal(ASTNode root)
protected org.apache.calcite.tools.Frameworks.PlannerAction<org.apache.calcite.rel.RelNode>	createPlannerAction(Map<String,PrunedPartitionList> partitionCache, StatsSource statsSource, ColumnAccessInfo columnAccessInfo)
protected ASTNode	fixUpAfterCbo(ASTNode originalAst, ASTNode newAst, CalcitePlanner.PreCboCtx cboCtx)

Methods inherited from class org.apache.hadoop.hive.ql.parse.CalcitePlanner

createPlanner, genLogicalPlan, genRexNode, getOptimizedSql, getTableObjectByName, resetCalciteConfiguration, warmup

Methods inherited from class org.apache.hadoop.hive.ql.parse.SemanticAnalyzer

addPartitionColsToInsert, addPartitionColsToInsert, analyzeCreateView, compilePlan, enableColumnStatsCollecting, executeUnParseTranslations, forViewCreation, generateErrorMessage, genExprNodeDesc, genFileSinkPlan, getAcidAnalyzeTable, getAliasId, getAllInputs, getAllOutputs, getAllRootTasks, getColNameToDefaultValueMap, getColumnInternalName, getDefaultConstraints, getDummyTable, getGenericUDAFEvaluator, getGenericUDAFEvaluator2, getGenericUDAFInfo, getGenericUDAFInfo2, getInvalidResultCacheReason, getLoadFileWork, getLoadTableWork, getMaterializationValidationResult, getOpContext, getOriginalResultSchema, getParseContext, getQB, getResultSchema, getSinkOp, getTableObjectByName, getTargetTable, getTopOps, getViewAliasToInput, groupByDescModeToUDAFMode, hasGroupBySibling, init, isConstantParameterInAggregationParameters, isInsertInto, isSelectDistinct, isValidQueryCaching, parseSelect, processNoScanCommand, processTableColumnNames, reset, rewriteGroupingFunctionAST, saveViewDefinition, setAST, setBit, setInvalidResultCacheReason, setLoadFileWork, setLoadTableWork, setMaterializationValidationResult, startAnalysis, translateFieldDesc, updating, validate

Methods inherited from class org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer

addPropertyReadEntry, analyze, analyzeStoredAdDirs, charSetString, checkColumnName, constructListBucketingCtx, createFetchTask, createHiveDB, endAnalysis, escapeSQLString, findTabRefIdxs, getAcidDdlDesc, getAcidFileSinks, getAutoCommitValue, getCacheUsage, getCboInfo, getColumnAccessInfo, getColumnNames, getColumnNamesOrder, getColumns, getColumns, getColumns, getCtx, getDatabase, getDatabase, getDataConnector, getDataConnector, getDb, getDbTableNamePair, getFetchTask, getFullTableNameForSQL, getIdToTableNameMap, getInputs, getLineageInfo, getOutputs, getPartitionSpecs, getPartSpec, getProps, getQualifiedTableName, getQualifiedTableName, getQueryProperties, getQueryState, getRootTasks, getSimpleTableNameBase, getTable, getTable, getTable, getTable, getTable, getTable, getTableAccessInfo, getTableAlias, getTxnMgr, getTypeStringFromAST, getUnescapedName, getUnescapedName, getUnescapedUnqualifiedTableName, getUpdateColumnAccessInfo, getUserSuppliedFunctions, getValidatedPartSpec, hasTransactionalInQuery, initCtx, isPrepareQuery, isValidPrefixSpec, processOrderSpec, raiseWrongType, readProps, setAutoCommitValue, setCacheUsage, setColumnAccessInfo, setFetchTask, setLineageInfo, setTableAccessInfo, setUpdateColumnAccessInfo, skipAuthorization, stripQuotes, toReadEntity, toReadEntity, toReadEntity, toWriteEntity, toWriteEntity, toWriteEntity, tryQualifyPath, unescapeIdentifier, unescapeSQLString, validatePartColumnType, validatePartSpec, validateUnsupportedPartitionClause

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

mvTable

protected Table mvTable

Constructor Detail

AlterMaterializedViewRebuildAnalyzer

public AlterMaterializedViewRebuildAnalyzer(QueryState queryState)
                                     throws SemanticException

Throws:

SemanticException

Method Detail

analyzeInternal

public void analyzeInternal(ASTNode root)
                     throws SemanticException

Overrides:

analyzeInternal in class CalcitePlanner

Throws:

SemanticException

createPlannerAction

protected org.apache.calcite.tools.Frameworks.PlannerAction<org.apache.calcite.rel.RelNode> createPlannerAction(Map<String,PrunedPartitionList> partitionCache,
                                                                                                                StatsSource statsSource,
                                                                                                                ColumnAccessInfo columnAccessInfo)

Overrides:

createPlannerAction in class CalcitePlanner

fixUpAfterCbo

protected ASTNode fixUpAfterCbo(ASTNode originalAst,
                                ASTNode newAst,
                                CalcitePlanner.PreCboCtx cboCtx)
                         throws SemanticException

Overrides:

fixUpAfterCbo in class CalcitePlanner

Throws:

SemanticException

allowOutputMultipleTimes

protected boolean allowOutputMultipleTimes()

Overrides:

allowOutputMultipleTimes in class SemanticAnalyzer