All Classes and Interfaces
Class
Description
This optimizer is responsible for changing the partition count of hash partitioned fragments
at runtime.
This class is responsible for re-optimizing the plan using exchange statistics in FTE.
This optimizer is needed for adaptive optimization in FTE.
For task retry mode, dynamic filters are removed from the JoinNode/SemiJoinNode
and rewritten into a DynamicFilterSourceNode below the remote exchange on the
build side of the join.
Transforms
Adds INTERMEDIATE aggregations between an un-grouped FINAL aggregation and its preceding
PARTIAL aggregation.
Group input data and produce a single block for each sequence of identical values.
Analysis for predicates such as
x IN (subquery) or x = SOME (subquery)An interface for anonymizing the plan in
PlanPrinterThis scalar function exists primarily to test lambda expression support.
Calculate the histogram approximately for topk elements based on the
Space-Saving algorithm and the Stream-Summary data structure
as described in:
Efficient Computation of Frequent and Top-k Elements in Data Streams
by Metwally, Agrawal, and Abbadi
A buffer that assigns pages to queues based on a first come, first served basis.
This class is inspired by com.google.common.io.Closer
Lightweight version of QueryInfo.
Lightweight version of QueryStats.
This class is a light representation of stage information -
StageInfo.This
WorkProcessorOperator adapter allows to adapt WorkProcessor operators
that do not require special input handling (e.g.
Aggregation function that approximates the frequency of the top-K elements.
This implementation assumes:
-There is only one join channel and it is of type bigint
-arrays used in the hash are always a power of 2.
Bind(values, targetFunction)
A set of values stored in preexisting blocks.
Value-based, immutable
byte[] equivalent.A resolver providing coercions and binary operators used for JSON path evaluation,
based on the operation type and the input types.
The transformPage() method in this class does two things:
Transform the input page into an "update" page format
Removes all rows whose operation number is DEFAULT_CASE_OPERATION_NUMBER
Concatenate all but the first argument with separators.
Adapts
ConnectorSplitSource to SplitSource interface.For every row from
CorrelatedJoinNode.input a CorrelatedJoinNode.subquery relation is calculated.Simple implementation of CostCalculator.
A wrapper around CostCalculator that estimates ExchangeNodes cost.
An implementation of
Anonymizer which uses simple counter-based mechanismThis rule finds correlated UnnestNode in CorrelatedJoinNode's subquery and folds
them into UnnestNode representing LEFT JOIN UNNEST.
This rule finds correlated UnnestNode in CorrelatedJoinNode's subquery and folds
them into UnnestNode representing LEFT JOIN UNNEST.
This rule decorrelates plans with correlated UnnestNode and optional EnforceSingleRowNode,
optional LimitNode, optional TopNNode and optional projections in the subquery.
The PagesHash object that handles all cases - single/multi channel joins
with any types.
This rule looks at the amount of data read and processed by the query to determine the value of partition count
used for remote partitioned exchanges.
This rule must run after the distribution type has already been set for delete queries,
since semi joins in delete queries must be replicated.
Determines whether a given Expression is deterministic
Determines whether a given Expression is deterministic
Apply saturated floor casts for implicit coercions on TupleDomain.
Only calling getDriverStats is ThreadSafe
Methods setWithDelay and reset should be used in pairs;
usually follow the following idiom:
When dynamic filter assignments are present on a Join node, they should be consumed by a Filter node on it's probe side
This operator acts as a simple "pass-through" pipe, while saving a summary of input pages.
Computes the effective predicate at the top of the specified PlanNode
This is a class representing empty input to a table function.
Makes equality based inferences to rewrite Expressions and generate equality sets in terms of specified symbol scopes
Converts an Intersect node with at least one empty branch to an empty Values node.
Replaces 0% sample node with empty values node.
Transforms:
Applies to broadcast spatial joins, inner and left, expressed via ST_Contains,
ST_Intersects and ST_Distance functions.
Implementation nodes
Only retains rows that have identical values for each respective fieldSet.
AggregationStatsRule does not have sufficient information to provide accurate estimates on aggregated symbols.
Extract expressions that are free (unbound) references to a lambda argument.
Implementations provide various top N page processing strategies for multiple groups.
Memory Layout:
This class finds the top N rows by rank value defined by and for each
group specified by .
Memory Layout:
This class finds the top N rows defined by for each group specified by .
Like
HashBuilderOperator but simplified,
without spill support.If heap nodes are numbered top down, and left to right as follows:
For each request, the addPage method will be called zero or more times,
followed by either requestComplete or clientFinished (if buffer complete).
Object registration system that allows looking up objects via stable IDs.
Transforms:
Implement EXCEPT ALL using union, window and filter.
Converts EXCEPT DISTINCT queries into UNION ALL..GROUP BY...WHERE
E.g.:
Implements filtered aggregations by transforming plans of the following shape:
Implement INTERSECT ALL using union, window and filter.
Converts INTERSECT DISTINCT queries into UNION ALL..GROUP BY...WHERE
E.g.:
Transforms:
Transforms:
This rule prepares cartesian product of partitions
from all inputs of table function.
Hash array size supplier that uses larger load factors for larger arrays
A priority queue with constant time contains(E) and log time remove(E)
Ties are broken by insertion order
Identify the mapping from the lookup symbols used at the top of the index plan to
the actual symbols produced by the IndexSource.
Replace filter predicate conjuncts with underlying project expressions.
Inlines expressions from a child project node into a parent project node
as long as they are simple constants, or they are referenced only once (to
avoid introducing duplicate computation) and the references don't appear
within a TRY block (to avoid changing semantics).
A node is a server in a cluster than can process queries.
Resource groups form a tree, and all access to a group is guarded by the root of the tree.
This scalar function exists primarily to test lambda expression support.
Remove empty pattern from pattern alternation and replace it with quantification of a neighbouring term.
Optimize row pattern:
- remove nested exclusions
- flatten alternations and concatenations
- remove redundant empty pattern
This class is to facilitate obtaining the type of an expression and its subexpressions
during planning (i.e., when interacting with IR expression).
A bridge that connects build, probe, and outer operators of a join.
JoinGraph represents sequence of Joins, where nodes in the graph
are PlanNodes that are being joined and edges are all equality join
conditions between pair of nodes.
This class eagerly calculates all join positions and stores them in an array
PageJoiner is responsible for ensuring that only the first position is processed for RLE with no or single build row match
This class must be public as it is accessed via join compiler reflection.
Extracts values from JSON
Read string input as JSON.
Format JSON as binary or character string, using given encoding.
Evaluates the JSON path expression using given JSON input and parameters,
respecting the path mode `strict` or `lax`.
A class representing state, used by JSON-processing functions: JSON_EXISTS, JSON_VALUE, and JSON_QUERY.
Implements feature ISO/IEC 9075-2:2023(E) 7.11 'JSON table'
including features T824, T827, T838
This class comprises all information necessary to execute the json_table function:
This representation does not contain all properties of the column as specified in json_table invocation.
The stack representation for JSON objects must have the keys in natural sorted order.
LikePattern can be a part of the cache key in projection/filter compiled class caches in ExpressionCompiler.
Represents inherent cost of some plan node, not including cost of its sources.
A type-specific array-based FIFO queue, supporting also deque operations.
Like
LookupJoinOperator but simplified,
without spill support.This page builder creates pages with dictionary blocks:
normal dictionary blocks for the probe side and the original blocks for the build side.
This page builder creates pages with dictionary blocks:
normal dictionary blocks for the probe side and the original blocks for the build side.
This class computes an aggregate function result in row pattern recognition context.
Materialize all values in a row
Special handling is added for Double types for approximate comparisons
Stores a plan in a form that's efficient to mutate locally (i.e.
Manages memory pools on this worker node
This class performs merge of previously hash sorted pages streams.
Transforms:
Transforms:
This rule handles both LimitNode with ties and LimitNode without ties.
This rule handles both LimitNode with ties and LimitNode without ties.
This class is intended to be used right after the PageProcessor to ensure
that the size of the pages returned by FilterAndProject and ScanFilterAndProject
is big enough so it does not introduce considerable synchronization overhead.
The node processes the result of the Searched CASE and RIGHT JOIN
derived from a MERGE statement.
This operator is used by operations like SQL MERGE.
Transforms:
Transforms:
Converts cardinality-insensitive aggregations (max, min, "distinct") over partition keys
into simple metadata queries
Implements distinct aggregations with different inputs by transforming plans of the following shape:
Represents a location in the network topology.
Implementations of this interface must be thread safe.
Used on workers.
Used on workers.
Used on workers.
NULLIF(V1,V2): CASE WHEN V1=V2 THEN NULL ELSE V1 END
Contains information about
Operator execution.For empty aggregations duplicate input rows can be skipped.
Provides a bridge between classic
Operator push input model
and WorkProcessorOperator pull model.Rewrite input references from columns in the input page (to the filter/project node)
into a compact list that can be used for method parameters.
PagesIndex a low-level data structure which contains the address of every value position of every channel.
This factory is used to pass spatial index built by SpatialIndexBuilderOperator
to SpatialJoinOperators.
Equals and hash evaluated at the specified page positions.
This class represents set of three collections representing implementations of operators
similar to partial template specialization from C++ allowing more optimized implementations to be provided for specific types.
Controls whenever partial aggregation is enabled across all
HashAggregationOperators
for a particular plan node on a single node.Like
PartitionedLookupSource but simplified,
without spill support.Schedules stages choosing to order to provide the best resource utilization.
This class is designed to facilitate the pipelined mode of execution.
Clones plan and assigns new PlanNodeIds to the copied PlanNodes.
Splits a logical plan into fragments that can be shipped and executed on distributed nodes
A carrier of core, global, non-derived services for planner and analyzer.
It is going to be executed to verify logical planner correctness
This class is responsible for iterating over build rows, which have
same values in hash columns as given probe row (but according to
filterFunction can have non matching values on some other column).
Rule that transforms selected aggregations from:
Replaces the prefix given by
packagePrefix with the domain base provided via configuration.Task (and split) priority is composed of a level and a within-level
priority.
This class represents a WindowIndex with additional "channels"
which are not part of the underlying PagesIndex.
This rule restricts the outputs of ApplyNode's input and subquery
based on which ApplyNode's output symbols are referenced.
This rule updates ApplyNode's correlation list.
This rule restricts outputs of ApplyNode's subquery to include only the symbols
needed for subqueryAssignments.
This rule restricts the outputs of CorrelatedJoinNode's input and subquery
based on which CorrelatedJoinNode's output symbols are referenced.
This rule updates CorrelatedJoinNode's correlation list.
A count over a subquery can be reduced to a VALUES(1) provided
the subquery is a scalar
This rule restricts the outputs of ExchangeNode based on which
ExchangeNode's output symbols are either referenced by the
parent node or used for partitioning, ordering or as a hash
symbol by the ExchangeNode.
This rule prunes GroupIdNode's aggregationArguments.
Joins support output symbol selection, so make any project-off of child columns explicit in project nodes.
Joins support output symbol selection, so absorb any project-off into the node.
This rule restricts the inputs to PatternRecognitionNode
based on which symbols are used by the inner structures
of the PatternRecognitionNode.
Spatial joins support output symbol selection, so absorb any project-off into the node.
TableFunctionProcessorNode has two kinds of outputs:
- proper outputs, which are the columns produced by the table function,
- pass-through outputs, which are the columns copied from table arguments.
This rule prunes unreferenced outputs of TableFunctionProcessorNode.
This is a special case of PushProjectionIntoTableScan that performs only column pruning.
Transforms
Absorb pruning projection into the node.
This optimizer pushes aggregations below outer joins when: the aggregation
is on top of the outer join, it groups by all columns in the outer table, and
the outer rows are guaranteed to be distinct.
Transforms expressions of the form
Transforms:
Transforms:
Transforms:
Transforms:
Transforms:
Transforms:
Transforms:
Transforms:
Transforms:
Transforms:
Transforms:
Transforms:
Transforms:
Aggregate functions in pattern recognition context have special semantics.
Push down aggregation's mask based on filter predicate.
Rule that transform plan like
Transforms:
Transforms:
Transforms:
This version support only constant updates
and fall back to default behaviour in all other cases
Transforms:
These rules should not be run after AddExchanges so as not to overwrite the TableLayout
chosen by AddExchanges
This rule pushes filter predicate concerning row number symbol into RowNumberNode
by modifying maxRowCountPerPartition.
This rule pushes filter predicate concerning a ranking symbol into WindowNode
by converting it into TopNRankingNode.
Transforms:
Utility class for pushing projections through inner join so that joins are not separated
by a project node and can participate in cross join elimination or join reordering.
Pushes RemoteExchange node down through the AssignUniqueId to preserve
partitioned_on(unique) and grouped(unique) properties for the output of
the AssignUniqueId.
Transforms:
Transforms:
The info will always contain column names and types.
Manage queries scheduled on this node
TODO: this needs to be merged with RowType at some point (when the type system is unified)
Remove the aggregation node that produces distinct rows from the Filtering source of a Semi join.
Flattens and removes duplicate conjuncts or disjuncts.
Removes branches from an ExceptNode that are guaranteed to produce 0 rows.
If the predicate for a mergeWriter is optimized to false, the target table scan
of the mergeWriter will be replaced with an empty values node.
If the predicate for a table execute is optimized to false, the target table scan
of table execute will be replaced with an empty values node.
Removes branches from a UnionNode that are guaranteed to produce 0 rows.
Removes 100% sample nodes.
Replace DistinctLimit node
1.
Given:
Removes projection nodes that only perform non-renaming identity projections
Remove Limit node when the subplan is guaranteed to produce fewer rows than the limit and
replace the plan with empty values if the limit count is 0.
Remove Offset node and its subplan when the subplan is guaranteed to produce no more rows than the offset
and replace the plan with empty values.
This rule removes SortNode being a source of LimitNode with ties.
Table function can take multiple table arguments.
Replace TopN node
1.
Dynamic filters are supported only right after TableScan and only if the subtree is on
1.
This rule transforms plans with join where one of the sources is
a single-row ValuesNode, and the join condition is `on true`.
This rule is complementary to RemoveRedundantJoin.
This rule transforms plans with joins, where:
- one of the sources is scalar and produces no output symbols.
Classes implementing this interface must be thread safe.
Defines the authentication and authorization required to access a REST resource.
Re-writes spatial_partitioning(geometry) aggregations into spatial_partitioning(envelope, partition_count)
on top of ST_Envelope(geometry) projection, e.g.
Strategy that combines both comparison and hash/equality over row IDs
Comparator strategy that evaluates over row IDs
Hash strategy that evaluates over row IDs
Reference to a row.
Page buffering manager that enables access to individual rows via stable row IDs.
Cursor that allows callers to advance through the registered page and dictate whether a specific position should
be preserved with a stable row ID.
A class representing a split that is running on the TaskRunner.
Provides runtime information from FTE execution.
Scale up local writers based on throughput and data processed by writers.
A wrapper for Expressions that can be used as a key in maps and sets.
For the MinHash algorithm, see "On the resemblance and containment of documents" by Andrei Z.
This class returns a set of positions to aggregate over for an aggregate function in row pattern matching context.
Determines whether, and how, a callsite matches a generic function signature.
It is not yet proven whether this heuristic is any better or worse.
Simplify conditional expressions in filter predicate.
Implements distinct aggregations with similar inputs by transforming plans of the following shape:
Helps in distributing big or skewed partitions across available tasks to improve the performance of
partitioned writes.
HashAggregationBuilder that does not aggregate input rows at all.Maintains position links in sorted order by build side expression.
Extracts sort expression to be used for creating
SortedPositionLinks from join filter expression.This class is merely a container used by coordinator to track tasks for a single stage.
Simple state machine which holds a single state.
Makes stats consistent
Current implementation is based on code points from Unicode and does ignore grapheme cluster boundaries.
Methods for creating and decoding synthetic addresses.
TableExecuteHandle wraps connectors ConnectorTableExecuteHandle which identifies instance of executing
specific table procedure o specific table.
Holder of
ScheduledExecutorService used for task management.Manages tasks on this worker node
The purpose of this class is to determine whether two pattern matching threads
are equivalent.
Returns the top N rows from the source sorted according to the specified ordering in the keyChannelIndex channel.
Optimized hash table for streaming Top N peer group lookup operations.
Returns the top N rows from the source sorted according to the specified ordering in the keyChannelIndex channel.
This rule decorrelates a correlated subquery of LEFT correlated join with distinct operator (grouped aggregation with no aggregation assignments)
It is similar to TransformCorrelatedDistinctAggregationWithProjection rule, but does not support projection over aggregation in the subquery
This rule decorrelates a correlated subquery of LEFT correlated join with distinct operator (grouped aggregation with no aggregation assignments)
This rule decorrelates a correlated subquery with:
- single global aggregation, or
- global aggregation over distinct operator (grouped aggregation with no aggregation assignments),
in case when the distinct operator cannot be de-correlated by PlanNodeDecorrelator
It is similar to TransformCorrelatedGlobalAggregationWithProjection rule, but does not support projection over aggregation in the subquery
This rule decorrelates a correlated subquery of LEFT or INNER correlated join with:
- single global aggregation, or
- global aggregation over distinct operator (grouped aggregation with no aggregation assignments),
in case when the distinct operator cannot be de-correlated by PlanNodeDecorrelator
This rule decorrelates a correlated subquery of INNER correlated join with:
- single grouped aggregation, or
- grouped aggregation over distinct operator (grouped aggregation with no aggregation assignments),
in case when the distinct operator cannot be de-correlated by PlanNodeDecorrelator
It is similar to TransformCorrelatedGroupedAggregationWithProjection rule, but does not support projection over aggregation in the subquery
This rule decorrelates a correlated subquery of INNER correlated join with:
- single grouped aggregation, or
- grouped aggregation over distinct operator (grouped aggregation with no aggregation assignments),
in case when the distinct operator cannot be de-correlated by PlanNodeDecorrelator
Replaces correlated ApplyNode with InPredicate expression with SemiJoin
Tries to decorrelate subquery and rewrite it using normal join.
Scalar filter scan query is something like:
This optimizer can rewrite correlated single row subquery to projection in a way described here:
From:
EXISTS is modeled as (if correlated predicates are equality comparisons):
Rewrite filtering semi-join to inner join.
This optimizers looks for InPredicate expressions in ApplyNodes and replaces the nodes with SemiJoin nodes.
Keeps mappings of fields and AST expressions to symbols in the current plan within query boundary.
Filters out rows that do not match the values from the specified tuple
This class is to facilitate obtaining the type of an expression and its subexpressions
during planning (i.e., when interacting with IR expression).
Ensures that all the expressions and FunctionCalls matches their output symbols
Re-maps symbol references that are just aliases of each other (e.g., due to projections like
$0 := $1)This is a layer of abstraction between
UnnestOperator and UnnestBlockBuilder to enable
translation of indices from input nested blocks to underlying element blocks.Dispatches the
UnnestingPositionsAppender.append(it.unimi.dsi.fastutil.ints.IntArrayList, io.trino.spi.block.Block) and UnnestingPositionsAppender.appendRle(io.trino.spi.block.ValueBlock, int) methods to the UnnestingPositionsAppender.delegate depending on the input Block class.Given s of type S, a constant expression t of type T, and when an implicit
cast exists between S->T, converts expression of the form:
Given constant temporal unit U and a constant date/time expression t that's rounded to unit,
converts expression of the form
Transforms expressions of the form
ROW(x, y)[1] to x
and
CAST(ROW(x, y) AS row(f1 type1, f2 type2))[1] to CAST(x AS type1)Given x::row(t) and y::row(t), converts assignments of the form
Transforms a constant date/time expression t
Annotates a program element that is (only) used by generated code.
Rule that transforms
When an aggregation has an empty grouping set then a default value needs to be returned in the output (e.g: 0 for COUNT(*)).
Ensures that all dependencies (i.e., symbols in expressions) for a plan node are provided by its source nodes
When a SCALED_WRITER_DISTRIBUTION is chosen as partitioning method then target writer should support for it.
Verifies that input of streaming aggregations is grouped on the grouping keys
Aggregation function that approximates the frequency of the top-K elements.
The interface is used when a function call needs to yield/pause during the process.
This
WorkProcessorOperator adapter allows to adapt WorkProcessor operators
that require customization of input handling (e.g.