Package org.apache.druid.query.groupby

Class GroupByQueryQueryToolChest

java.lang.Object

org.apache.druid.query.QueryToolChest<ResultRow,GroupByQuery>

org.apache.druid.query.groupby.GroupByQueryQueryToolChest

public class GroupByQueryQueryToolChest
extends QueryToolChest<ResultRow,GroupByQuery>

Toolchest for GroupBy queries

Constructor Summary

Constructors

Constructor	Description
GroupByQueryQueryToolChest(GroupingEngine groupingEngine, com.google.common.base.Supplier<GroupByQueryConfig> queryConfigSupplier, GroupByQueryMetricsFactory queryMetricsFactory, GroupByResourcesReservationPool groupByResourcesReservationPool)
GroupByQueryQueryToolChest(GroupingEngine groupingEngine, GroupByResourcesReservationPool groupByResourcesReservationPool)

Method Summary

Modifier and Type	Method	Description
boolean	canPerformSubquery(Query<?> subquery)	Returns whether this toolchest is able to handle the provided subquery.
BinaryOperator<ResultRow>	createMergeFn(Query<ResultRow> query)	Creates a merge function that is used to merge intermediate aggregates from historicals in broker.
Comparator<ResultRow>	createResultComparator(Query<ResultRow> query)	Creates an ordering comparator that is used to order results.
com.fasterxml.jackson.databind.ObjectMapper	decorateObjectMapper(com.fasterxml.jackson.databind.ObjectMapper objectMapper, GroupByQuery query)	Perform any per-query decoration of an ObjectMapper that enables it to read and write objects of the query's QueryToolChest.
CacheStrategy<ResultRow,Object,GroupByQuery>	getCacheStrategy(GroupByQuery query)	Returns a CacheStrategy to be used to load data into the cache and remove it from the cache.
com.fasterxml.jackson.core.type.TypeReference<ResultRow>	getResultTypeReference()	Returns a TypeReference object that is just passed through to Jackson in order to deserialize the results of this type of query.
static boolean	isNestedQueryPushDown(GroupByQuery q)
GroupByQueryMetrics	makeMetrics(GroupByQuery query)	Creates a QueryMetrics object that is used to generate metrics for this specific query type.
com.google.common.base.Function<ResultRow,ResultRow>	makePostComputeManipulatorFn(GroupByQuery query, MetricManipulationFn fn)	This manipulator functions primary purpose is to conduct finalization of aggregator values.
com.google.common.base.Function<ResultRow,ResultRow>	makePreComputeManipulatorFn(GroupByQuery query, MetricManipulationFn fn)	Creates a Function that can take in a ResultType and return a new ResultType having applied the MetricManipulatorFn to each of the metrics.
QueryRunner<ResultRow>	mergeResults(QueryRunner<ResultRow> runner)	This method wraps a QueryRunner.
QueryRunner<ResultRow>	mergeResults(QueryRunner<ResultRow> runner, boolean willMergeRunner)	Like QueryToolChest.mergeResults(QueryRunner), but with an additional flag that indicates the type of runner that is passeed to the call.
QueryRunner<ResultRow>	preMergeQueryDecoration(QueryRunner<ResultRow> runner)	Wraps a QueryRunner.
RowSignature	resultArraySignature(GroupByQuery query)	Returns a RowSignature for the arrays returned by QueryToolChest.resultsAsArrays(QueryType, org.apache.druid.java.util.common.guava.Sequence<ResultType>).
Sequence<Object[]>	resultsAsArrays(GroupByQuery query, Sequence<ResultRow> resultSequence)	Converts a sequence of this query's ResultType into arrays.
Optional<Sequence<FrameSignaturePair>>	resultsAsFrames(GroupByQuery query, Sequence<ResultRow> resultSequence, MemoryAllocatorFactory memoryAllocatorFactory, boolean useNestedForUnknownTypes)	This returns a single frame containing the results of the group by query.

Methods inherited from class org.apache.druid.query.QueryToolChest

filterSegments, getBaseResultType, getBySegmentResultType, postMergeQueryDecoration

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

GroupByQueryQueryToolChest

public GroupByQueryQueryToolChest(GroupingEngine groupingEngine,
                                  GroupByResourcesReservationPool groupByResourcesReservationPool)

GroupByQueryQueryToolChest

@Inject
public GroupByQueryQueryToolChest(GroupingEngine groupingEngine,
                                  com.google.common.base.Supplier<GroupByQueryConfig> queryConfigSupplier,
                                  GroupByQueryMetricsFactory queryMetricsFactory,
                                  GroupByResourcesReservationPool groupByResourcesReservationPool)

Method Detail

mergeResults

public QueryRunner<ResultRow> mergeResults(QueryRunner<ResultRow> runner)

Description copied from class: QueryToolChest

This method wraps a QueryRunner. The input QueryRunner, by contract, will provide a series of ResultType objects in time order (ascending or descending). This method should return a new QueryRunner that merges the stream of ordered ResultType objects.

A default implementation constructs a ResultMergeQueryRunner which creates a CombiningSequence using the supplied QueryRunner with QueryToolChest.createResultComparator(Query) and QueryToolChest.createMergeFn(Query)} supplied by this toolchest.

Generally speaking, the logic that exists in makePostComputeManipulatorFn should actually exist in this method. Additionally, if a query supports PostAggregations, this method should take steps to ensure that it computes PostAggregations a minimum number of times. This is most commonly achieved by computing the PostAgg results during merge and also rewriting the query such that it has the minimum number of PostAggs (most often zero).

Overrides:

mergeResults in class QueryToolChest<ResultRow,GroupByQuery>

Parameters:

runner - A QueryRunner that provides a series of ResultType objects in time order (ascending or descending)

Returns:

a QueryRunner that merges the stream of ordered ResultType objects

mergeResults

public QueryRunner<ResultRow> mergeResults(QueryRunner<ResultRow> runner,
                                           boolean willMergeRunner)

Description copied from class: QueryToolChest

Like QueryToolChest.mergeResults(QueryRunner), but with an additional flag that indicates the type of runner that is passeed to the call. willMergeRunner specifies that the input runner to the mergeResults would be the one created by the corresponding QueryRunnerFactory.mergeRunners(java.util.concurrent.ExecutorService, java.lang.Iterable<org.apache.druid.query.QueryRunner<T>>). While it depends on the input runner, it is usually true since most of the time the same server is generating a runner that it wants to merge. The notable deviation from this norm is when the broker is accumulating the results from the data servers and needs to merge them together. In this case willMergeRunner is false. Currently, the sole consumer of this parameter is GroupByQueryQueryToolChest, where it is used to determine if the mergeResults is called with GroupByMergingQueryRunner to estimate the number of merge buffers required for the query to succeed. It is set false on the brokers, because they (mostly) fetch the results from the historicals, while the data servers set it to false (because they call this method with QueryRunnerFactory.mergeRunners(java.util.concurrent.ExecutorService, java.lang.Iterable<org.apache.druid.query.QueryRunner<T>>). By default, the willMergeRunners is ignored, and the QueryToolChest.mergeResults(QueryRunner) is called. For the toolchests that override this method must ensure that QueryToolChest.mergeResults(QueryRunner) delegates to it (else it will use the default implementation for QueryToolChest.mergeResults(QueryRunner)) which would be undesirable.

Overrides:

mergeResults in class QueryToolChest<ResultRow,GroupByQuery>

createMergeFn

public BinaryOperator<ResultRow> createMergeFn(Query<ResultRow> query)

Description copied from class: QueryToolChest

Creates a merge function that is used to merge intermediate aggregates from historicals in broker. This merge function is used in the default ResultMergeQueryRunner provided by QueryToolChest.mergeResults(QueryRunner) and also used in ParallelMergeCombiningSequence by 'CachingClusteredClient' if it does not return null.

Returning null from this function means that a query does not support result merging, at least via the mechanisms that utilize this function.

Overrides:

createMergeFn in class QueryToolChest<ResultRow,GroupByQuery>

createResultComparator

public Comparator<ResultRow> createResultComparator(Query<ResultRow> query)

Description copied from class: QueryToolChest

Creates an ordering comparator that is used to order results. This comparator is used in the default ResultMergeQueryRunner provided by QueryToolChest.mergeResults(QueryRunner)

Overrides:

createResultComparator in class QueryToolChest<ResultRow,GroupByQuery>

isNestedQueryPushDown

public static boolean isNestedQueryPushDown(GroupByQuery q)

makeMetrics

public GroupByQueryMetrics makeMetrics(GroupByQuery query)

Description copied from class: QueryToolChest

Creates a QueryMetrics object that is used to generate metrics for this specific query type. This exists to allow for query-specific dimensions and metrics. That is, the ToolChest is expected to set some meaningful dimensions for metrics given this query type. Examples might be the topN threshold for a TopN query or the number of dimensions included for a groupBy query.

QueryToolChests for query types in core (druid-processing) and public extensions (belonging to the Druid source tree) should use delegate this method to GenericQueryMetricsFactory.makeMetrics(Query) on an injected instance of GenericQueryMetricsFactory, as long as they don't need to emit custom dimensions and/or metrics.

If some custom dimensions and/or metrics should be emitted for a query type, a plan described in "Making subinterfaces of QueryMetrics" section in QueryMetrics's class-level Javadocs should be followed.

One way or another, this method should ensure that QueryMetrics.query(Query) is called with the given query passed on the created QueryMetrics object before returning.

Specified by:

makeMetrics in class QueryToolChest<ResultRow,GroupByQuery>

Parameters:

query - The query that is being processed

Returns:

A QueryMetrics that can be used to make metrics for the provided query

makePreComputeManipulatorFn

public com.google.common.base.Function<ResultRow,ResultRow> makePreComputeManipulatorFn(GroupByQuery query,
                                                                                              MetricManipulationFn fn)

Description copied from class: QueryToolChest

Creates a Function that can take in a ResultType and return a new ResultType having applied the MetricManipulatorFn to each of the metrics.

This function's primary purpose is to help work around some challenges that exist around deserializing results across the wire. Specifically, different aggregators will generate different object types in a result set, if we wanted jackson to be able to deserialize these directly, we'd need to generate a response class for each query that jackson could use to deserialize things. That is not what we do. Instead, we have jackson deserialize Object instances and then use a MetricManipulatorFn to convert from those object instances to the actual object that the aggregator expects. As such, this would be more effectively named "makeObjectDeserializingFn".

It is safe and acceptable for implementations of this method to first validate that the MetricManipulationFn is MetricManipulatorFns.DESERIALIZING_INSTANCE and throw an exception if it is not. If such an exception is ever thrown, it is indicative of a bug in the caller which should be fixed by not calling this method with anything other than the deserializing manipulator function.

There are some implementations where this was also tasked with computing PostAggregators, but this is actually not a good place to compute those as this function can be called in a number of cases when PostAggs are not really meaningful to compute. Instead, PostAggs should be computed in the mergeResults call and the mergeResults implementation should take care to ensure that PostAggs are only computed the minimum number of times necessary.

This function is called very early in the processing pipeline on the Broker.

Specified by:

makePreComputeManipulatorFn in class QueryToolChest<ResultRow,GroupByQuery>

Parameters:

query - The Query that is currently being processed

fn - The function that should be applied to all metrics in the results

Returns:

A function that will apply the provided fn to all metrics in the input ResultType object

makePostComputeManipulatorFn

public com.google.common.base.Function<ResultRow,ResultRow> makePostComputeManipulatorFn(GroupByQuery query,
                                                                                               MetricManipulationFn fn)

Description copied from class: QueryToolChest

This manipulator functions primary purpose is to conduct finalization of aggregator values. It would be better named "makeFinalizingManipulatorFn", even that should really be done as part of QueryToolChest.mergeResults(org.apache.druid.query.QueryRunner<ResultType>) instead of with this separate method.

It is safe and acceptable for implementations of this method to first validate that the MetricManipulationFn is either MetricManipulatorFns.FINALIZING_INSTANCE or MetricManipulatorFns.IDENTITY_INSTANCE and throw an exception if it is not. If such an exception is ever thrown, it is indicative of a bug in the caller which should be fixed by not calling this method with unsupported manipulator functions.

Overrides:

makePostComputeManipulatorFn in class QueryToolChest<ResultRow,GroupByQuery>

Parameters:

query - The Query that is currently being processed

fn - The function that should be applied to all metrics in the results

Returns:

A function that will apply the provided fn to all metrics in the input ResultType object

getResultTypeReference

public com.fasterxml.jackson.core.type.TypeReference<ResultRow> getResultTypeReference()

Description copied from class: QueryToolChest

Returns a TypeReference object that is just passed through to Jackson in order to deserialize the results of this type of query.

Specified by:

getResultTypeReference in class QueryToolChest<ResultRow,GroupByQuery>

Returns:

A TypeReference to indicate to Jackson what type of data will exist for this query

decorateObjectMapper

public com.fasterxml.jackson.databind.ObjectMapper decorateObjectMapper(com.fasterxml.jackson.databind.ObjectMapper objectMapper,
                                                                        GroupByQuery query)

Description copied from class: QueryToolChest

Perform any per-query decoration of an ObjectMapper that enables it to read and write objects of the query's QueryToolChest. It is used by QueryResource on the write side, and DirectDruidClient on the read side.

For most queries, this is a no-op, but it can be useful for query types that support more than one result serialization format. Queries that implement this method must not modify the provided ObjectMapper, but instead must return a copy.

Overrides:

decorateObjectMapper in class QueryToolChest<ResultRow,GroupByQuery>

preMergeQueryDecoration

public QueryRunner<ResultRow> preMergeQueryDecoration(QueryRunner<ResultRow> runner)

Description copied from class: QueryToolChest

Wraps a QueryRunner. The input QueryRunner is the QueryRunner as it exists *before* being passed to mergeResults().

In fact, the return value of this method is always passed to mergeResults, so it is equivalent to just implement this functionality as extra decoration on the QueryRunner during mergeResults().

In the interests of potentially simplifying these interfaces, the recommendation is to actually not override this method and instead apply anything that might be needed here in the mergeResults() call.

Overrides:

preMergeQueryDecoration in class QueryToolChest<ResultRow,GroupByQuery>

Parameters:

runner - The runner to be wrapped

Returns:

The wrapped runner

getCacheStrategy

public CacheStrategy<ResultRow,Object,GroupByQuery> getCacheStrategy(GroupByQuery query)

Description copied from class: QueryToolChest

Returns a CacheStrategy to be used to load data into the cache and remove it from the cache.

This is optional. If it returns null, caching is effectively disabled for the query.

Overrides:

getCacheStrategy in class QueryToolChest<ResultRow,GroupByQuery>

Parameters:

query - The query whose results might be cached

Returns:

A CacheStrategy that can be used to populate and read from the Cache

canPerformSubquery

public boolean canPerformSubquery(Query<?> subquery)

Description copied from class: QueryToolChest

Returns whether this toolchest is able to handle the provided subquery.

When this method returns true, the core query stack will pass subquery datasources over to the toolchest and will assume they are properly handled.

When this method returns false, the core query stack will throw an error if subqueries are present. In the future, instead of throwing an error, the core query stack will handle the subqueries on its own.

Overrides:

canPerformSubquery in class QueryToolChest<ResultRow,GroupByQuery>

resultArraySignature

public RowSignature resultArraySignature(GroupByQuery query)

Description copied from class: QueryToolChest

Returns a RowSignature for the arrays returned by QueryToolChest.resultsAsArrays(QueryType, org.apache.druid.java.util.common.guava.Sequence<ResultType>). The returned signature will be the same length as each array returned by QueryToolChest.resultsAsArrays(QueryType, org.apache.druid.java.util.common.guava.Sequence<ResultType>).

Overrides:

resultArraySignature in class QueryToolChest<ResultRow,GroupByQuery>

Parameters:

query - same query passed to QueryToolChest.resultsAsArrays(QueryType, org.apache.druid.java.util.common.guava.Sequence<ResultType>)

Returns:

row signature

resultsAsArrays

public Sequence<Object[]> resultsAsArrays(GroupByQuery query,
                                          Sequence<ResultRow> resultSequence)

Description copied from class: QueryToolChest

Converts a sequence of this query's ResultType into arrays. The array signature is given by QueryToolChest.resultArraySignature(QueryType). This functionality is useful because it allows higher-level processors to operate on the results of any query in a consistent way. This is useful for the SQL layer and for any algorithm that might operate on the results of an inner query.

Not all query types support this method. They will throw UnsupportedOperationException, and they cannot be used by the SQL layer or by generic higher-level algorithms.

Some query types return less information after translating their results into arrays, especially in situations where there is no clear way to translate fully rich results into flat arrays. For example, the scan query does not include the segmentId in its array-based results, because it could potentially conflict with a 'segmentId' field in the actual datasource being scanned.

It is possible that there will be multiple arrays returned for a single result object. For example, in the topN query, each TopNResultValue will generate a separate array for each of its values.

By convention, the array form should include the __time column, if present, as a long (milliseconds since epoch).

Overrides:

resultsAsArrays in class QueryToolChest<ResultRow,GroupByQuery>

resultSequence - results of the form returned by QueryToolChest.mergeResults(org.apache.druid.query.QueryRunner<ResultType>)

Returns:

results in array form

resultsAsFrames

public Optional<Sequence<FrameSignaturePair>> resultsAsFrames(GroupByQuery query,
                                                              Sequence<ResultRow> resultSequence,
                                                              MemoryAllocatorFactory memoryAllocatorFactory,
                                                              boolean useNestedForUnknownTypes)

This returns a single frame containing the results of the group by query.

Overrides:

resultsAsFrames in class QueryToolChest<ResultRow,GroupByQuery>

Parameters:

query - Query being executed by the toolchest. Used to determine the rowSignature of the Frames

resultSequence - results of the form returned by QueryToolChest.mergeResults(QueryRunner)

useNestedForUnknownTypes - true if the unknown types in the results can be serded using complex types