ai.databand.spark

Class DbndSparkQueryExecutionListener

java.lang.Object

ai.databand.spark.DbndSparkQueryExecutionListener

All Implemented Interfaces:

org.apache.spark.sql.util.QueryExecutionListener

public class DbndSparkQueryExecutionListener
extends java.lang.Object
implements org.apache.spark.sql.util.QueryExecutionListener

Constructor Summary

Constructors

Constructor and Description
DbndSparkQueryExecutionListener()
DbndSparkQueryExecutionListener(DbndWrapper dbnd)

Method Summary

Modifier and Type	Method and Description
protected java.lang.String	exctractPath(java.lang.String path) Extract data path from spark query plan.
protected java.util.Optional<org.apache.spark.sql.execution.SparkPlan>	extractChildFromShuffle(org.apache.spark.sql.execution.SparkPlan shuffleQuery) Shuffle queries was introduced in Spark 3 and doesn't have "children" nodes.
protected java.util.Optional<org.apache.spark.sql.execution.SparkPlan>	extractFinalFromAdaptive(org.apache.spark.sql.execution.SparkPlan adaptivePlan) Adaptive plans was introduced in Spark 2.4.8+ and turned on by default on some configurations.
protected java.util.List<org.apache.spark.sql.execution.SparkPlan>	getAllChildren(org.apache.spark.sql.execution.SparkPlan root) DFS across spark plan nodes.
protected java.lang.String	getPlanVerboseString(org.apache.spark.sql.catalyst.plans.logical.LogicalPlan plan) This is workaround for Spark 3.
protected boolean	isAdaptivePlan(java.lang.Object plan)
protected void	log(java.lang.String path, DatasetOperationType operationType, org.apache.spark.sql.types.StructType datasetSchema, long rows)
void	onFailure(java.lang.String funcName, org.apache.spark.sql.execution.QueryExecution qe, java.lang.Exception exception)
void	onSuccess(java.lang.String funcName, org.apache.spark.sql.execution.QueryExecution qe, long durationNs)
protected boolean	submitReadOps(org.apache.spark.sql.execution.SparkPlan executedPlan)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DbndSparkQueryExecutionListener

public DbndSparkQueryExecutionListener(DbndWrapper dbnd)

DbndSparkQueryExecutionListener

public DbndSparkQueryExecutionListener()

Method Detail

onSuccess

public void onSuccess(java.lang.String funcName,
                      org.apache.spark.sql.execution.QueryExecution qe,
                      long durationNs)

Specified by:

onSuccess in interface org.apache.spark.sql.util.QueryExecutionListener

submitReadOps

protected boolean submitReadOps(org.apache.spark.sql.execution.SparkPlan executedPlan)

isAdaptivePlan

protected boolean isAdaptivePlan(java.lang.Object plan)

getPlanVerboseString

protected java.lang.String getPlanVerboseString(org.apache.spark.sql.catalyst.plans.logical.LogicalPlan plan)

This is workaround for Spark 3. verboseString() method was replaced in Spark 3 by verboseString(int).

Parameters:

plan -

Returns:

extractFinalFromAdaptive

protected java.util.Optional<org.apache.spark.sql.execution.SparkPlan> extractFinalFromAdaptive(org.apache.spark.sql.execution.SparkPlan adaptivePlan)

Adaptive plans was introduced in Spark 2.4.8+ and turned on by default on some configurations. For proper reporting we use reflection to extract relevant pieces from Adaptive Plans. Reflection allows us to avoid direct dependency bump.

Parameters:

adaptivePlan -

Returns:

log

protected void log(java.lang.String path,
                   DatasetOperationType operationType,
                   org.apache.spark.sql.types.StructType datasetSchema,
                   long rows)

exctractPath

protected java.lang.String exctractPath(java.lang.String path)

Extract data path from spark query plan.

Parameters:

path -

Returns:

getAllChildren

protected java.util.List<org.apache.spark.sql.execution.SparkPlan> getAllChildren(org.apache.spark.sql.execution.SparkPlan root)

DFS across spark plan nodes.

Parameters:

root -

Returns:

extractChildFromShuffle

protected java.util.Optional<org.apache.spark.sql.execution.SparkPlan> extractChildFromShuffle(org.apache.spark.sql.execution.SparkPlan shuffleQuery)

Shuffle queries was introduced in Spark 3 and doesn't have "children" nodes. Instead, the child node can be accessed using plan() method. We use reflection to gain access to this method in runtime and avoid direct dependency to Spark 3.

Parameters:

shuffleQuery -

Returns:

onFailure

public void onFailure(java.lang.String funcName,
                      org.apache.spark.sql.execution.QueryExecution qe,
                      java.lang.Exception exception)

Specified by:

onFailure in interface org.apache.spark.sql.util.QueryExecutionListener