Type Members

1.  class BatchPartitionIdPassthrough extends Partitioner

   A dummy partitioner for use with records whose partition ids have been pre-computed (i.e.

   A dummy partitioner for use with records whose partition ids have been pre-computed (i.e. for use on RDDs of (Int, Row) pairs where the Int is a partition id in the expected range).

2.  class CoalescedBatchPartitioner extends Partitioner

   A Partitioner that might group together one or more partitions from the parent.

3.  class CrossJoinIterator extends Iterator[ColumnarBatch] with Arm

   An iterator that does a cross join against a stream of batches.

4.  abstract class GpuBroadcastExchangeExecBase extends Exchange with GpuExec

   In some versions of databricks we need to return the completionFuture in a different way.

5.  abstract class GpuBroadcastExchangeExecBaseWithFuture extends GpuBroadcastExchangeExecBase
6.  class GpuBroadcastMeta extends SparkPlanMeta[BroadcastExchangeExec]
7.  abstract class GpuBroadcastNestedLoopJoinExecBase extends SparkPlan with BinaryExecNode with GpuExec
8.  class GpuBroadcastNestedLoopJoinMeta extends GpuBroadcastJoinMeta[BroadcastNestedLoopJoinExec]
9.  case class GpuBroadcastToCpuExec(mode: BroadcastMode, child: SparkPlan) extends GpuBroadcastExchangeExecBaseWithFuture with Product with Serializable

   This is a specialized version of GpuColumnarToRow that wraps a GpuBroadcastExchange and converts the columnar results containing cuDF tables into Spark rows so that the results can feed a CPU BroadcastHashJoin.

   This is a specialized version of GpuColumnarToRow that wraps a GpuBroadcastExchange and converts the columnar results containing cuDF tables into Spark rows so that the results can feed a CPU BroadcastHashJoin. This is required for exchange reuse in AQE.

   mode

   Broadcast mode

   child

   Input to broadcast

10.  class GpuColumnToRowMapPartitionsRDD extends MapPartitionsRDD[InternalRow, ColumnarBatch]
11.  case class GpuCustomShuffleReaderExec(child: SparkPlan, partitionSpecs: Seq[ShufflePartitionSpec]) extends SparkPlan with UnaryExecNode with GpuExec with Product with Serializable

    A wrapper of shuffle query stage, which follows the given partition arrangement.

    A wrapper of shuffle query stage, which follows the given partition arrangement.

    child

    It is usually ShuffleQueryStageExec, but can be the shuffle exchange node during canonicalization.

    partitionSpecs

    The partition specs that defines the arrangement.

12.  trait GpuHashJoin extends SparkPlan with GpuExec
13.  abstract class GpuShuffleExchangeExecBase extends Exchange with GpuExec

    Performs a shuffle that will result in the desired partitioning.

14.  abstract class GpuShuffleExchangeExecBaseWithMetrics extends GpuShuffleExchangeExecBase

    Performs a shuffle that will result in the desired partitioning.

15.  class GpuShuffleMeta extends SparkPlanMeta[ShuffleExchangeExec]
16.  class HashJoinIterator extends Iterator[ColumnarBatch] with Arm with Logging

    An iterator that does a hash join against a stream of batches.

17.  class SerializeBatchDeserializeHostBuffer extends Serializable with AutoCloseable

    Annotations

    @SerialVersionUID()

18.  class SerializeConcatHostBuffersDeserializeBatch extends Serializable with Arm with AutoCloseable

    Annotations

    @SerialVersionUID()

19.  class ShuffledBatchRDD extends RDD[ColumnarBatch]

    This is a specialized version of org.apache.spark.rdd.ShuffledRDD that is optimized for shuffling ColumnarBatch instead of Java key-value pairs.

    This is a specialized version of org.apache.spark.rdd.ShuffledRDD that is optimized for shuffling ColumnarBatch instead of Java key-value pairs.

    This RDD takes a ShuffleDependency (dependency), and an optional array of partition start indices as input arguments (specifiedPartitionStartIndices).

    The dependency has the parent RDD of this RDD, which represents the dataset before shuffle (i.e. map output). Elements of this RDD are (partitionId, Row) pairs. Partition ids should be in the range [0, numPartitions - 1]. dependency.partitioner is the original partitioner used to partition map output, and dependency.partitioner.numPartitions is the number of pre-shuffle partitions (i.e. the number of partitions of the map output).

    When specifiedPartitionStartIndices is defined, specifiedPartitionStartIndices.length will be the number of post-shuffle partitions. For this case, the ith post-shuffle partition includes specifiedPartitionStartIndices[i] to specifiedPartitionStartIndices[i+1] - 1 (inclusive).

    When specifiedPartitionStartIndices is not defined, there will be dependency.partitioner.numPartitions post-shuffle partitions. For this case, a post-shuffle partition is created for every pre-shuffle partition.

20.  case class ShuffledBatchRDDPartition(index: Int, spec: ShufflePartitionSpec) extends Partition with Product with Serializable