Packages

package shims

Ordering
  1. Alphabetic
Visibility
  1. Public
  2. All

Type Members

  1. case class AvoidAdaptiveTransitionToRow(child: SparkPlan) extends SparkPlan with ShimUnaryExecNode with GpuExec with Product with Serializable

    This operator will attempt to optimize the case when we are writing the results of an adaptive query to disk so that we remove the redundant transitions from columnar to row within AdaptiveSparkPlanExec followed by a row to columnar transition.

    This operator will attempt to optimize the case when we are writing the results of an adaptive query to disk so that we remove the redundant transitions from columnar to row within AdaptiveSparkPlanExec followed by a row to columnar transition.

    Specifically, this is the plan we see in this case:

    GpuRowToColumnar(AdaptiveSparkPlanExec(GpuColumnarToRow(child))

    We perform this optimization at runtime rather than during planning, because when the adaptive plan is being planned and executed, we don't know whether it is being called from an operation that wants rows (such as CollectTailExec) or from an operation that wants columns (such as GpuDataWritingCommandExec).

    Spark does not provide a mechanism for executing an adaptive plan and retrieving columnar results and the internal methods that we need to call are private, so we use reflection to call them.

    child

    The plan to execute

  2. class BatchScanExecMeta extends SparkPlanMeta[BatchScanExec]
  3. final class CreateDataSourceTableAsSelectCommandMeta extends DataWritingCommandMeta[CreateDataSourceTableAsSelectCommand]
  4. abstract class GetMapValueMeta extends BinaryExprMeta[GetMapValue]

    We define this type in the shim layer because GetMapValue doesn't have the field failOnError since Spark 3.4.0 and it always returns null on invalid access to map column in ANSI mode.

  5. class GpuAggregateInPandasExecMeta extends SparkPlanMeta[AggregateInPandasExec]
  6. case class GpuBatchScanExec(output: Seq[AttributeReference], scan: GpuScan) extends SparkPlan with DataSourceV2ScanExecBase with GpuBatchScanExecMetrics with Product with Serializable
  7. abstract class GpuBroadcastJoinMeta[INPUT <: SparkPlan] extends SparkPlanMeta[INPUT]
  8. trait GpuCreateHiveTableAsSelectBase extends LogicalPlan with GpuDataWritingCommand

    GPU version of Spark's CreateHiveTableAsSelectBase

  9. class GpuCustomShuffleReaderMeta extends SparkPlanMeta[CustomShuffleReaderExec]
  10. class GpuDataSourceRDD extends DataSourceRDD

    A replacement for DataSourceRDD that does NOT compute the bytes read input metric.

    A replacement for DataSourceRDD that does NOT compute the bytes read input metric. DataSourceRDD assumes all reads occur on the task thread, and some GPU input sources use multithreaded readers that cannot generate proper metrics with DataSourceRDD.

    Note

    It is the responsibility of users of this RDD to generate the bytes read input metric explicitly!

  11. trait GpuDeterministicFirstLastCollectShim extends Expression
  12. case class GpuHashPartitioning(expressions: Seq[Expression], numPartitions: Int) extends GpuHashPartitioningBase with Product with Serializable
  13. case class GpuOptimizedCreateHiveTableAsSelectCommand(tableDesc: CatalogTable, query: LogicalPlan, outputColumnNames: Seq[String], mode: SaveMode, cpuCmd: OptimizedCreateHiveTableAsSelectCommand) extends LogicalPlan with GpuCreateHiveTableAsSelectBase with Product with Serializable
  14. class GpuOrcDataReader extends DataReader

    File cache is not supported for Spark 3.1.x so this is a thin wrapper around the ORC DataReader.

  15. case class GpuRangePartitioning(gpuOrdering: Seq[SortOrder], numPartitions: Int) extends Expression with GpuExpression with ShimExpression with GpuPartitioning with Product with Serializable

    A GPU accelerated org.apache.spark.sql.catalyst.plans.physical.Partitioning that partitions sortable records by range into roughly equal ranges.

    A GPU accelerated org.apache.spark.sql.catalyst.plans.physical.Partitioning that partitions sortable records by range into roughly equal ranges. The ranges are determined by sampling the content of the RDD passed in.

    Note

    The actual number of partitions created might not be the same as the numPartitions parameter, in the case where the number of sampled records is less than the value of partitions. The GpuRangePartitioner is where all of the processing actually happens.

  16. class GpuSpecifiedWindowFrameMeta extends GpuSpecifiedWindowFrameMetaBase
  17. class GpuWindowExpressionMeta extends GpuWindowExpressionMetaBase
  18. case class GpuWindowInPandasExec(windowExpression: Seq[Expression], gpuPartitionSpec: Seq[Expression], cpuOrderSpec: Seq[SortOrder], child: SparkPlan)(cpuPartitionSpec: Seq[Expression]) extends SparkPlan with GpuWindowInPandasExecBase with Product with Serializable
  19. abstract class OffsetWindowFunctionMeta[INPUT <: OffsetWindowFunction] extends ExprMeta[INPUT]

    Spark 3.1.1-specific replacement for com.nvidia.spark.rapids.OffsetWindowFunctionMeta.

    Spark 3.1.1-specific replacement for com.nvidia.spark.rapids.OffsetWindowFunctionMeta. This is required primarily for two reasons:

    1. com.nvidia.spark.rapids.OffsetWindowFunctionMeta (compiled against Spark 3.0.x) fails class load in Spark 3.1.x. (expr.input is not recognized as an Expression.) 2. The semantics of offsets in LAG() are reversed/negated in Spark 3.1.1. E.g. The expression LAG(col, 5) causes Lag.offset to be set to -5, as opposed to 5, in prior versions of Spark. This class adjusts the LAG offset to use similar semantics to Spark 3.0.x.
  20. final class OptimizedCreateHiveTableAsSelectCommandMeta extends DataWritingCommandMeta[OptimizedCreateHiveTableAsSelectCommand]
  21. class OrcProtoWriterShim extends AnyRef
  22. trait OrcShims311until320Base extends AnyRef
  23. class PlanShimsImpl extends PlanShims
  24. class RapidsOrcScanMeta extends ScanMeta[OrcScan]
  25. class RapidsParquetScanMeta extends ScanMeta[ParquetScan]
  26. trait ShimBinaryExecNode extends SparkPlan with BinaryExecNode
  27. trait ShimBinaryExpression extends BinaryExpression
  28. trait ShimBroadcastExchangeLike extends Exchange with BroadcastExchangeLike

    This shim handles the completion future differences between Apache Spark and Databricks.

  29. trait ShimExpression extends Expression
  30. abstract class ShimFilePartitionReaderFactory extends FilePartitionReaderFactory
  31. trait ShimGetArrayItem extends Expression with ExtractValue
  32. trait ShimGetArrayStructFields extends Expression with ExtractValue
  33. trait ShimGetStructField extends Expression with ExtractValue
  34. trait ShimLeafExecNode extends SparkPlan with LeafExecNode
  35. trait ShimPredicateHelper extends PredicateHelper
  36. trait ShimSparkPlan extends SparkPlan
  37. trait ShimSupportsRuntimeFiltering extends AnyRef

    Shim interface for Apache Spark's SupportsRuntimeFiltering interface which was added in Spark 3.2.0.

  38. trait ShimTernaryExpression extends TernaryExpression
  39. trait ShimUnaryCommand extends LogicalPlan with Command
  40. trait ShimUnaryExecNode extends SparkPlan with UnaryExecNode
  41. trait ShimUnaryExpression extends UnaryExpression
  42. abstract class Spark31XShims extends Spark31Xuntil33XShims with Logging
  43. trait Spark31Xuntil33XShims extends SparkShims

Value Members

  1. object AQEUtils

    Utility methods for manipulating Catalyst classes involved in Adaptive Query Execution

  2. object AggregationTagging
  3. object AnsiCastShim
  4. object AnsiUtil
  5. object BloomFilterShims
  6. object CastCheckShims
  7. object CastingConfigShim
  8. object CharVarcharUtilsShims
  9. object DecimalArithmeticOverrides
  10. object DeltaLakeUtils
  11. object DistributionUtil
  12. object FileIndexOptionsShims
  13. object GlobalLimitShims
  14. object GpuCastShims
  15. object GpuDataSourceRDD extends Serializable
  16. object GpuFileFormatDataWriterShim
  17. object GpuHashPartitioning extends Serializable
  18. object GpuIntervalUtils

    Should not support in this Shim

  19. object GpuOrcDataReader
  20. object GpuParquetCrypto
  21. object GpuTypeShims
  22. object GpuWindowUtil
  23. object HashUtils
  24. object InSubqueryShims
  25. object LegacyBehaviorPolicyShim
  26. object NullOutputStreamShim
  27. object OrcCastingShims
  28. object OrcProtoWriterShim
  29. object OrcReadingShims
  30. object OrcShims extends OrcShims311until320Base
  31. object ParquetFieldIdShims
  32. object ParquetLegacyNanoAsLongShims
  33. object ParquetSchemaClipShims
  34. object ParquetStringPredShims
  35. object ParquetTimestampNTZShims
  36. object PartitionedFileUtilsShim
  37. object PythonUDFShim
  38. object RapidsFileSourceMetaUtils
  39. object ReaderUtils
  40. object ShuffleOriginUtil
  41. object SparkShimImpl extends Spark31XShims
  42. object TypeSigUtil extends TypeSigUtilBase

    TypeSig Support for [3.1.1, 3.2.0)

  43. object TypeUtilsShims

    Reimplement the function checkForNumericExpr which has been removed since Spark 3.4.0

  44. object XxHash64Shims
  45. object YearParseUtil

Ungrouped