Packages

package shims

Ordering
  1. Alphabetic
Visibility
  1. Public
  2. All

Type Members

  1. trait AnsiCastRuleShims extends SparkShims
  2. class BatchScanExecMeta extends SparkPlanMeta[BatchScanExec]
  3. final class CreateDataSourceTableAsSelectCommandMeta extends DataWritingCommandMeta[CreateDataSourceTableAsSelectCommand]
  4. final class CudfUnsafeRow extends CudfUnsafeRowBase
  5. abstract class CudfUnsafeRowBase extends InternalRow
  6. trait CudfUnsafeRowTrait extends AnyRef
  7. class FileSourceScanExecMeta extends SparkPlanMeta[FileSourceScanExec]
  8. trait FilteredPartitions extends AnyRef
  9. abstract class GetMapValueMeta extends BinaryExprMeta[GetMapValue]

    We define this type in the shim layer because GetMapValue doesn't have the field failOnError since Spark 3.4.0 and it always returns null on invalid access to map column in ANSI mode.

  10. class GpuAggregateInPandasExecMeta extends SparkPlanMeta[AggregateInPandasExec]
  11. case class GpuBatchScanExec(output: Seq[AttributeReference], scan: GpuScan, runtimeFilters: Seq[Expression] = Seq.empty) extends GpuBatchScanExecBase with Product with Serializable
  12. abstract class GpuBatchScanExecBase extends SparkPlan with DataSourceV2ScanExecBase with GpuBatchScanExecMetrics with FilteredPartitions
  13. abstract class GpuBroadcastJoinMeta[INPUT <: SparkPlan] extends SparkPlanMeta[INPUT]
  14. trait GpuCreateHiveTableAsSelectBase extends LogicalPlan with GpuDataWritingCommand

    GPU version of Spark's CreateHiveTableAsSelectBase

  15. class GpuCustomShuffleReaderMeta extends SparkPlanMeta[AQEShuffleReadExec]
  16. class GpuDataSourceRDD extends DataSourceRDD

    A replacement for DataSourceRDD that does NOT compute the bytes read input metric.

    A replacement for DataSourceRDD that does NOT compute the bytes read input metric. DataSourceRDD assumes all reads occur on the task thread, and some GPU input sources use multithreaded readers that cannot generate proper metrics with DataSourceRDD.

    Note

    It is the responsibility of users of this RDD to generate the bytes read input metric explicitly!

  17. trait GpuDeterministicFirstLastCollectShim extends Expression
  18. case class GpuHashPartitioning(expressions: Seq[Expression], numPartitions: Int) extends GpuHashPartitioningBase with Product with Serializable
  19. case class GpuOptimizedCreateHiveTableAsSelectCommand(tableDesc: CatalogTable, query: LogicalPlan, outputColumnNames: Seq[String], mode: SaveMode, cpuCmd: OptimizedCreateHiveTableAsSelectCommand) extends LogicalPlan with GpuCreateHiveTableAsSelectBase with Product with Serializable
  20. class GpuOrcDataReader extends GpuOrcDataReader320Plus
  21. abstract class GpuOrcDataReader320Plus extends GpuOrcDataReaderBase
  22. abstract class GpuOrcDataReaderBase extends DataReader
  23. case class GpuRangePartitioning(gpuOrdering: Seq[SortOrder], numPartitions: Int) extends Expression with GpuExpression with ShimExpression with GpuPartitioning with Product with Serializable

    A GPU accelerated org.apache.spark.sql.catalyst.plans.physical.Partitioning that partitions sortable records by range into roughly equal ranges.

    A GPU accelerated org.apache.spark.sql.catalyst.plans.physical.Partitioning that partitions sortable records by range into roughly equal ranges. The ranges are determined by sampling the content of the RDD passed in.

    Note

    The actual number of partitions created might not be the same as the numPartitions parameter, in the case where the number of sampled records is less than the value of partitions. The GpuRangePartitioner is where all of the processing actually happens.

  24. class GpuSpecifiedWindowFrameMeta extends GpuSpecifiedWindowFrameMetaBase
  25. class GpuWindowExpressionMeta extends GpuWindowExpressionMetaBase
  26. case class GpuWindowInPandasExec(windowExpression: Seq[Expression], gpuPartitionSpec: Seq[Expression], cpuOrderSpec: Seq[SortOrder], child: SparkPlan)(cpuPartitionSpec: Seq[Expression]) extends SparkPlan with GpuWindowInPandasExecBase with Product with Serializable
  27. abstract class OffsetWindowFunctionMeta[INPUT <: OffsetWindowFunction] extends ExprMeta[INPUT]

    Spark 3.2-specific replacement for com.nvidia.spark.rapids.OffsetWindowFunctionMeta.

    Spark 3.2-specific replacement for com.nvidia.spark.rapids.OffsetWindowFunctionMeta. This is required primarily for two reasons:

    1. com.nvidia.spark.rapids.OffsetWindowFunctionMeta (compiled against Spark 3.0.x) fails class load in Spark 3.1.x. (expr.input is not recognized as an Expression.) 2. The semantics of offsets in LAG() are reversed/negated in Spark 3.1.1. E.g. The expression LAG(col, 5) causes Lag.offset to be set to -5, as opposed to 5, in prior versions of Spark. 3. IGNORE NULL and RESPECT NULL was added in 3.2.0 and we need to fall back until this can be supported. This class adjusts the LAG offset to use similar semantics to Spark 3.0.x.
  28. final class OptimizedCreateHiveTableAsSelectCommandMeta extends DataWritingCommandMeta[OptimizedCreateHiveTableAsSelectCommand]
  29. class OrcProtoWriterShim extends AnyRef
  30. trait OrcShims320untilAllBase extends AnyRef
  31. class PlanShimsImpl extends PlanShims
  32. class RapidsCsvScanMeta extends ScanMeta[CSVScan]
  33. class RapidsOrcScanMeta extends ScanMeta[OrcScan]
  34. class RapidsParquetScanMeta extends ScanMeta[ParquetScan]
  35. trait RebaseShims extends AnyRef
  36. trait ShimBaseSubqueryExec extends BaseSubqueryExec
  37. trait ShimBinaryExecNode extends SparkPlan with BinaryExecNode
  38. trait ShimBinaryExpression extends BinaryExpression
  39. trait ShimBroadcastExchangeLike extends Exchange with BroadcastExchangeLike

    This shim handles the completion future differences between Apache Spark and Databricks.

  40. trait ShimExpression extends Expression
  41. abstract class ShimFilePartitionReaderFactory extends FilePartitionReaderFactory
  42. trait ShimGetArrayItem extends Expression with ExtractValue
  43. trait ShimGetArrayStructFields extends Expression with ExtractValue
  44. trait ShimGetStructField extends Expression with ExtractValue
  45. trait ShimLeafExecNode extends SparkPlan with LeafExecNode
  46. trait ShimPredicateHelper extends PredicateHelper
  47. trait ShimSparkPlan extends SparkPlan
  48. trait ShimSupportsRuntimeFiltering extends SupportsRuntimeFiltering

    Shim interface for Apache Spark's SupportsRuntimeFiltering interface which was added in Spark 3.2.0.

  49. trait ShimTernaryExpression extends TernaryExpression
  50. trait ShimUnaryCommand extends LogicalPlan with UnaryCommand
  51. trait ShimUnaryExecNode extends SparkPlan with UnaryExecNode
  52. trait ShimUnaryExpression extends UnaryExpression
  53. trait Spark31Xuntil33XShims extends SparkShims
  54. trait Spark320PlusNonDBShims extends SparkShims

    Shim methods that can be compiled with every supported 3.2.0+ except Databricks versions

  55. trait Spark320PlusShims extends SparkShims with RebaseShims with Logging

    Shim base class that can be compiled with every supported 3.2.0+

Value Members

  1. object AQEUtils

    Utility methods for manipulating Catalyst classes involved in Adaptive Query Execution

  2. object AggregationTagging
  3. object AnsiCastShim
  4. object AnsiUtil
  5. object BloomFilterShims
  6. object BucketSpecForHiveShim
  7. object BucketingUtilsShim
  8. object CastCheckShims
  9. object CastingConfigShim
  10. object CharVarcharUtilsShims
  11. object ColumnDefaultValuesShims
  12. object CudfUnsafeRow extends CudfUnsafeRowTrait with Serializable
  13. object DateTimeUtilsShims
  14. object DecimalArithmeticOverrides
  15. object DecimalMultiply128
  16. object DeltaLakeUtils
  17. object DistributionUtil
  18. object FileIndexOptionsShims
  19. object GetSequenceSize
  20. object GlobalLimitShims
  21. object GpuCastShims
  22. object GpuDataSourceRDD extends Serializable
  23. object GpuFileFormatDataWriterShim
  24. object GpuHashPartitioning extends Serializable
  25. object GpuIntervalUtils

    Should not support in this Shim

  26. object GpuOrcDataReader
  27. object GpuParquetCrypto
  28. object GpuTypeShims
  29. object GpuWindowUtil
  30. object HashUtils
  31. object InSubqueryShims
  32. object LegacyBehaviorPolicyShim
  33. object NullOutputStreamShim
  34. object OrcCastingShims
  35. object OrcProtoWriterShim
  36. object OrcReadingShims
  37. object OrcShims extends OrcShims320untilAllBase
  38. object ParquetFieldIdShims
  39. object ParquetLegacyNanoAsLongShims
  40. object ParquetSchemaClipShims
  41. object ParquetStringPredShims
  42. object ParquetTimestampNTZShims
  43. object PartitionedFileUtilsShim
  44. object PythonUDFShim
  45. object RaiseErrorShim
  46. object RapidsFileSourceMetaUtils
  47. object ScanExecShims
  48. object ShuffleOriginUtil
  49. object SparkShimImpl extends Spark320PlusShims with Spark320PlusNonDBShims with Spark31Xuntil33XShims with AnsiCastRuleShims
  50. object StaticPartitionShims
  51. object TagScanForRuntimeFiltering
  52. object TypeSigUtil extends TypeSigUtilBase

    Add DayTimeIntervalType and YearMonthIntervalType support

  53. object TypeUtilsShims

    Reimplement the function checkForNumericExpr which has been removed since Spark 3.4.0

  54. object XxHash64Shims
  55. object YearParseUtil

Ungrouped