package rapids
- Alphabetic
- Public
- All
Type Members
- trait BasePad extends TernaryExpression with GpuTernaryExpression with ImplicitCastInputTypes with NullIntolerant
-
class
BasicColumnarWriteJobStatsTracker extends ColumnarWriteJobStatsTracker
Simple ColumnarWriteJobStatsTracker implementation that's serializable, capable ofinstantiating BasicColumnarWriteTaskStatsTracker on executors and processing the
BasicColumnarWriteTaskStatsthey produce by aggregating the metrics and posting them as DriverMetricUpdates. -
case class
BasicColumnarWriteTaskStats(numPartitions: Int, numFiles: Int, numBytes: Long, numRows: Long) extends WriteTaskStats with Product with Serializable
Simple metrics collected during an instance of GpuFileFormatDataWriter.
Simple metrics collected during an instance of GpuFileFormatDataWriter. These were first introduced in https://github.com/apache/spark/pull/18159 (SPARK-20703).
-
class
BasicColumnarWriteTaskStatsTracker extends ColumnarWriteTaskStatsTracker with Logging
Simple metrics collected during an instance of GpuFileFormatDataWriter.
Simple metrics collected during an instance of GpuFileFormatDataWriter. This is the columnar version of
org.apache.spark.sql.execution.datasources.BasicWriteTaskStatsTracker. -
trait
ColumnarWriteJobStatsTracker extends Serializable
A class implementing this trait is basically a collection of parameters that are necessary for instantiating a (derived type of) ColumnarWriteTaskStatsTracker on all executors and then process the statistics produced by them (e.g.
A class implementing this trait is basically a collection of parameters that are necessary for instantiating a (derived type of) ColumnarWriteTaskStatsTracker on all executors and then process the statistics produced by them (e.g. save them to memory/disk, issue warnings, etc). It is therefore important that such an objects is
Serializable, as it will be sent from the driver to all executors. -
trait
ColumnarWriteTaskStatsTracker extends AnyRef
A trait for classes that are capable of collecting statistics on columnar data that's being processed by a single write task in GpuFileFormatDataWriter - i.e.
A trait for classes that are capable of collecting statistics on columnar data that's being processed by a single write task in GpuFileFormatDataWriter - i.e. there should be one instance per executor.
This trait is coupled with the way GpuFileFormatWriter works, in the sense that its methods will be called according to how column batches are being written out to disk, namely in sorted order according to partitionValue(s), then bucketId.
As such, a typical call scenario is:
newPartition -> newBucket -> newFile -> newRow -. ^ |^_ ^| | | || | || ||
newPartition and newBucket events are only triggered if the relation to be written out is partitioned and/or bucketed, respectively.
- abstract case class CudfAggregate(ref: Expression) extends Expression with GpuUnevaluable with Product with Serializable
- abstract class CudfBinaryArithmetic extends CudfBinaryOperator with NullIntolerant
- abstract class CudfBinaryComparison extends CudfBinaryOperator with Predicate
- abstract class CudfBinaryMathExpression extends BinaryExpression with CudfBinaryExpression with Serializable with ImplicitCastInputTypes
- class CudfCount extends CudfAggregate
- class CudfFirstExcludeNulls extends CudfFirstLastBase
- class CudfFirstIncludeNulls extends CudfFirstLastBase
- abstract class CudfFirstLastBase extends CudfAggregate
- class CudfLastExcludeNulls extends CudfFirstLastBase
- class CudfLastIncludeNulls extends CudfFirstLastBase
- class CudfMax extends CudfAggregate
- class CudfMin extends CudfAggregate
- class CudfSum extends CudfAggregate
- abstract class CudfUnaryMathExpression extends GpuUnaryMathExpression with CudfUnaryExpression
- case class GpuAbs(child: Expression) extends GpuUnaryExpression with CudfUnaryExpression with ExpectsInputTypes with NullIntolerant with Product with Serializable
- case class GpuAcos(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
- case class GpuAcoshCompat(child: Expression) extends GpuUnaryMathExpression with Product with Serializable
- case class GpuAcoshImproved(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
- case class GpuAdd(left: Expression, right: Expression) extends CudfBinaryArithmetic with Product with Serializable
- case class GpuAggregateExpression(origAggregateFunction: GpuAggregateFunction, mode: AggregateMode, isDistinct: Boolean, filter: Option[Expression], resultId: ExprId) extends Expression with GpuExpression with GpuUnevaluable with Product with Serializable
- trait GpuAggregateFunction extends Expression with GpuExpression with GpuUnevaluable
- case class GpuAnd(left: Expression, right: Expression) extends CudfBinaryOperator with Predicate with Product with Serializable
-
case class
GpuArrayContains(left: Expression, right: Expression) extends BinaryExpression with GpuBinaryExpression with NullIntolerant with Product with Serializable
Checks if the array (left) has the element (right)
- case class GpuAsin(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
- case class GpuAsinhCompat(child: Expression) extends GpuUnaryMathExpression with Product with Serializable
- case class GpuAsinhImproved(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
- case class GpuAtan(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
- case class GpuAtanh(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
- case class GpuAverage(child: Expression) extends Expression with GpuAggregateFunction with GpuAggregateWindowFunction[MeanAggregation] with Product with Serializable
- case class GpuBRound(child: Expression, scale: Expression) extends GpuRoundBase with Product with Serializable
- case class GpuBitwiseAnd(left: Expression, right: Expression) extends CudfBinaryArithmetic with Product with Serializable
- case class GpuBitwiseNot(child: Expression) extends GpuUnaryExpression with CudfUnaryExpression with ExpectsInputTypes with Product with Serializable
- case class GpuBitwiseOr(left: Expression, right: Expression) extends CudfBinaryArithmetic with Product with Serializable
- case class GpuBitwiseXor(left: Expression, right: Expression) extends CudfBinaryArithmetic with Product with Serializable
- class GpuCartesianPartition extends Partition
- case class GpuCartesianProductExec(left: SparkPlan, right: SparkPlan, condition: Option[Expression], targetSizeBytes: Long) extends SparkPlan with BinaryExecNode with GpuExec with Product with Serializable
- class GpuCartesianRDD extends RDD[ColumnarBatch] with Serializable with Arm
- case class GpuCbrt(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
- case class GpuCeil(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
- trait GpuCollectBase[T <: Aggregation with RollingAggregation[T]] extends Expression with GpuAggregateFunction with GpuAggregateWindowFunction[T]
-
case class
GpuCollectList(childExpression: Expression, mutableAggBufferOffset: Int = 0, inputAggBufferOffset: Int = 0) extends Expression with GpuCollectBase[CollectListAggregation] with Product with Serializable
Collects and returns a list of non-unique elements.
Collects and returns a list of non-unique elements.
The two 'offset' parameters are not used by GPU version, but are here for the compatibility with the CPU version and automated checks.
-
case class
GpuCollectSet(childExpression: Expression, mutableAggBufferOffset: Int = 0, inputAggBufferOffset: Int = 0) extends Expression with GpuCollectBase[CollectSetAggregation] with Product with Serializable
Collects and returns a set of unique elements.
Collects and returns a set of unique elements.
The two 'offset' parameters are not used by GPU version, but are here for the compatibility with the CPU version and automated checks.
- case class GpuConcat(children: Seq[Expression]) extends Expression with GpuComplexTypeMergingExpression with Product with Serializable
- case class GpuConcatWs(children: Seq[Expression]) extends Expression with GpuExpression with ImplicitCastInputTypes with Product with Serializable
- case class GpuContains(left: Expression, right: Expression) extends BinaryExpression with GpuBinaryExpression with Predicate with ImplicitCastInputTypes with NullIntolerant with Product with Serializable
- case class GpuCos(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
- case class GpuCosh(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
- case class GpuCot(child: Expression) extends GpuUnaryMathExpression with Product with Serializable
- case class GpuCount(children: Seq[Expression]) extends Expression with GpuAggregateFunction with GpuBatchedRunningWindowWithFixer with GpuAggregateWindowFunction[CountAggregation] with GpuRunningWindowFunction with Product with Serializable
- case class GpuCreateArray(children: Seq[Expression], useStringTypeWhenEmpty: Boolean) extends Expression with GpuExpression with Product with Serializable
- case class GpuCreateDataSourceTableAsSelectCommand(table: CatalogTable, mode: SaveMode, query: LogicalPlan, outputColumnNames: Seq[String], origProvider: Class[_], gpuFileFormat: ColumnarFileFormat) extends LogicalPlan with GpuDataWritingCommand with Product with Serializable
- case class GpuCreateNamedStruct(children: Seq[Expression]) extends Expression with GpuExpression with Product with Serializable
-
case class
GpuDataSource(sparkSession: SparkSession, className: String, paths: Seq[String] = Nil, userSpecifiedSchema: Option[StructType] = None, partitionColumns: Seq[String] = Seq.empty, bucketSpec: Option[BucketSpec] = None, options: Map[String, String] = Map.empty, catalogTable: Option[CatalogTable] = None, origProvider: Class[_], gpuFileFormat: ColumnarFileFormat) extends Logging with Product with Serializable
A truncated version of Spark DataSource that converts to use the GPU version of InsertIntoHadoopFsRelationCommand for FileFormats we support.
A truncated version of Spark DataSource that converts to use the GPU version of InsertIntoHadoopFsRelationCommand for FileFormats we support. This does not support DataSource V2 writing at this point because at the time of copying, it did not.
-
trait
GpuDataSourceScanExec extends SparkPlan with LeafExecNode
GPU implementation of Spark's
DataSourceScanExec - case class GpuDateAdd(startDate: Expression, days: Expression) extends BinaryExpression with GpuDateMathBase with Product with Serializable
- case class GpuDateAddInterval(start: Expression, interval: Expression, timeZoneId: Option[String] = None) extends GpuTimeMath with Product with Serializable
- case class GpuDateDiff(endDate: Expression, startDate: Expression) extends BinaryExpression with GpuBinaryExpression with ImplicitCastInputTypes with Product with Serializable
- case class GpuDateFormatClass(timestamp: Expression, format: Expression, strfFormat: String, timeZoneId: Option[String] = None) extends BinaryExpression with GpuBinaryExpression with TimeZoneAwareExpression with ImplicitCastInputTypes with Product with Serializable
- trait GpuDateMathBase extends BinaryExpression with GpuBinaryExpression with ExpectsInputTypes
- case class GpuDateSub(startDate: Expression, days: Expression) extends BinaryExpression with GpuDateMathBase with Product with Serializable
- trait GpuDateUnaryExpression extends GpuUnaryExpression with ImplicitCastInputTypes
- case class GpuDayOfMonth(child: Expression) extends GpuUnaryExpression with GpuDateUnaryExpression with Product with Serializable
- case class GpuDayOfWeek(child: Expression) extends GpuUnaryExpression with GpuDateUnaryExpression with Product with Serializable
- case class GpuDayOfYear(child: Expression) extends GpuUnaryExpression with GpuDateUnaryExpression with Product with Serializable
- trait GpuDivModLike extends CudfBinaryArithmetic
- case class GpuDivide(left: Expression, right: Expression, failOnErrorOverride: Boolean = ...) extends CudfBinaryArithmetic with GpuDivModLike with Product with Serializable
-
class
GpuDynamicPartitionDataWriter extends GpuFileFormatDataWriter
Writes data to using dynamic partition writes, meaning this single function can write to multiple directories (partitions) or files (bucketing).
- case class GpuElementAt(left: Expression, right: Expression, failOnError: Boolean) extends BinaryExpression with GpuBinaryExpression with ExpectsInputTypes with Product with Serializable
-
class
GpuEmptyDirectoryDataWriter extends GpuFileFormatDataWriter
GPU data writer for empty partitions
- case class GpuEndsWith(left: Expression, right: Expression) extends BinaryExpression with GpuBinaryExpression with Predicate with ImplicitCastInputTypes with NullIntolerant with Product with Serializable
- case class GpuEqualNullSafe(left: Expression, right: Expression) extends CudfBinaryComparison with NullIntolerant with Product with Serializable
-
case class
GpuEqualTo(left: Expression, right: Expression) extends CudfBinaryComparison with NullIntolerant with Product with Serializable
The table below shows how the result is calculated for Equal-to.
The table below shows how the result is calculated for Equal-to. To make calculation easier we are leveraging the fact that the cudf-result(r) always returns false. So that result is used in place of false when needed.
Return (lhs.nan && rhs.nan) || result[i]
+-------------+------------+------------------+---------------+----+ | lhs.isNan()| rhs.isNan | cudf-result(r) | final-result | eq | +-------------+------------+------------------+---------------+----+ | t | f | f | r | f | | f | t | f | r | f | | t | t | f | t | t | | f | f | r | r | na | +-------------+------------+------------------+---------------+----+
- case class GpuExp(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
- case class GpuExpm1(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
-
abstract
class
GpuFileFormatDataWriter extends DataWriter[ColumnarBatch]
Abstract class for writing out data in a single Spark task using the GPU.
Abstract class for writing out data in a single Spark task using the GPU. This is the GPU version of
org.apache.spark.sql.execution.datasources.FileFormatDataWriter. -
case class
GpuFileSourceScanExec(relation: HadoopFsRelation, output: Seq[Attribute], requiredSchema: StructType, partitionFilters: Seq[Expression], optionalBucketSet: Option[BitSet], optionalNumCoalescedBuckets: Option[Int], dataFilters: Seq[Expression], tableIdentifier: Option[TableIdentifier], queryUsesInputFile: Boolean = false)(rapidsConf: RapidsConf) extends SparkPlan with GpuDataSourceScanExec with GpuExec with Product with Serializable
GPU version of Spark's
FileSourceScanExecGPU version of Spark's
FileSourceScanExec- relation
The file-based relation to scan.
- output
Output attributes of the scan, including data attributes and partition attributes.
- requiredSchema
Required schema of the underlying relation, excluding partition columns.
- partitionFilters
Predicates to use for partition pruning.
- optionalBucketSet
Bucket ids for bucket pruning.
- optionalNumCoalescedBuckets
Number of coalesced buckets.
- dataFilters
Filters on non-partition columns.
- tableIdentifier
identifier for the table in the metastore.
- queryUsesInputFile
This is a parameter to easily allow turning it off in GpuTransitionOverrides if InputFileName, InputFileBlockStart, or InputFileBlockLength are used
- rapidsConf
Rapids conf
- case class GpuFirst(child: Expression, ignoreNulls: Boolean) extends Expression with GpuAggregateFunction with ImplicitCastInputTypes with Serializable with Product
- case class GpuFloor(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
- case class GpuFromUnixTime(sec: Expression, format: Expression, strfFormat: String, timeZoneId: Option[String] = None) extends BinaryExpression with GpuBinaryExpression with TimeZoneAwareExpression with ImplicitCastInputTypes with Product with Serializable
-
case class
GpuGetArrayItem(child: Expression, ordinal: Expression, failOnError: Boolean) extends BinaryExpression with GpuBinaryExpression with ExpectsInputTypes with ExtractValue with Product with Serializable
Returns the field at
ordinalin the Arraychild.Returns the field at
ordinalin the Arraychild.We need to do type checking here as
ordinalexpression maybe unresolved. - class GpuGetArrayItemMeta extends BinaryExprMeta[GetArrayItem]
- case class GpuGetMapValue(child: Expression, key: Expression, failOnError: Boolean) extends BinaryExpression with GpuBinaryExpression with ImplicitCastInputTypes with NullIntolerant with Product with Serializable
- class GpuGetMapValueMeta extends BinaryExprMeta[GetMapValue]
- case class GpuGetStructField(child: Expression, ordinal: Int, name: Option[String] = None) extends UnaryExpression with GpuExpression with ExtractValue with NullIntolerant with Product with Serializable
- case class GpuGetTimestamp(strTs: Expression, format: Expression, sparkFormat: String, strf: String, timeZoneId: Option[String] = None) extends GpuToTimestamp with Product with Serializable
-
case class
GpuGreaterThan(left: Expression, right: Expression) extends CudfBinaryComparison with NullIntolerant with Product with Serializable
The table below shows how the result is calculated for greater-than.
The table below shows how the result is calculated for greater-than. To make calculation easier we are leveraging the fact that the cudf-result(r) always returns false. So that result is used in place of false when needed.
In this case return (lhs.nan && !lhs.nan) || result[i]
+-------------+------------+-----------------+---------------+----+ | lhs.isNan()| rhs.isNan | cudf-result(r) | final-result | gt | +-------------+------------+-----------------+---------------+----+ | t | f | f | t | t | | f | t | f | r | f | | t | t | f | r | f | | f | f | r | r | na | +-------------+------------+-----------------+---------------+----+
-
case class
GpuGreaterThanOrEqual(left: Expression, right: Expression) extends CudfBinaryComparison with NullIntolerant with Product with Serializable
The table below shows how the result is calculated for Greater-than-Eq.
The table below shows how the result is calculated for Greater-than-Eq. To make calculation easier we are leveraging the fact that the cudf-result(r) always returns false. So that result is used in place of false when needed.
In this case return lhs.isNan || result[i]
+-------------+------------+-----------------+---------------+-----+ | lhs.isNan()| rhs.isNan | cudf-result(r) | final-result | gte | +-------------+------------+-----------------+---------------+-----+ | t | f | f | t | t | | f | t | f | r | f | | t | t | f | t | t | | f | f | r | r | NA | +-------------+------------+-----------------+---------------+-----+
- case class GpuGreatest(children: Seq[Expression]) extends Expression with GpuGreatestLeastBase with Product with Serializable
- trait GpuGreatestLeastBase extends Expression with ComplexTypeMergingExpression with GpuExpression
- case class GpuHour(child: Expression, timeZoneId: Option[String] = None) extends GpuUnaryExpression with GpuTimeUnaryExpression with Product with Serializable
- case class GpuInitCap(child: Expression) extends GpuUnaryExpression with ImplicitCastInputTypes with Product with Serializable
-
case class
GpuInputFileBlockLength() extends GpuLeafExpression with Product with Serializable
Returns the length of the block being read, or -1 if not available.
Returns the length of the block being read, or -1 if not available. This is extra difficult because we cannot coalesce batches in between when this is used and the input file or else we could run into problems with returning the wrong thing.
-
case class
GpuInputFileBlockStart() extends GpuLeafExpression with Product with Serializable
Returns the start offset of the block being read, or -1 if not available.
Returns the start offset of the block being read, or -1 if not available. This is extra difficult because we cannot coalesce batches in between when this is used and the input file or else we could run into problems with returning the wrong thing.
-
case class
GpuInputFileName() extends GpuLeafExpression with Product with Serializable
Returns the name of the file being read, or empty string if not available.
Returns the name of the file being read, or empty string if not available. This is extra difficult because we cannot coalesce batches in between when this is used and the input file or else we could run into problems with returning the wrong thing.
- case class GpuInsertIntoHadoopFsRelationCommand(outputPath: Path, staticPartitions: TablePartitionSpec, ifPartitionNotExists: Boolean, partitionColumns: Seq[Attribute], bucketSpec: Option[BucketSpec], fileFormat: ColumnarFileFormat, options: Map[String, String], query: LogicalPlan, mode: SaveMode, catalogTable: Option[CatalogTable], fileIndex: Option[FileIndex], outputColumnNames: Seq[String]) extends LogicalPlan with GpuDataWritingCommand with Product with Serializable
- case class GpuIntegralDivide(left: Expression, right: Expression) extends CudfBinaryArithmetic with GpuDivModLike with Product with Serializable
- case class GpuLast(child: Expression, ignoreNulls: Boolean) extends Expression with GpuAggregateFunction with ImplicitCastInputTypes with Serializable with Product
- case class GpuLastDay(startDate: Expression) extends GpuUnaryExpression with ImplicitCastInputTypes with Product with Serializable
- case class GpuLeast(children: Seq[Expression]) extends Expression with GpuGreatestLeastBase with Product with Serializable
- case class GpuLength(child: Expression) extends GpuUnaryExpression with ExpectsInputTypes with Product with Serializable
-
case class
GpuLessThan(left: Expression, right: Expression) extends CudfBinaryComparison with NullIntolerant with Product with Serializable
The table below shows how the result is calculated for Less-than.
The table below shows how the result is calculated for Less-than. To make calculation easier we are leveraging the fact that the cudf-result(r) always returns false. So that result is used in place of false when needed.
In this case return !lhs.nan && rhs.nan || result[i]
+-------------+------------+-----------------+---------------+-----+ | lhs.isNan()| rhs.isNan | cudf-result(r) | final-result | lt | +-------------+------------+-----------------+---------------+-----+ | t | f | f | r | f | | f | t | f | t | t | | t | t | f | r | f | | f | f | r | r | NA | +-------------+------------+-----------------+---------------+-----+
-
case class
GpuLessThanOrEqual(left: Expression, right: Expression) extends CudfBinaryComparison with NullIntolerant with Product with Serializable
The table below shows how the result is calculated for Less-than-Eq.
The table below shows how the result is calculated for Less-than-Eq. To make calculation easier we are leveraging the fact that the cudf-result(r) always returns false. So that result is used in place of false when needed.
In this case, return rhs.nan || result[i]
+-------------+------------+------------------+---------------+-----+ | lhs.isNan()| rhs.isNan | cudf-result(r) | final-result | lte | +-------------+------------+------------------+---------------+-----+ | t | f | f | r | f | | f | t | f | t | t | | t | t | f | t | t | | f | f | r | r | NA | +-------------+------------+------------------+---------------+-----+
- case class GpuLike(left: Expression, right: Expression, escapeChar: Char) extends BinaryExpression with GpuBinaryExpression with ImplicitCastInputTypes with NullIntolerant with Product with Serializable
- case class GpuLog(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
- case class GpuLogarithm(left: Expression, right: Expression) extends CudfBinaryMathExpression with Product with Serializable
- case class GpuLower(child: Expression) extends GpuUnaryString2StringExpression with Product with Serializable
- case class GpuMax(child: Expression) extends Expression with GpuAggregateFunction with GpuBatchedRunningWindowWithFixer with GpuAggregateWindowFunction[MaxAggregation] with GpuRunningWindowFunction with Product with Serializable
- case class GpuMd5(child: Expression) extends GpuUnaryExpression with ImplicitCastInputTypes with NullIntolerant with Product with Serializable
- case class GpuMin(child: Expression) extends Expression with GpuAggregateFunction with GpuBatchedRunningWindowWithFixer with GpuAggregateWindowFunction[MinAggregation] with GpuRunningWindowFunction with Product with Serializable
- case class GpuMinute(child: Expression, timeZoneId: Option[String] = None) extends GpuUnaryExpression with GpuTimeUnaryExpression with Product with Serializable
- case class GpuMonth(child: Expression) extends GpuUnaryExpression with GpuDateUnaryExpression with Product with Serializable
- case class GpuMultiply(left: Expression, right: Expression) extends CudfBinaryArithmetic with Product with Serializable
- case class GpuMurmur3Hash(children: Seq[Expression], seed: Int) extends Expression with GpuExpression with Product with Serializable
- case class GpuNormalizeNaNAndZero(child: Expression) extends GpuUnaryExpression with ExpectsInputTypes with Product with Serializable
- case class GpuNot(child: Expression) extends GpuUnaryExpression with CudfUnaryExpression with Predicate with ImplicitCastInputTypes with NullIntolerant with Product with Serializable
- case class GpuOr(left: Expression, right: Expression) extends CudfBinaryOperator with Predicate with Product with Serializable
- class GpuOrcFileFormat extends ColumnarFileFormat with Logging
- class GpuOrcWriter extends ColumnarOutputWriter
- case class GpuPivotFirst(pivotColumn: Expression, valueColumn: Expression, pivotColumnValues: Seq[Any]) extends Expression with GpuAggregateFunction with Product with Serializable
- case class GpuPmod(left: Expression, right: Expression) extends CudfBinaryArithmetic with GpuDivModLike with Product with Serializable
- case class GpuPow(left: Expression, right: Expression) extends CudfBinaryMathExpression with Product with Serializable
- trait GpuPredicateHelper extends AnyRef
- case class GpuQuarter(child: Expression) extends GpuUnaryExpression with GpuDateUnaryExpression with Product with Serializable
- case class GpuRemainder(left: Expression, right: Expression) extends CudfBinaryArithmetic with GpuDivModLike with Product with Serializable
- case class GpuRint(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
- case class GpuRound(child: Expression, scale: Expression) extends GpuRoundBase with Product with Serializable
- abstract class GpuRoundBase extends BinaryExpression with GpuBinaryExpression with Serializable with ImplicitCastInputTypes
- case class GpuScalaUDF(function: RapidsUDF, dataType: DataType, children: Seq[Expression], udfName: Option[String], nullable: Boolean, udfDeterministic: Boolean) extends Expression with GpuUserDefinedFunction with Product with Serializable
-
case class
GpuScalarSubquery(plan: BaseSubqueryExec, exprId: ExprId) extends ExecSubqueryExpression with GpuExpression with Product with Serializable
GPU placeholder of ScalarSubquery, which returns the scalar result with columnarEval method.
GPU placeholder of ScalarSubquery, which returns the scalar result with columnarEval method. This placeholder is to make ScalarSubquery working as a GPUExpression to cooperate other GPU overrides.
- case class GpuSecond(child: Expression, timeZoneId: Option[String] = None) extends GpuUnaryExpression with GpuTimeUnaryExpression with Product with Serializable
-
class
GpuSerializableBatch extends Serializable with AutoCloseable with Arm
- Annotations
- @SerialVersionUID()
- trait GpuShiftBase extends BinaryExpression with GpuBinaryExpression with ImplicitCastInputTypes
- case class GpuShiftLeft(left: Expression, right: Expression) extends BinaryExpression with GpuShiftBase with Product with Serializable
- case class GpuShiftRight(left: Expression, right: Expression) extends BinaryExpression with GpuShiftBase with Product with Serializable
- case class GpuShiftRightUnsigned(left: Expression, right: Expression) extends BinaryExpression with GpuShiftBase with Product with Serializable
- class GpuShuffleBlockResolver extends ShuffleBlockResolver with Logging
- class GpuShuffleDependency[K, V, C] extends ShuffleDependency[K, V, C]
- class GpuShuffleEnv extends Logging
- class GpuShuffleHandle[K, V] extends BaseShuffleHandle[K, V, V]
- case class GpuSignum(child: Expression) extends GpuUnaryMathExpression with Product with Serializable
- case class GpuSin(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
-
class
GpuSingleDirectoryDataWriter extends GpuFileFormatDataWriter
Writes data to a single directory (used for non-dynamic-partition writes).
- case class GpuSinh(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
- case class GpuSize(child: Expression, legacySizeOfNull: Boolean) extends GpuUnaryExpression with Product with Serializable
- case class GpuSortArray(base: Expression, ascendingOrder: Expression) extends BinaryExpression with GpuBinaryExpression with ExpectsInputTypes with Product with Serializable
- case class GpuSqrt(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
- case class GpuStartsWith(left: Expression, right: Expression) extends BinaryExpression with GpuBinaryExpression with Predicate with ImplicitCastInputTypes with NullIntolerant with Product with Serializable
- case class GpuStringLPad(str: Expression, len: Expression, pad: Expression) extends TernaryExpression with BasePad with Product with Serializable
- case class GpuStringLocate(substr: Expression, col: Expression, start: Expression) extends TernaryExpression with GpuTernaryExpression with ImplicitCastInputTypes with Product with Serializable
- case class GpuStringRPad(str: Expression, len: Expression, pad: Expression) extends TernaryExpression with BasePad with Product with Serializable
- case class GpuStringReplace(srcExpr: Expression, searchExpr: Expression, replaceExpr: Expression) extends TernaryExpression with GpuTernaryExpression with ImplicitCastInputTypes with Product with Serializable
- case class GpuStringSplit(str: Expression, regex: Expression, limit: Expression) extends TernaryExpression with GpuTernaryExpression with ImplicitCastInputTypes with Product with Serializable
- class GpuStringSplitMeta extends TernaryExprMeta[StringSplit]
- case class GpuStringTrim(column: Expression, trimParameters: Option[Expression] = None) extends Expression with GpuString2TrimExpression with ImplicitCastInputTypes with Product with Serializable
- case class GpuStringTrimLeft(column: Expression, trimParameters: Option[Expression] = None) extends Expression with GpuString2TrimExpression with ImplicitCastInputTypes with Product with Serializable
- case class GpuStringTrimRight(column: Expression, trimParameters: Option[Expression] = None) extends Expression with GpuString2TrimExpression with ImplicitCastInputTypes with Product with Serializable
- case class GpuSubstring(str: Expression, pos: Expression, len: Expression) extends TernaryExpression with GpuTernaryExpression with ImplicitCastInputTypes with NullIntolerant with Product with Serializable
- case class GpuSubstringIndex(strExpr: Expression, regexp: String, ignoredDelimExpr: Expression, ignoredCountExpr: Expression) extends TernaryExpression with GpuTernaryExpression with ImplicitCastInputTypes with Product with Serializable
- case class GpuSubtract(left: Expression, right: Expression) extends CudfBinaryArithmetic with Product with Serializable
- case class GpuSum(child: Expression, resultType: DataType) extends Expression with GpuAggregateFunction with ImplicitCastInputTypes with GpuBatchedRunningWindowWithFixer with GpuAggregateWindowFunction[SumAggregation] with GpuRunningWindowFunction with Product with Serializable
- case class GpuTan(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
- case class GpuTanh(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
- case class GpuTimeAdd(start: Expression, interval: Expression, timeZoneId: Option[String] = None) extends GpuTimeMath with Product with Serializable
- abstract class GpuTimeMath extends BinaryExpression with GpuExpression with TimeZoneAwareExpression with ExpectsInputTypes with Serializable
- case class GpuTimeSub(start: Expression, interval: Expression, timeZoneId: Option[String] = None) extends GpuTimeMath with Product with Serializable
- trait GpuTimeUnaryExpression extends GpuUnaryExpression with TimeZoneAwareExpression with ImplicitCastInputTypes with NullIntolerant
- case class GpuToDegrees(child: Expression) extends GpuUnaryMathExpression with Product with Serializable
- case class GpuToRadians(child: Expression) extends GpuUnaryMathExpression with Product with Serializable
-
abstract
class
GpuToTimestamp extends BinaryExpression with GpuBinaryExpression with TimeZoneAwareExpression with ExpectsInputTypes
A direct conversion of Spark's ToTimestamp class which converts time to UNIX timestamp by first converting to microseconds and then dividing by the downScaleFactor
-
abstract
class
GpuToTimestampImproved extends GpuToTimestamp
An improved version of GpuToTimestamp conversion which converts time to UNIX timestamp without first converting to microseconds
- case class GpuToUnixTimestamp(strTs: Expression, format: Expression, sparkFormat: String, strf: String, timeZoneId: Option[String] = None) extends GpuToTimestamp with Product with Serializable
- case class GpuToUnixTimestampImproved(strTs: Expression, format: Expression, sparkFormat: String, strf: String, timeZoneId: Option[String] = None) extends GpuToTimestampImproved with Product with Serializable
- abstract class GpuUnaryMathExpression extends GpuUnaryExpression with Serializable with ImplicitCastInputTypes
- case class GpuUnaryMinus(child: Expression) extends GpuUnaryExpression with ExpectsInputTypes with NullIntolerant with Product with Serializable
- case class GpuUnaryPositive(child: Expression) extends GpuUnaryExpression with ExpectsInputTypes with NullIntolerant with Product with Serializable
- abstract class GpuUnaryString2StringExpression extends GpuUnaryExpression with ExpectsInputTypes
- case class GpuUnixTimestamp(strTs: Expression, format: Expression, sparkFormat: String, strf: String, timeZoneId: Option[String] = None) extends GpuToTimestamp with Product with Serializable
- case class GpuUnixTimestampImproved(strTs: Expression, format: Expression, sparkFormat: String, strf: String, timeZoneId: Option[String] = None) extends GpuToTimestampImproved with Product with Serializable
- case class GpuUpper(child: Expression) extends GpuUnaryString2StringExpression with Product with Serializable
- case class GpuWeekDay(child: Expression) extends GpuUnaryExpression with GpuDateUnaryExpression with Product with Serializable
-
class
GpuWriteJobDescription extends Serializable
A shared job description for all the GPU write tasks.
A shared job description for all the GPU write tasks. This is the GPU version of
org.apache.spark.sql.execution.datasources.WriteJobDescription. -
class
GpuWriteJobStatsTracker extends BasicColumnarWriteJobStatsTracker
Simple ColumnarWriteJobStatsTracker implementation that's serializable, capable of instantiating GpuWriteTaskStatsTracker on executors and processing the
WriteTaskStatsthey produce by aggregating the metrics and posting them as DriverMetricUpdates. -
class
GpuWriteTaskStatsTracker extends BasicColumnarWriteTaskStatsTracker
ColumnarWriteTaskStatsTracker implementation that produces
WriteTaskStatsand tracks writing times per task. - case class GpuYear(child: Expression) extends GpuUnaryExpression with GpuDateUnaryExpression with Product with Serializable
- case class LegacyParseFormat(separator: Char, isTimestamp: Boolean, validRegex: String) extends Product with Serializable
-
trait
OrcFiltersBase extends AnyRef
Methods that can be shared when upgrading the built-in Hive.
Methods that can be shared when upgrading the built-in Hive.
Derived from Apache Spark to avoid depending upon it directly, since its API has changed between Spark versions.
- class RapidsCachingReader[K, C] extends ShuffleReader[K, C] with Arm with Logging
- class RapidsCachingWriter[K, V] extends ShuffleWriter[K, V] with Logging
-
class
RapidsDiskBlockManager extends AnyRef
Maps logical blocks to local disk locations.
-
abstract
class
RapidsShuffleInternalManagerBase extends ShuffleManager with RapidsShuffleHeartbeatHandler with Logging
A shuffle manager optimized for the RAPIDS Plugin For Apache Spark.
A shuffle manager optimized for the RAPIDS Plugin For Apache Spark.
- Note
This is an internal class to obtain access to the private
ShuffleManagerandSortShuffleManagerclasses. When configuring Apache Spark to use the RAPIDS shuffle manager,
- case class RegexReplace(search: String, replace: String) extends Product with Serializable
- trait ShuffleManagerShimBase extends AnyRef
- trait ShuffleMetricsUpdater extends AnyRef
- class SubstringIndexMeta extends TernaryExprMeta[SubstringIndex]
- case class TempSpillBufferId extends RapidsBufferId with Product with Serializable
- sealed trait TimeParserPolicy extends Serializable
- abstract class UnixTimeExprMeta[A <: BinaryExpression with TimeZoneAwareExpression] extends BinaryExprMeta[A]
- case class WrappedAggFunction(aggregateFunction: GpuAggregateFunction, filter: Expression) extends Expression with GpuAggregateFunction with Product with Serializable
Value Members
- object BasicColumnarWriteJobStatsTracker extends Serializable
- object CorrectedTimeParserPolicy extends TimeParserPolicy
- object CudfRegexp
- object ExceptionTimeParserPolicy extends TimeParserPolicy
- object GpuDataSource extends Logging with Serializable
- object GpuDataSourceScanExec extends Serializable
- object GpuDivModLike
- object GpuDivideUtil
-
object
GpuFileFormatWriter extends Logging
A helper object for writing columnar data out to a location.
- object GpuFileSourceScanExec extends Serializable
- object GpuLogarithm extends Arm with Serializable
- object GpuLogicHelper
- object GpuMultiplyUtil
- object GpuMurmur3Hash extends Arm with Serializable
- object GpuOrcFileFormat extends Logging
- object GpuScalaUDF extends Serializable
- object GpuShuffleEnv extends Logging
- object GpuSubstringIndex extends Serializable
- object GpuToTimestamp extends Arm
- object GpuWriteJobStatsTracker extends Serializable
- object InputFileUtils
- object LegacyTimeParserPolicy extends TimeParserPolicy
-
object
OrcFilters extends OrcFiltersBase
Helper object for building ORC
SearchArguments, which are used for ORC predicate push-down.Helper object for building ORC
SearchArguments, which are used for ORC predicate push-down.Due to limitation of ORC
SearchArgumentbuilder, we had to implement separate checking and conversion passes through the Filter to make sure we only convert predicates that are known to be convertible.An ORC
SearchArgumentmust be built in one pass using a single builder. For example, you can't builda = 1andb = 2first, and then combine them intoa = 1 AND b = 2. This is quite different from the cases in Spark SQL or Parquet, where complex filters can be easily built using existing simpler ones.The annoying part is that,
SearchArgumentbuilder methods likestartAnd(),startOr(), andstartNot()mutate internal state of the builder instance. This forces us to translate all convertible filters with a single builder instance. However, if we try to translate a filter before checking whether it can be converted or not, we may end up with a builder whose internal state is inconsistent in the case of an inconvertible filter.For example, to convert an
Andfilter with builderb, we callb.startAnd()first, and then try to convert its children. Say we convertleftchild successfully, but find thatrightchild is inconvertible. Alas,b.startAnd()call can't be rolled back, andbis inconsistent now.The workaround employed here is to trim the Spark filters before trying to convert them. This way, we can only do the actual conversion on the part of the Filter that is known to be convertible.
P.S.: Hive seems to use
SearchArgumenttogether withExprNodeGenericFuncDesconly. Usage of builder methods mentioned above can only be found in test code, where all tested filters are known to be convertible. - object RapidsShuffleInternalManagerBase extends Logging
- object ShiftHelper extends Arm
- object TempSpillBufferId extends Serializable