package rapids
- Alphabetic
- Public
- All
Type Members
- trait AdaptiveSparkPlanHelperShim extends AnyRef
- case class AvroBatchContext(origChunkedBlocks: LinkedHashMap[Path, ArrayBuffer[DataBlockBase]], schema: SchemaBase, mergedHeader: Header) extends BatchContext with Product with Serializable
-
case class
AvroBlockMeta(header: Header, headerSize: Long, blocks: Seq[BlockInfo]) extends Product with Serializable
Avro block meta info
Avro block meta info
- header
the header of avro file
- blocks
the total block info of avro file
-
case class
AvroDataBlock(blockInfo: BlockInfo) extends DataBlockBase with Product with Serializable
avro BlockInfo wrapper
-
case class
AvroExtraInfo() extends ExtraInfo with Product with Serializable
Extra information
-
case class
AvroFileFilterHandler(hadoopConf: Configuration, options: AvroOptions) extends Logging with Product with Serializable
A tool to filter Avro blocks
- class AvroProviderImpl extends AvroProvider
-
case class
AvroSchemaWrapper(schema: Schema) extends SchemaBase with Product with Serializable
avro schema wrapper
- case class AvroSingleDataBlockInfo(filePath: Path, dataBlock: AvroDataBlock, partitionValues: InternalRow, schema: AvroSchemaWrapper, readSchema: StructType, extraInfo: AvroExtraInfo) extends SingleDataBlockInfo with Product with Serializable
- trait BasePad extends TernaryExpression with GpuTernaryExpressionArgsAnyScalarScalar with ImplicitCastInputTypes with NullIntolerant
-
class
BasicColumnarWriteJobStatsTracker extends ColumnarWriteJobStatsTracker
Simple ColumnarWriteJobStatsTracker implementation that's serializable, capable of instantiating BasicColumnarWriteTaskStatsTracker on executors and processing the
BasicColumnarWriteTaskStatsthey produce by aggregating the metrics and posting them as DriverMetricUpdates. -
case class
BasicColumnarWriteTaskStats(numPartitions: Int, numFiles: Int, numBytes: Long, numRows: Long) extends WriteTaskStats with Product with Serializable
Simple metrics collected during an instance of GpuFileFormatDataWriter.
Simple metrics collected during an instance of GpuFileFormatDataWriter. These were first introduced in https://github.com/apache/spark/pull/18159 (SPARK-20703).
-
class
BasicColumnarWriteTaskStatsTracker extends ColumnarWriteTaskStatsTracker with Logging
Simple metrics collected during an instance of GpuFileFormatDataWriter.
Simple metrics collected during an instance of GpuFileFormatDataWriter. This is the columnar version of
org.apache.spark.sql.execution.datasources.BasicWriteTaskStatsTracker. - class BytesInFlightLimiter extends AnyRef
-
trait
ColumnarWriteJobStatsTracker extends Serializable
A class implementing this trait is basically a collection of parameters that are necessary for instantiating a (derived type of) ColumnarWriteTaskStatsTracker on all executors and then process the statistics produced by them (e.g.
A class implementing this trait is basically a collection of parameters that are necessary for instantiating a (derived type of) ColumnarWriteTaskStatsTracker on all executors and then process the statistics produced by them (e.g. save them to memory/disk, issue warnings, etc). It is therefore important that such an objects is
Serializable, as it will be sent from the driver to all executors. -
trait
ColumnarWriteTaskStatsTracker extends AnyRef
A trait for classes that are capable of collecting statistics on columnar data that's being processed by a single write task in GpuFileFormatDataWriter - i.e.
A trait for classes that are capable of collecting statistics on columnar data that's being processed by a single write task in GpuFileFormatDataWriter - i.e. there should be one instance per executor.
newPartition event is only triggered if the relation to be written out is partitioned.
- abstract class ConvertUTCTimestampExprMetaBase[INPUT <: BinaryExpression] extends BinaryExprMeta[INPUT]
- abstract class CudfBinaryArithmetic extends CudfBinaryOperator with NullIntolerant
- abstract class CudfBinaryComparison extends CudfBinaryOperator with Predicate
- abstract class CudfBinaryMathExpression extends BinaryExpression with CudfBinaryExpression with Serializable with ImplicitCastInputTypes
- abstract class CudfBinaryPredicateWithSideEffect extends CudfBinaryOperator with Predicate
- abstract class CudfUnaryMathExpression extends GpuUnaryMathExpression with CudfUnaryExpression
-
class
ExecutionPlanCaptureCallback extends QueryExecutionListener
Used as a part of testing to capture the executed query plan.
- trait ExecutionPlanCaptureCallbackBase extends AnyRef
- class FromUTCTimestampExprMeta extends ConvertUTCTimestampExprMetaBase[FromUTCTimestamp]
- class FromUnixTimeMeta extends UnixTimeExprMeta[FromUnixTime]
- case class GpuAbs(child: Expression, failOnError: Boolean) extends GpuUnaryExpression with CudfUnaryExpression with ExpectsInputTypes with NullIntolerant with Product with Serializable
- case class GpuAcos(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
- case class GpuAcoshCompat(child: Expression) extends GpuUnaryMathExpression with Product with Serializable
- case class GpuAcoshImproved(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
- case class GpuAdd(left: Expression, right: Expression, failOnError: Boolean) extends GpuAddBase with Product with Serializable
- abstract class GpuAddBase extends CudfBinaryArithmetic with Serializable
- case class GpuAnd(left: Expression, right: Expression) extends CudfBinaryPredicateWithSideEffect with Product with Serializable
- trait GpuArrayBinaryLike extends Expression with GpuComplexTypeMergingExpression with NullIntolerant
-
case class
GpuArrayContains(left: Expression, right: Expression) extends BinaryExpression with GpuBinaryExpression with NullIntolerant with Product with Serializable
Checks if the array (left) has the element (right)
- case class GpuArrayExcept(left: Expression, right: Expression) extends Expression with GpuArrayBinaryLike with ExpectsInputTypes with Product with Serializable
- case class GpuArrayIntersect(left: Expression, right: Expression) extends Expression with GpuArrayBinaryLike with ExpectsInputTypes with Product with Serializable
- case class GpuArrayJoin(children: Seq[Expression]) extends Expression with GpuExpression with ShimExpression with Product with Serializable
- abstract class GpuArrayMax extends GpuUnaryExpression with ImplicitCastInputTypes with Serializable
- abstract class GpuArrayMin extends GpuUnaryExpression with ImplicitCastInputTypes with Serializable
- case class GpuArrayRemove(left: Expression, right: Expression) extends BinaryExpression with GpuBinaryExpression with Product with Serializable
- case class GpuArrayRepeat(left: Expression, right: Expression) extends BinaryExpression with GpuBinaryExpression with Product with Serializable
- case class GpuArrayUnion(left: Expression, right: Expression) extends Expression with GpuArrayBinaryLike with ExpectsInputTypes with Product with Serializable
- case class GpuArraysOverlap(left: Expression, right: Expression) extends BinaryExpression with GpuBinaryExpression with ExpectsInputTypes with NullIntolerant with Product with Serializable
- case class GpuArraysZip(children: Seq[Expression]) extends Expression with GpuExpression with ShimExpression with ExpectsInputTypes with Product with Serializable
- case class GpuAsin(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
- case class GpuAsinhCompat(child: Expression) extends GpuUnaryMathExpression with Product with Serializable
- case class GpuAsinhImproved(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
- case class GpuAtan(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
- case class GpuAtanh(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
-
case class
GpuAvroMultiFilePartitionReaderFactory(sqlConf: SQLConf, rapidsConf: RapidsConf, broadcastedConf: Broadcast[SerializableConfiguration], dataSchema: StructType, readDataSchema: StructType, partitionSchema: StructType, options: AvroOptions, metrics: Map[String, GpuMetric], filters: Array[Filter], queryUsesInputFile: Boolean) extends MultiFilePartitionReaderFactoryBase with Product with Serializable
The multi-file partition reader factory for cloud or coalescing reading of avro file format.
-
class
GpuAvroPartitionReader extends FilePartitionReaderBase with GpuAvroReaderBase
A PartitionReader that reads an AVRO file split on the GPU.
-
case class
GpuAvroPartitionReaderFactory(sqlConf: SQLConf, rapidsConf: RapidsConf, broadcastedConf: Broadcast[SerializableConfiguration], dataSchema: StructType, readDataSchema: StructType, partitionSchema: StructType, avroOptions: AvroOptions, metrics: Map[String, GpuMetric], params: Map[String, String]) extends ShimFilePartitionReaderFactory with Logging with Product with Serializable
Avro partition reader factory to build columnar reader
-
trait
GpuAvroReaderBase extends Logging
A trait collecting common methods across the 3 kinds of avro readers
- case class GpuAvroScan(sparkSession: SparkSession, fileIndex: PartitioningAwareFileIndex, dataSchema: StructType, readDataSchema: StructType, readPartitionSchema: StructType, options: CaseInsensitiveStringMap, pushedFilters: Array[Filter], rapidsConf: RapidsConf, partitionFilters: Seq[Expression] = Seq.empty, dataFilters: Seq[Expression] = Seq.empty, queryUsesInputFile: Boolean = false) extends FileScan with GpuScan with Product with Serializable
- case class GpuBRound(child: Expression, scale: Expression, outputType: DataType) extends GpuRoundBase with Product with Serializable
-
case class
GpuBasicArrayMax(child: Expression) extends GpuArrayMax with Product with Serializable
ArrayMax without
NaNhandling -
case class
GpuBasicArrayMin(child: Expression) extends GpuArrayMin with Product with Serializable
ArrayMin without
Nanhandling - case class GpuBitLength(child: Expression) extends GpuUnaryExpression with ExpectsInputTypes with Product with Serializable
- case class GpuBitwiseAnd(left: Expression, right: Expression) extends CudfBinaryArithmetic with Product with Serializable
- case class GpuBitwiseNot(child: Expression) extends GpuUnaryExpression with CudfUnaryExpression with ExpectsInputTypes with Product with Serializable
- case class GpuBitwiseOr(left: Expression, right: Expression) extends CudfBinaryArithmetic with Product with Serializable
- case class GpuBitwiseXor(left: Expression, right: Expression) extends CudfBinaryArithmetic with Product with Serializable
- class GpuCartesianPartition extends Partition
- case class GpuCartesianProductExec(left: SparkPlan, right: SparkPlan, condition: Option[Expression], targetSizeBytes: Long) extends SparkPlan with ShimBinaryExecNode with GpuExec with Product with Serializable
- class GpuCartesianRDD extends RDD[ColumnarBatch] with Serializable
- case class GpuCbrt(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
- case class GpuCeil(child: Expression, outputType: DataType) extends CudfUnaryMathExpression with Product with Serializable
- case class GpuConcat(children: Seq[Expression]) extends Expression with GpuComplexTypeMergingExpression with Product with Serializable
- case class GpuConcatWs(children: Seq[Expression]) extends Expression with GpuExpression with ShimExpression with ImplicitCastInputTypes with Product with Serializable
- case class GpuContains(left: Expression, right: Expression) extends BinaryExpression with GpuBinaryExpressionArgsAnyScalar with Predicate with ImplicitCastInputTypes with NullIntolerant with Product with Serializable
- case class GpuConv(num: Expression, fromBase: Expression, toBase: Expression) extends TernaryExpression with GpuTernaryExpression with Product with Serializable
- class GpuConvMeta extends TernaryExprMeta[Conv]
- case class GpuCos(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
- case class GpuCosh(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
- case class GpuCot(child: Expression) extends GpuUnaryMathExpression with Product with Serializable
- case class GpuCreateArray(children: Seq[Expression], useStringTypeWhenEmpty: Boolean) extends Expression with GpuExpression with ShimExpression with Product with Serializable
- case class GpuCreateMap(children: Seq[Expression], useStringTypeWhenEmpty: Boolean) extends Expression with GpuExpression with ShimExpression with Product with Serializable
- case class GpuCreateNamedStruct(children: Seq[Expression]) extends Expression with GpuExpression with ShimExpression with Product with Serializable
- case class GpuDataSource(sparkSession: SparkSession, className: String, paths: Seq[String] = Nil, userSpecifiedSchema: Option[StructType] = None, partitionColumns: Seq[String] = Seq.empty, bucketSpec: Option[BucketSpec] = None, options: Map[String, String] = Map.empty, catalogTable: Option[CatalogTable] = None, origProvider: Class[_], gpuFileFormat: ColumnarFileFormat) extends GpuDataSourceBase with Product with Serializable
-
abstract
class
GpuDataSourceBase extends Logging
A truncated version of Spark DataSource that converts to use the GPU version of InsertIntoHadoopFsRelationCommand for FileFormats we support.
A truncated version of Spark DataSource that converts to use the GPU version of InsertIntoHadoopFsRelationCommand for FileFormats we support. This does not support DataSource V2 writing at this point because at the time of copying, it did not.
-
trait
GpuDataSourceScanExec extends SparkPlan with ShimLeafExecNode with GpuExec
GPU implementation of Spark's
DataSourceScanExec - case class GpuDateAdd(startDate: Expression, days: Expression) extends BinaryExpression with GpuDateMathBase with Product with Serializable
- case class GpuDateAddInterval(start: Expression, interval: Expression, timeZoneId: Option[String] = None, ansiEnabled: Boolean = SQLConf.get.ansiEnabled) extends GpuTimeMath with Product with Serializable
- case class GpuDateDiff(endDate: Expression, startDate: Expression) extends BinaryExpression with GpuBinaryExpression with ImplicitCastInputTypes with Product with Serializable
- case class GpuDateFormatClass(timestamp: Expression, format: Expression, strfFormat: String, timeZoneId: Option[String] = None) extends BinaryExpression with GpuBinaryExpressionArgsAnyScalar with TimeZoneAwareExpression with ImplicitCastInputTypes with Product with Serializable
- trait GpuDateMathBase extends BinaryExpression with GpuBinaryExpression with ExpectsInputTypes
- case class GpuDateSub(startDate: Expression, days: Expression) extends BinaryExpression with GpuDateMathBase with Product with Serializable
- trait GpuDateUnaryExpression extends GpuUnaryExpression with ImplicitCastInputTypes
- case class GpuDayOfMonth(child: Expression) extends GpuUnaryExpression with GpuDateUnaryExpression with Product with Serializable
- case class GpuDayOfWeek(child: Expression) extends GpuUnaryExpression with GpuDateUnaryExpression with Product with Serializable
- case class GpuDayOfYear(child: Expression) extends GpuUnaryExpression with GpuDateUnaryExpression with Product with Serializable
- case class GpuDecimalDivide(left: Expression, right: Expression, dataType: DecimalType, failOnError: Boolean = SQLConf.get.ansiEnabled) extends Expression with ShimExpression with GpuDecimalDivideBase with Product with Serializable
-
trait
GpuDecimalDivideBase extends Expression with GpuExpression
A version of Divide specifically for DecimalType that does not force the left and right to be the same type.
A version of Divide specifically for DecimalType that does not force the left and right to be the same type. This lets us calculate the correct result on a wider range of values without the need for unbounded precision in the processing.
- case class GpuDecimalMultiply(left: Expression, right: Expression, dataType: DecimalType, useLongMultiply: Boolean = false, failOnError: Boolean = SQLConf.get.ansiEnabled) extends Expression with ShimExpression with GpuDecimalMultiplyBase with Product with Serializable
- trait GpuDecimalMultiplyBase extends Expression with GpuExpression
- trait GpuDivModLike extends CudfBinaryArithmetic
- case class GpuDivide(left: Expression, right: Expression, failOnError: Boolean = SQLConf.get.ansiEnabled) extends CudfBinaryArithmetic with GpuDivModLike with Product with Serializable
-
class
GpuDynamicPartitionDataConcurrentWriter extends GpuDynamicPartitionDataSingleWriter with Logging
Dynamic partition writer with concurrent writers, meaning multiple concurrent writers are opened for writing.
Dynamic partition writer with concurrent writers, meaning multiple concurrent writers are opened for writing.
The process has the following steps:
- Step 1: Maintain a map of output writers per each partition columns. Keep all writers opened; Cache the inputted batches by splitting them into sub-groups and each partition holds a list of spillable sub-groups; Find and write the max pending partition data if the total caches exceed the limitation.
- Step 2: If number of concurrent writers exceeds limit, fall back to sort-based write
(
GpuDynamicPartitionDataSingleWriter), sort rest of batches on partition. Write batch by batch, and eagerly close the writer when finishing Caller is expected to callwriteWithIterator()instead ofwrite()to write records. Note: when fall back toGpuDynamicPartitionDataSingleWriter, the single writer should restore un-closed writers and should handle un-flushed spillable caches.
-
class
GpuDynamicPartitionDataSingleWriter extends GpuFileFormatDataWriter
Dynamic partition writer with single writer, meaning only one writer is opened at any time for writing, meaning this single function can write to multiple directories (partitions) or files (bucketing).
Dynamic partition writer with single writer, meaning only one writer is opened at any time for writing, meaning this single function can write to multiple directories (partitions) or files (bucketing). The data to be written are required to be sorted on partition and/or bucket column(s) before writing.
- case class GpuElementAt(left: Expression, right: Expression, failOnError: Boolean) extends BinaryExpression with GpuBinaryExpression with ExpectsInputTypes with Product with Serializable
-
class
GpuEmptyDirectoryDataWriter extends GpuFileFormatDataWriter
GPU data writer for empty partitions
- case class GpuEndsWith(left: Expression, right: Expression) extends BinaryExpression with GpuBinaryExpressionArgsAnyScalar with Predicate with ImplicitCastInputTypes with NullIntolerant with Product with Serializable
- case class GpuEqualNullSafe(left: Expression, right: Expression) extends CudfBinaryComparison with NullIntolerant with Product with Serializable
-
case class
GpuEqualTo(left: Expression, right: Expression) extends CudfBinaryComparison with NullIntolerant with Product with Serializable
The table below shows how the result is calculated for Equal-to.
The table below shows how the result is calculated for Equal-to. To make calculation easier we are leveraging the fact that the cudf-result(r) always returns false. So that result is used in place of false when needed.
Return (lhs.nan && rhs.nan) || result[i]
+-------------+------------+------------------+---------------+----+ | lhs.isNan()| rhs.isNan | cudf-result(r) | final-result | eq | +-------------+------------+------------------+---------------+----+ | t | f | f | r | f | | f | t | f | r | f | | t | t | f | t | t | | f | f | r | r | na | +-------------+------------+------------------+---------------+----+
-
case class
GpuEqualToNoNans(left: Expression, right: Expression) extends CudfBinaryComparison with NullIntolerant with Product with Serializable
This implementation leverages the default implementation of equal-to on the GPU to perform the binary equals comparison.
This implementation leverages the default implementation of equal-to on the GPU to perform the binary equals comparison. This is used for operations like PivotFirst, where NaN != NaN (unlike most other cases) when pivoting on a float or double column.
- case class GpuExp(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
- case class GpuExpm1(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
-
abstract
class
GpuFileFormatDataWriter extends DataWriter[ColumnarBatch]
Abstract class for writing out data in a single Spark task using the GPU.
Abstract class for writing out data in a single Spark task using the GPU. This is the GPU version of
org.apache.spark.sql.execution.datasources.FileFormatDataWriter. -
case class
GpuFileSourceScanExec(relation: HadoopFsRelation, originalOutput: Seq[Attribute], requiredSchema: StructType, partitionFilters: Seq[Expression], optionalBucketSet: Option[BitSet], optionalNumCoalescedBuckets: Option[Int], dataFilters: Seq[Expression], tableIdentifier: Option[TableIdentifier], disableBucketedScan: Boolean = false, queryUsesInputFile: Boolean = false, alluxioPathsMap: Option[Map[String, String]], requiredPartitionSchema: Option[StructType] = None)(rapidsConf: RapidsConf) extends SparkPlan with GpuDataSourceScanExec with GpuExec with Product with Serializable
GPU version of Spark's
FileSourceScanExecGPU version of Spark's
FileSourceScanExec- relation
The file-based relation to scan.
- originalOutput
Output attributes of the scan, including data attributes and partition attributes.
- requiredSchema
Required schema of the underlying relation, excluding partition columns.
- partitionFilters
Predicates to use for partition pruning.
- optionalBucketSet
Bucket ids for bucket pruning.
- optionalNumCoalescedBuckets
Number of coalesced buckets.
- dataFilters
Filters on non-partition columns.
- tableIdentifier
identifier for the table in the metastore.
- disableBucketedScan
Disable bucketed scan based on physical query plan.
- queryUsesInputFile
This is a parameter to easily allow turning it off in GpuTransitionOverrides if InputFileName, InputFileBlockStart, or InputFileBlockLength are used
- alluxioPathsMap
Map containing mapping of DFS scheme to Alluxio scheme
- rapidsConf
Rapids conf
- case class GpuFlattenArray(child: Expression) extends GpuUnaryExpression with NullIntolerant with Product with Serializable
-
case class
GpuFloatArrayMax(child: Expression) extends GpuArrayMax with Product with Serializable
ArrayMax for FloatType and DoubleType to handle
Nans.ArrayMax for FloatType and DoubleType to handle
Nans.In Spark,
Nanis the max float value, however in cuDF, the calculation involvingNanis undefined. We design a workaround method here to match the Spark's behaviour. The high level idea is that, we firstly check if each list containsNan. If it is, the max value isNan, else we use the cuDF kernel to calculate the max value. -
case class
GpuFloatArrayMin(child: Expression) extends GpuArrayMin with Product with Serializable
ArrayMin for FloatType and DoubleType to handle
Nans.ArrayMin for FloatType and DoubleType to handle
Nans.In Spark,
Nanis the max float value, however in cuDF, the calculation involvingNanis undefined. We design a workaround method here to match the Spark's behaviour. The high level idea is: if one list contains onlyNans ornulls then if the list containsNanthen returnNanelse return null else replace allNans with nulls; use cuDF kernel to find the min value - case class GpuFloor(child: Expression, outputType: DataType) extends CudfUnaryMathExpression with Product with Serializable
- case class GpuFormatNumber(x: Expression, d: Expression) extends BinaryExpression with GpuBinaryExpression with ExpectsInputTypes with NullIntolerant with Product with Serializable
- case class GpuFromUTCTimestamp(timestamp: Expression, timezone: Expression, zoneId: ZoneId) extends BinaryExpression with GpuBinaryExpressionArgsAnyScalar with ImplicitCastInputTypes with NullIntolerant with Product with Serializable
- case class GpuFromUnixTime(sec: Expression, format: Expression, strfFormat: String, colConverter: Option[(ColumnView) ⇒ ColumnVector], timeZoneId: Option[String]) extends BinaryExpression with GpuBinaryExpressionArgsAnyScalar with TimeZoneAwareExpression with ImplicitCastInputTypes with Product with Serializable
-
case class
GpuGetArrayItem(child: Expression, ordinal: Expression, failOnError: Boolean) extends BinaryExpression with GpuBinaryExpression with ExpectsInputTypes with ShimGetArrayItem with Product with Serializable
Returns the field at
ordinalin the Arraychild.Returns the field at
ordinalin the Arraychild.We need to do type checking here as
ordinalexpression maybe unresolved. -
case class
GpuGetArrayStructFields(child: Expression, field: StructField, ordinal: Int, numFields: Int, containsNull: Boolean) extends GpuUnaryExpression with ShimGetArrayStructFields with NullIntolerant with Product with Serializable
For a child whose data type is an array of structs, extracts the
ordinal-th fields of all array elements, and returns them as a new array.For a child whose data type is an array of structs, extracts the
ordinal-th fields of all array elements, and returns them as a new array.No need to do type checking since it is handled by 'ExtractValue'.
- class GpuGetArrayStructFieldsMeta extends UnaryExprMeta[GetArrayStructFields]
- case class GpuGetMapValue(child: Expression, key: Expression, failOnError: Boolean) extends BinaryExpression with GpuBinaryExpression with ImplicitCastInputTypes with NullIntolerant with Product with Serializable
- case class GpuGetStructField(child: Expression, ordinal: Int, name: Option[String] = None) extends UnaryExpression with ShimUnaryExpression with GpuExpression with ShimGetStructField with NullIntolerant with Product with Serializable
- case class GpuGetTimestamp(strTs: Expression, format: Expression, sparkFormat: String, strf: String, timeZoneId: Option[String] = None) extends GpuToTimestamp with Product with Serializable
-
case class
GpuGreaterThan(left: Expression, right: Expression) extends CudfBinaryComparison with NullIntolerant with Product with Serializable
The table below shows how the result is calculated for greater-than.
The table below shows how the result is calculated for greater-than. To make calculation easier we are leveraging the fact that the cudf-result(r) always returns false. So that result is used in place of false when needed.
In this case return (lhs.nan && !lhs.nan) || result[i]
+-------------+------------+-----------------+---------------+----+ | lhs.isNan()| rhs.isNan | cudf-result(r) | final-result | gt | +-------------+------------+-----------------+---------------+----+ | t | f | f | t | t | | f | t | f | r | f | | t | t | f | r | f | | f | f | r | r | na | +-------------+------------+-----------------+---------------+----+
-
case class
GpuGreaterThanOrEqual(left: Expression, right: Expression) extends CudfBinaryComparison with NullIntolerant with Product with Serializable
The table below shows how the result is calculated for Greater-than-Eq.
The table below shows how the result is calculated for Greater-than-Eq. To make calculation easier we are leveraging the fact that the cudf-result(r) always returns false. So that result is used in place of false when needed.
In this case return lhs.isNan || result[i]
+-------------+------------+-----------------+---------------+-----+ | lhs.isNan()| rhs.isNan | cudf-result(r) | final-result | gte | +-------------+------------+-----------------+---------------+-----+ | t | f | f | t | t | | f | t | f | r | f | | t | t | f | t | t | | f | f | r | r | NA | +-------------+------------+-----------------+---------------+-----+
- case class GpuGreatest(children: Seq[Expression]) extends Expression with GpuGreatestLeastBase with Product with Serializable
- trait GpuGreatestLeastBase extends Expression with ComplexTypeMergingExpression with GpuExpression with ShimExpression
- abstract class GpuHashExpression extends Expression with GpuExpression with ShimExpression
- case class GpuHiveHash(children: Seq[Expression]) extends GpuHashExpression with Product with Serializable
- case class GpuHour(child: Expression, timeZoneId: Option[String] = None) extends GpuUnaryExpression with GpuTimeUnaryExpression with Product with Serializable
- case class GpuHypot(left: Expression, right: Expression) extends CudfBinaryMathExpression with Product with Serializable
- case class GpuInMemoryTableScanExec(attributes: Seq[Attribute], predicates: Seq[Expression], relation: InMemoryRelation) extends SparkPlan with ShimLeafExecNode with GpuExec with Product with Serializable
- case class GpuInitCap(child: Expression) extends GpuUnaryExpression with ImplicitCastInputTypes with Product with Serializable
-
case class
GpuInputFileBlockLength() extends GpuLeafExpression with Product with Serializable
Returns the length of the block being read, or -1 if not available.
Returns the length of the block being read, or -1 if not available. This is extra difficult because we cannot coalesce batches in between when this is used and the input file or else we could run into problems with returning the wrong thing.
-
case class
GpuInputFileBlockStart() extends GpuLeafExpression with Product with Serializable
Returns the start offset of the block being read, or -1 if not available.
Returns the start offset of the block being read, or -1 if not available. This is extra difficult because we cannot coalesce batches in between when this is used and the input file or else we could run into problems with returning the wrong thing.
-
case class
GpuInputFileName() extends GpuLeafExpression with Product with Serializable
Returns the name of the file being read, or empty string if not available.
Returns the name of the file being read, or empty string if not available. This is extra difficult because we cannot coalesce batches in between when this is used and the input file or else we could run into problems with returning the wrong thing.
- case class GpuInsertIntoHadoopFsRelationCommand(outputPath: Path, staticPartitions: TablePartitionSpec, ifPartitionNotExists: Boolean, partitionColumns: Seq[Attribute], bucketSpec: Option[BucketSpec], fileFormat: ColumnarFileFormat, options: Map[String, String], query: LogicalPlan, mode: SaveMode, catalogTable: Option[CatalogTable], fileIndex: Option[FileIndex], outputColumnNames: Seq[String], useStableSort: Boolean, concurrentWriterPartitionFlushSize: Long) extends LogicalPlan with GpuDataWritingCommand with Product with Serializable
- case class GpuIntegralDivide(left: Expression, right: Expression, failOnError: Boolean = SQLConf.get.ansiEnabled) extends GpuIntegralDivideParent with Product with Serializable
- abstract class GpuIntegralDivideParent extends CudfBinaryArithmetic with GpuDivModLike with Serializable
- case class GpuJsonToStructs(schema: DataType, options: Map[String, String], child: Expression, timeZoneId: Option[String] = None) extends GpuUnaryExpression with TimeZoneAwareExpression with ExpectsInputTypes with NullIntolerant with Product with Serializable
- case class GpuLastDay(startDate: Expression) extends GpuUnaryExpression with ImplicitCastInputTypes with Product with Serializable
- case class GpuLeast(children: Seq[Expression]) extends Expression with GpuGreatestLeastBase with Product with Serializable
- case class GpuLength(child: Expression) extends GpuUnaryExpression with ExpectsInputTypes with Product with Serializable
-
case class
GpuLessThan(left: Expression, right: Expression) extends CudfBinaryComparison with NullIntolerant with Product with Serializable
The table below shows how the result is calculated for Less-than.
The table below shows how the result is calculated for Less-than. To make calculation easier we are leveraging the fact that the cudf-result(r) always returns false. So that result is used in place of false when needed.
In this case return !lhs.nan && rhs.nan || result[i]
+-------------+------------+-----------------+---------------+-----+ | lhs.isNan()| rhs.isNan | cudf-result(r) | final-result | lt | +-------------+------------+-----------------+---------------+-----+ | t | f | f | r | f | | f | t | f | t | t | | t | t | f | r | f | | f | f | r | r | NA | +-------------+------------+-----------------+---------------+-----+
-
case class
GpuLessThanOrEqual(left: Expression, right: Expression) extends CudfBinaryComparison with NullIntolerant with Product with Serializable
The table below shows how the result is calculated for Less-than-Eq.
The table below shows how the result is calculated for Less-than-Eq. To make calculation easier we are leveraging the fact that the cudf-result(r) always returns false. So that result is used in place of false when needed.
In this case, return rhs.nan || result[i]
+-------------+------------+------------------+---------------+-----+ | lhs.isNan()| rhs.isNan | cudf-result(r) | final-result | lte | +-------------+------------+------------------+---------------+-----+ | t | f | f | r | f | | f | t | f | t | t | | t | t | f | t | t | | f | f | r | r | NA | +-------------+------------+------------------+---------------+-----+
- case class GpuLike(left: Expression, right: Expression, escapeChar: Char) extends BinaryExpression with GpuBinaryExpressionArgsAnyScalar with ImplicitCastInputTypes with NullIntolerant with Product with Serializable
- case class GpuLiteralRangePattern(left: Expression, right: Expression, length: Int, start: Int, end: Int) extends BinaryExpression with GpuBinaryExpressionArgsAnyScalar with ImplicitCastInputTypes with NullIntolerant with Product with Serializable
- case class GpuLog(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
- case class GpuLogarithm(left: Expression, right: Expression) extends CudfBinaryMathExpression with Product with Serializable
- case class GpuLower(child: Expression) extends GpuUnaryString2StringExpression with Product with Serializable
- case class GpuMapConcat(children: Seq[Expression]) extends Expression with GpuComplexTypeMergingExpression with Product with Serializable
- case class GpuMapEntries(child: Expression) extends GpuUnaryExpression with ExpectsInputTypes with Product with Serializable
- case class GpuMapFromArrays(left: Expression, right: Expression) extends BinaryExpression with GpuBinaryExpression with Product with Serializable
- case class GpuMapKeys(child: Expression) extends GpuUnaryExpression with ExpectsInputTypes with Product with Serializable
- case class GpuMapValues(child: Expression) extends GpuUnaryExpression with ExpectsInputTypes with Product with Serializable
- case class GpuMd5(child: Expression) extends GpuUnaryExpression with ImplicitCastInputTypes with NullIntolerant with Product with Serializable
- case class GpuMicrosToTimestamp(child: Expression) extends GpuUnaryExpression with GpuNumberToTimestampUnaryExpression with Product with Serializable
- case class GpuMillisToTimestamp(child: Expression) extends GpuUnaryExpression with GpuNumberToTimestampUnaryExpression with Product with Serializable
- case class GpuMinute(child: Expression, timeZoneId: Option[String] = None) extends GpuUnaryExpression with GpuTimeUnaryExpression with Product with Serializable
- case class GpuMonth(child: Expression) extends GpuUnaryExpression with GpuDateUnaryExpression with Product with Serializable
-
class
GpuMultiFileAvroPartitionReader extends MultiFileCoalescingPartitionReaderBase with GpuAvroReaderBase
A PartitionReader that can read multiple AVRO files up to the certain size.
A PartitionReader that can read multiple AVRO files up to the certain size. It will coalesce small files together and copy the block data in a separate thread pool to speed up processing the small files before sending down to the GPU.
-
class
GpuMultiFileCloudAvroPartitionReader extends MultiFileCloudPartitionReaderBase with MultiFileReaderFunctions with GpuAvroReaderBase
A PartitionReader that can read multiple AVRO files in parallel.
A PartitionReader that can read multiple AVRO files in parallel. This is most efficient running in a cloud environment where the I/O of reading is slow.
When reading a file, it
- seeks to the start position of the first block located in this partition.
- next, parses the meta and sync, rewrites the meta and sync, and copies the data to a batch buffer per block, until reaching the last one of the current partition.
- sends batches to GPU at last.
- case class GpuMultipleContains(input: Expression, searchList: Seq[String]) extends GpuUnaryExpression with ImplicitCastInputTypes with NullIntolerant with Product with Serializable
- case class GpuMultiply(left: Expression, right: Expression, failOnError: Boolean = SQLConf.get.ansiEnabled) extends CudfBinaryArithmetic with Product with Serializable
- case class GpuMurmur3Hash(children: Seq[Expression], seed: Int) extends GpuHashExpression with Product with Serializable
- case class GpuNormalizeNaNAndZero(child: Expression) extends GpuUnaryExpression with ExpectsInputTypes with Product with Serializable
- case class GpuNot(child: Expression) extends GpuUnaryExpression with CudfUnaryExpression with Predicate with ImplicitCastInputTypes with NullIntolerant with Product with Serializable
- trait GpuNumberToTimestampUnaryExpression extends GpuUnaryExpression
- case class GpuOctetLength(child: Expression) extends GpuUnaryExpression with ExpectsInputTypes with Product with Serializable
- case class GpuOr(left: Expression, right: Expression) extends CudfBinaryPredicateWithSideEffect with Product with Serializable
- class GpuOrcFileFormat extends ColumnarFileFormat with Logging
- class GpuOrcWriter extends ColumnarOutputWriter
- case class GpuParseUrl(children: Seq[Expression]) extends Expression with GpuExpression with ShimExpression with ExpectsInputTypes with Product with Serializable
- class GpuPartitionwiseSampledRDD extends PartitionwiseSampledRDD[ColumnarBatch, ColumnarBatch]
- case class GpuPmod(left: Expression, right: Expression, failOnError: Boolean = SQLConf.get.ansiEnabled) extends GpuPmodBase with Product with Serializable
- abstract class GpuPmodBase extends CudfBinaryArithmetic with GpuDivModLike with Serializable
- class GpuPoissonSampler extends PoissonSampler[ColumnarBatch]
- case class GpuPow(left: Expression, right: Expression) extends CudfBinaryMathExpression with Product with Serializable
-
case class
GpuPreciseTimestampConversion(child: Expression, fromType: DataType, toType: DataType) extends GpuUnaryExpression with ExpectsInputTypes with Product with Serializable
Expression used internally to convert the TimestampType to Long and back without losing precision, i.e.
Expression used internally to convert the TimestampType to Long and back without losing precision, i.e. in microseconds. Used in time windowing.
- case class GpuQuarter(child: Expression) extends GpuUnaryExpression with GpuDateUnaryExpression with Product with Serializable
- case class GpuRLike(left: Expression, right: Expression, pattern: String) extends BinaryExpression with GpuBinaryExpressionArgsAnyScalar with ImplicitCastInputTypes with NullIntolerant with Product with Serializable
- class GpuRLikeMeta extends BinaryExprMeta[RLike]
-
class
GpuReadAvroFileFormat extends AvroFileFormat with GpuReadFileFormatWithMetrics
A FileFormat that allows reading Avro files with the GPU.
- case class GpuRegExpExtract(subject: Expression, regexp: Expression, idx: Expression)(cudfRegexPattern: String) extends GpuRegExpTernaryBase with ImplicitCastInputTypes with NullIntolerant with Product with Serializable
- case class GpuRegExpExtractAll(str: Expression, regexp: Expression, idx: Expression)(cudfRegexPattern: String) extends GpuRegExpTernaryBase with ImplicitCastInputTypes with NullIntolerant with Product with Serializable
- class GpuRegExpExtractAllMeta extends TernaryExprMeta[RegExpExtractAll]
- class GpuRegExpExtractMeta extends TernaryExprMeta[RegExpExtract]
- case class GpuRegExpReplace(srcExpr: Expression, searchExpr: Expression, replaceExpr: Expression)(javaRegexpPattern: String, cudfRegexPattern: String, cudfReplacementString: String, searchList: Option[Seq[String]], replaceOpt: Option[GpuRegExpReplaceOpt]) extends GpuRegExpTernaryBase with ImplicitCastInputTypes with HasGpuStringReplace with Product with Serializable
- case class GpuRegExpReplaceWithBackref(child: Expression, searchExpr: Expression, replaceExpr: Expression)(javaRegexpPattern: String, cudfRegexPattern: String, cudfReplacementString: String) extends GpuUnaryExpression with ImplicitCastInputTypes with Product with Serializable
- abstract class GpuRegExpTernaryBase extends TernaryExpression with GpuTernaryExpressionArgsAnyScalarScalar
- case class GpuRemainder(left: Expression, right: Expression, failOnError: Boolean = SQLConf.get.ansiEnabled) extends GpuRemainderBase with Product with Serializable
- abstract class GpuRemainderBase extends CudfBinaryArithmetic with GpuDivModLike with Serializable
- case class GpuReverse(child: Expression) extends GpuUnaryExpression with Product with Serializable
- case class GpuRint(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
- case class GpuRound(child: Expression, scale: Expression, outputType: DataType) extends GpuRoundBase with Product with Serializable
- abstract class GpuRoundBase extends BinaryExpression with GpuBinaryExpressionArgsAnyScalar with Serializable with ImplicitCastInputTypes
- case class GpuRowBasedScalaUDF(sparkFunc: AnyRef, dataType: DataType, children: Seq[Expression], inputEncoders: Seq[Option[ExpressionEncoder[_]]], outputEncoder: Option[ExpressionEncoder[_]], udfName: Option[String], nullable: Boolean, udfDeterministic: Boolean) extends Expression with GpuRowBasedUserDefinedFunction with Product with Serializable
- case class GpuScalaUDF(function: RapidsUDF, dataType: DataType, children: Seq[Expression], udfName: Option[String], nullable: Boolean, udfDeterministic: Boolean) extends Expression with GpuUserDefinedFunction with Product with Serializable
-
case class
GpuScalarSubquery(plan: BaseSubqueryExec, exprId: ExprId) extends ExecSubqueryExpression with GpuExpression with ShimExpression with Product with Serializable
GPU placeholder of ScalarSubquery, which returns the scalar result with columnarEval method.
GPU placeholder of ScalarSubquery, which returns the scalar result with columnarEval method. This placeholder is to make ScalarSubquery working as a GPUExpression to cooperate other GPU overrides.
- case class GpuSecond(child: Expression, timeZoneId: Option[String] = None) extends GpuUnaryExpression with GpuTimeUnaryExpression with Product with Serializable
- case class GpuSecondsToTimestamp(child: Expression) extends GpuUnaryExpression with GpuNumberToTimestampUnaryExpression with Product with Serializable
- case class GpuSequence(start: Expression, stop: Expression, stepOpt: Option[Expression], timeZoneId: Option[String] = None) extends Expression with TimeZoneAwareExpression with GpuExpression with ShimExpression with Product with Serializable
- class GpuSequenceMeta extends ExprMeta[Sequence]
-
class
GpuSerializableBatch extends Serializable with AutoCloseable
- Annotations
- @SerialVersionUID()
- trait GpuShiftBase extends BinaryExpression with GpuBinaryExpression with ImplicitCastInputTypes
- case class GpuShiftLeft(left: Expression, right: Expression) extends BinaryExpression with GpuShiftBase with Product with Serializable
- case class GpuShiftRight(left: Expression, right: Expression) extends BinaryExpression with GpuShiftBase with Product with Serializable
- case class GpuShiftRightUnsigned(left: Expression, right: Expression) extends BinaryExpression with GpuShiftBase with Product with Serializable
- abstract class GpuShuffleBlockResolverBase extends ShuffleBlockResolver with Logging
- class GpuShuffleDependency[K, V, C] extends ShuffleDependency[K, V, C]
- class GpuShuffleEnv extends Logging
- class GpuShuffleHandle[K, V] extends BaseShuffleHandle[K, V, V]
- case class GpuSignum(child: Expression) extends GpuUnaryMathExpression with Product with Serializable
- case class GpuSin(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
-
class
GpuSingleDirectoryDataWriter extends GpuFileFormatDataWriter
Writes data to a single directory (used for non-dynamic-partition writes).
- case class GpuSinh(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
- case class GpuSize(child: Expression, legacySizeOfNull: Boolean) extends GpuUnaryExpression with Product with Serializable
- case class GpuSortArray(base: Expression, ascendingOrder: Expression) extends BinaryExpression with GpuBinaryExpressionArgsAnyScalar with ExpectsInputTypes with Product with Serializable
- case class GpuSqrt(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
- case class GpuStartsWith(left: Expression, right: Expression) extends BinaryExpression with GpuBinaryExpressionArgsAnyScalar with Predicate with ImplicitCastInputTypes with NullIntolerant with Product with Serializable
- case class GpuStringInstr(str: Expression, substr: Expression) extends BinaryExpression with GpuBinaryExpressionArgsAnyScalar with ImplicitCastInputTypes with NullIntolerant with Product with Serializable
- case class GpuStringLPad(str: Expression, len: Expression, pad: Expression) extends TernaryExpression with BasePad with Product with Serializable
- case class GpuStringLocate(substr: Expression, col: Expression, start: Expression) extends TernaryExpression with GpuTernaryExpressionArgsScalarAnyScalar with ImplicitCastInputTypes with Product with Serializable
- case class GpuStringRPad(str: Expression, len: Expression, pad: Expression) extends TernaryExpression with BasePad with Product with Serializable
- case class GpuStringRepeat(input: Expression, repeatTimes: Expression) extends BinaryExpression with GpuBinaryExpression with ImplicitCastInputTypes with NullIntolerant with Product with Serializable
- case class GpuStringReplace(srcExpr: Expression, searchExpr: Expression, replaceExpr: Expression) extends TernaryExpression with GpuTernaryExpressionArgsAnyScalarScalar with ImplicitCastInputTypes with HasGpuStringReplace with Product with Serializable
- case class GpuStringSplit(str: Expression, regex: Expression, limit: Expression, pattern: String, isRegExp: Boolean) extends TernaryExpression with GpuTernaryExpression with ImplicitCastInputTypes with Product with Serializable
- class GpuStringSplitMeta extends StringSplitRegExpMeta[StringSplit]
- case class GpuStringToMap(strExpr: Expression, pairDelimExpr: Expression, keyValueDelimExpr: Expression, pairDelim: String, isPairDelimRegExp: Boolean, keyValueDelim: String, isKeyValueDelimRegExp: Boolean) extends Expression with GpuExpression with ShimExpression with ExpectsInputTypes with Product with Serializable
- class GpuStringToMapMeta extends StringSplitRegExpMeta[StringToMap]
- case class GpuStringTranslate(srcExpr: Expression, fromExpr: Expression, toExpr: Expression) extends TernaryExpression with GpuTernaryExpressionArgsAnyScalarScalar with ImplicitCastInputTypes with Product with Serializable
- case class GpuStringTrim(column: Expression, trimParameters: Option[Expression] = None) extends Expression with GpuString2TrimExpression with ImplicitCastInputTypes with Product with Serializable
- case class GpuStringTrimLeft(column: Expression, trimParameters: Option[Expression] = None) extends Expression with GpuString2TrimExpression with ImplicitCastInputTypes with Product with Serializable
- case class GpuStringTrimRight(column: Expression, trimParameters: Option[Expression] = None) extends Expression with GpuString2TrimExpression with ImplicitCastInputTypes with Product with Serializable
- case class GpuStructsToJson(options: Map[String, String], child: Expression, timeZoneId: Option[String] = None) extends GpuUnaryExpression with Product with Serializable
- class GpuStructsToJsonMeta extends UnaryExprMeta[StructsToJson]
- case class GpuSubstring(str: Expression, pos: Expression, len: Expression) extends TernaryExpression with GpuTernaryExpression with ImplicitCastInputTypes with NullIntolerant with Product with Serializable
- case class GpuSubstringIndex(strExpr: Expression, ignoredDelimExpr: Expression, ignoredCountExpr: Expression) extends TernaryExpression with GpuTernaryExpressionArgsAnyScalarScalar with ImplicitCastInputTypes with Product with Serializable
- case class GpuSubtract(left: Expression, right: Expression, failOnError: Boolean) extends GpuSubtractBase with Product with Serializable
- abstract class GpuSubtractBase extends CudfBinaryArithmetic with Serializable
- case class GpuTan(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
- case class GpuTanh(child: Expression) extends CudfUnaryMathExpression with Product with Serializable
- class GpuTaskMetrics extends Serializable
- abstract class GpuTimeMath extends BinaryExpression with ShimBinaryExpression with GpuExpression with TimeZoneAwareExpression with ExpectsInputTypes with Serializable
- trait GpuTimeUnaryExpression extends GpuUnaryExpression with TimeZoneAwareExpression with ImplicitCastInputTypes with NullIntolerant
- case class GpuToDegrees(child: Expression) extends GpuUnaryMathExpression with Product with Serializable
- case class GpuToRadians(child: Expression) extends GpuUnaryMathExpression with Product with Serializable
-
abstract
class
GpuToTimestamp extends BinaryExpression with GpuBinaryExpressionArgsAnyScalar with TimeZoneAwareExpression with ExpectsInputTypes
A direct conversion of Spark's ToTimestamp class which converts time to UNIX timestamp by first converting to microseconds and then dividing by the downScaleFactor
- case class GpuToUTCTimestamp(timestamp: Expression, timezone: Expression, zoneId: ZoneId) extends BinaryExpression with GpuBinaryExpressionArgsAnyScalar with ImplicitCastInputTypes with NullIntolerant with Product with Serializable
- case class GpuToUnixTimestamp(strTs: Expression, format: Expression, sparkFormat: String, strf: String, timeZoneId: Option[String] = None) extends GpuToTimestamp with Product with Serializable
- abstract class GpuUnaryMathExpression extends GpuUnaryExpression with Serializable with ImplicitCastInputTypes
- case class GpuUnaryMinus(child: Expression, failOnError: Boolean) extends GpuUnaryExpression with ExpectsInputTypes with NullIntolerant with Product with Serializable
- case class GpuUnaryPositive(child: Expression) extends GpuUnaryExpression with ExpectsInputTypes with NullIntolerant with Product with Serializable
- abstract class GpuUnaryString2StringExpression extends GpuUnaryExpression with ExpectsInputTypes
- case class GpuUnixTimestamp(strTs: Expression, format: Expression, sparkFormat: String, strf: String, timeZoneId: Option[String] = None) extends GpuToTimestamp with Product with Serializable
- case class GpuUpper(child: Expression) extends GpuUnaryString2StringExpression with Product with Serializable
- case class GpuWeekDay(child: Expression) extends GpuUnaryExpression with GpuDateUnaryExpression with Product with Serializable
-
class
GpuWriteJobDescription extends Serializable
A shared job description for all the GPU write tasks.
A shared job description for all the GPU write tasks. This is the GPU version of
org.apache.spark.sql.execution.datasources.WriteJobDescription. -
class
GpuWriteJobStatsTracker extends BasicColumnarWriteJobStatsTracker
Simple ColumnarWriteJobStatsTracker implementation that's serializable, capable of instantiating GpuWriteTaskStatsTracker on executors and processing the
WriteTaskStatsthey produce by aggregating the metrics and posting them as DriverMetricUpdates. -
class
GpuWriteTaskStatsTracker extends BasicColumnarWriteTaskStatsTracker
ColumnarWriteTaskStatsTracker implementation that produces
WriteTaskStatsand tracks writing times per task. -
case class
GpuWriterBucketSpec(bucketIdExpression: GpuExpression, bucketFileNamePrefix: (Int) ⇒ String) extends Product with Serializable
Bucketing specification for all the write tasks.
Bucketing specification for all the write tasks. This is the GPU version of
org.apache.spark.sql.execution.datasources.WriterBucketSpec- bucketIdExpression
Expression to calculate bucket id based on bucket column(s).
- bucketFileNamePrefix
Prefix of output file name based on bucket id.
- case class GpuXxHash64(children: Seq[Expression], seed: Long) extends GpuHashExpression with Product with Serializable
- case class GpuYear(child: Expression) extends GpuUnaryExpression with GpuDateUnaryExpression with Product with Serializable
- trait HasGpuStringReplace extends AnyRef
- class HighWatermarkAccumulator extends AccumulatorV2[Long, Long]
- class InMemoryTableScanMeta extends SparkPlanMeta[InMemoryTableScanExec]
- class JsonDeviceDataSource extends DataSource
-
class
JsonParsingException extends RuntimeException
Exception thrown when cudf cannot parse the JSON data because some Json to Struct cases are not currently supported.
- class NanoSecondAccumulator extends AccumulatorV2[Long, NanoTime]
- case class NanoTime(value: Long) extends Product with Serializable
- case class ParseFormatMeta(separator: Option[Char], isTimestamp: Boolean, validRegex: String) extends Product with Serializable
- class RapidsCachingReader[K, C] extends ShuffleReader[K, C] with Logging
- class RapidsCachingWriter[K, V] extends ShuffleWriter[K, V] with Logging
-
class
RapidsDiskBlockManager extends AnyRef
Maps logical blocks to local disk locations.
-
class
RapidsShuffleInternalManagerBase extends ShuffleManager with RapidsShuffleHeartbeatHandler with Logging
A shuffle manager optimized for the RAPIDS Plugin For Apache Spark.
A shuffle manager optimized for the RAPIDS Plugin For Apache Spark.
- Note
This is an internal class to obtain access to the private
ShuffleManagerandSortShuffleManagerclasses. When configuring Apache Spark to use the RAPIDS shuffle manager,
- abstract class RapidsShuffleThreadedReaderBase[K, C] extends ShuffleReader[K, C] with Logging
- abstract class RapidsShuffleThreadedWriterBase[K, V] extends ShuffleWriter[K, V] with RapidsShuffleWriterShimHelper with Logging
- trait RapidsShuffleWriterShimHelper extends AnyRef
- case class RegexReplace(search: String, replace: String) extends Product with Serializable
-
class
ShimmedExecutionPlanCaptureCallbackImpl extends ExecutionPlanCaptureCallbackBase
Note that the name is prefixed with "Shimmed" such that wildcard rules under unshimmed-common-from-spark320.txt don't get confused and pick this class to be un-shimmed.
- class ShuffleHandleWithMetrics[K, V, C] extends BaseShuffleHandle[K, V, C]
- trait ShuffleMetricsUpdater extends AnyRef
- abstract class StringSplitRegExpMeta[INPUT <: TernaryExpression] extends TernaryExprMeta[INPUT]
- class SubstringIndexMeta extends TernaryExprMeta[SubstringIndex]
- case class TempSpillBufferId extends RapidsBufferId with Product with Serializable
-
class
ThreadSafeShuffleWriteMetricsReporter extends ShuffleWriteMetrics
The
ShuffleWriteMetricsReporteris based on accumulators, which are not thread safe.The
ShuffleWriteMetricsReporteris based on accumulators, which are not thread safe. This class is a thin wrapper that adds synchronization, since these metrics will be written by multiple threads. - sealed trait TimeParserPolicy extends Serializable
- class ToUTCTimestampExprMeta extends ConvertUTCTimestampExprMetaBase[ToUTCTimestamp]
- abstract class UnixTimeExprMeta[A <: BinaryExpression with TimeZoneAwareExpression] extends BinaryExprMeta[A]
Value Members
- object AddOverflowChecks
- object BasicColumnarWriteJobStatsTracker extends Serializable
- object BucketIdMetaUtils
- object CorrectedTimeParserPolicy extends TimeParserPolicy
- object CudfRegexp
- object DecimalDivideChecks
- object DecimalMultiplyChecks
- object ExceptionTimeParserPolicy extends TimeParserPolicy
- object ExecutionPlanCaptureCallback extends ExecutionPlanCaptureCallbackBase
-
object
ExternalSource extends Logging
The subclass of AvroProvider imports spark-avro classes.
The subclass of AvroProvider imports spark-avro classes. This file should not imports spark-avro classes because
class not foundexception may throw if spark-avro does not exist at runtime. Details see: https://github.com/NVIDIA/spark-rapids/issues/5648 - object GpuAnsi
- object GpuArrayMax extends Serializable
- object GpuArrayMin extends Serializable
- object GpuAvroScan extends Serializable
- object GpuCreateMap extends Serializable
- object GpuDataSourceBase extends Logging
- object GpuDataSourceScanExec extends Serializable
- object GpuDivModLike
- object GpuElementAtMeta
- object GpuFileFormatDataWriter
-
object
GpuFileFormatWriter extends Logging
A helper object for writing columnar data out to a location.
- object GpuFileSourceScanExec extends Serializable
- object GpuFloorCeil
- object GpuHypot extends Serializable
-
object
GpuJsonReadCommon
This is a utility method intended to provide common functionality between JsonToStructs and ScanJson
- object GpuLogarithm extends Serializable
- object GpuMurmur3Hash extends Serializable
- object GpuOrcFileFormat extends Logging
- object GpuParseUrl extends Serializable
- object GpuReadAvroFileFormat extends Serializable
- object GpuRegExpUtils
- object GpuScalaUDF extends Serializable
- object GpuScalaUDFMeta
- object GpuSequenceUtil
- object GpuShuffleEnv extends Logging
- object GpuStringInstr extends Serializable
-
object
GpuTaskMetrics extends Logging with Serializable
Provides task level metrics
- object GpuToTimestamp
- object GpuV1WriteUtils
- object GpuWriteJobStatsTracker extends Serializable
- object InputFileUtils
- object LegacyTimeParserPolicy extends TimeParserPolicy
- object PCBSSchemaHelper
- object RapidsPrivateUtil
- object RapidsShuffleInternalManagerBase extends Logging
- object ShiftHelper
- object SubtractOverflowChecks
- object TempSpillBufferId extends Serializable
- object TimeZoneDB