Packages

package v2

Ordering
  1. Alphabetic
Visibility
  1. Public
  2. Protected

Package Members

  1. package csv
  2. package jdbc
  3. package json
  4. package orc
  5. package parquet
  6. package text

Type Members

  1. case class AddPartitionExec(table: SupportsPartitionManagement, partSpecs: Seq[ResolvedPartitionSpec], ignoreIfExists: Boolean, refreshCache: () => Unit) extends V2CommandExec with LeafV2CommandExec with Product with Serializable

    Physical plan node for adding partitions of table.

  2. case class AlterNamespaceSetPropertiesExec(catalog: SupportsNamespaces, namespace: Seq[String], props: Map[String, String]) extends V2CommandExec with LeafV2CommandExec with Product with Serializable

    Physical plan node for setting properties of namespace.

  3. case class AlterTableExec(catalog: TableCatalog, ident: Identifier, changes: Seq[TableChange]) extends V2CommandExec with LeafV2CommandExec with Product with Serializable

    Physical plan node for altering a table.

  4. case class AppendDataExec(query: SparkPlan, refreshCache: () => Unit, write: Write) extends V2CommandExec with V2ExistingTableWriteExec with Product with Serializable

    Physical plan node for append into a v2 table.

    Physical plan node for append into a v2 table.

    Rows in the output data set are appended.

  5. case class AppendDataExecV1(table: SupportsWrite, plan: LogicalPlan, refreshCache: () => Unit, write: V1Write) extends V2CommandExec with V1FallbackWriters with Product with Serializable

    Physical plan node for append into a v2 table using V1 write interfaces.

    Physical plan node for append into a v2 table using V1 write interfaces.

    Rows in the output data set are appended.

  6. case class AtomicCreateTableAsSelectExec(catalog: StagingTableCatalog, ident: Identifier, partitioning: Seq[Transform], query: LogicalPlan, tableSpec: TableSpec, writeOptions: Map[String, String], ifNotExists: Boolean) extends V2CommandExec with V2CreateTableAsSelectBaseExec with Product with Serializable

    Physical plan node for v2 create table as select, when the catalog is determined to support staging table creation.

    Physical plan node for v2 create table as select, when the catalog is determined to support staging table creation.

    A new table will be created using the schema of the query, and rows from the query are appended. The CTAS operation is atomic. The creation of the table is staged and the commit of the write should bundle the commitment of the metadata and the table contents in a single unit. If the write fails, the table is instructed to roll back all staged changes.

  7. case class AtomicReplaceTableAsSelectExec(catalog: StagingTableCatalog, ident: Identifier, partitioning: Seq[Transform], query: LogicalPlan, tableSpec: TableSpec, writeOptions: Map[String, String], orCreate: Boolean, invalidateCache: (TableCatalog, Table, Identifier) => Unit) extends V2CommandExec with V2CreateTableAsSelectBaseExec with Product with Serializable

    Physical plan node for v2 replace table as select when the catalog supports staging table replacement.

    Physical plan node for v2 replace table as select when the catalog supports staging table replacement.

    A new table will be created using the schema of the query, and rows from the query are appended. If the table exists, its contents and schema should be replaced with the schema and the contents of the query. This implementation is atomic. The table replacement is staged, and the commit operation at the end should perform the replacement of the table's metadata and contents. If the write fails, the table is instructed to roll back staged changes and any previously written table is left untouched.

  8. case class AtomicReplaceTableExec(catalog: StagingTableCatalog, identifier: Identifier, columns: Array[connector.catalog.Column], partitioning: Seq[Transform], tableSpec: TableSpec, orCreate: Boolean, invalidateCache: (TableCatalog, Table, Identifier) => Unit) extends V2CommandExec with LeafV2CommandExec with Product with Serializable
  9. trait BaseCacheTableExec extends V2CommandExec with LeafV2CommandExec
  10. case class BatchScanExec(output: Seq[AttributeReference], scan: Scan, runtimeFilters: Seq[Expression], ordering: Option[Seq[SortOrder]] = None, table: Table, spjParams: StoragePartitionJoinParams = StoragePartitionJoinParams()) extends SparkPlan with DataSourceV2ScanExecBase with Product with Serializable

    Physical plan node for scanning a batch of data from a data source v2.

  11. case class CacheTableAsSelectExec(tempViewName: String, query: LogicalPlan, originalText: String, isLazy: Boolean, options: Map[String, String], referredTempFunctions: Seq[String]) extends V2CommandExec with BaseCacheTableExec with Product with Serializable
  12. case class CacheTableExec(relation: LogicalPlan, multipartIdentifier: Seq[String], isLazy: Boolean, options: Map[String, String]) extends V2CommandExec with BaseCacheTableExec with Product with Serializable
  13. case class ContinuousScanExec(output: Seq[Attribute], scan: Scan, stream: ContinuousStream, start: Offset, keyGroupedPartitioning: Option[Seq[Expression]] = None, ordering: Option[Seq[SortOrder]] = None) extends SparkPlan with DataSourceV2ScanExecBase with Product with Serializable

    Physical plan node for scanning data from a streaming data source with continuous mode.

  14. case class CreateIndexExec(table: SupportsIndex, indexName: String, indexType: String, ignoreIfExists: Boolean, columns: Seq[(NamedReference, Map[String, String])], properties: Map[String, String]) extends V2CommandExec with LeafV2CommandExec with Product with Serializable

    Physical plan node for creating an index.

  15. case class CreateNamespaceExec(catalog: SupportsNamespaces, namespace: Seq[String], ifNotExists: Boolean, properties: Map[String, String]) extends V2CommandExec with LeafV2CommandExec with Product with Serializable

    Physical plan node for creating a namespace.

  16. case class CreateTableAsSelectExec(catalog: TableCatalog, ident: Identifier, partitioning: Seq[Transform], query: LogicalPlan, tableSpec: TableSpec, writeOptions: Map[String, String], ifNotExists: Boolean) extends V2CommandExec with V2CreateTableAsSelectBaseExec with Product with Serializable

    Physical plan node for v2 create table as select when the catalog does not support staging the table creation.

    Physical plan node for v2 create table as select when the catalog does not support staging the table creation.

    A new table will be created using the schema of the query, and rows from the query are appended. If either table creation or the append fails, the table will be deleted. This implementation is not atomic; for an atomic variant for catalogs that support the appropriate features, see CreateTableAsSelectStagingExec.

  17. case class CreateTableExec(catalog: TableCatalog, identifier: Identifier, columns: Array[connector.catalog.Column], partitioning: Seq[Transform], tableSpec: TableSpec, ignoreIfExists: Boolean) extends V2CommandExec with LeafV2CommandExec with Product with Serializable
  18. class DataSourceRDD extends RDD[InternalRow]
  19. class DataSourceRDDPartition extends Partition with Serializable
  20. trait DataSourceV2ScanExecBase extends SparkPlan with LeafExecNode
  21. class DataSourceV2Strategy extends Strategy with PredicateHelper
  22. case class DeleteFromTableExec(table: SupportsDeleteV2, condition: Array[Predicate], refreshCache: () => Unit) extends V2CommandExec with LeafV2CommandExec with Product with Serializable
  23. case class DeltaWithMetadataWritingSparkTask(projections: WriteDeltaProjections) extends WritingSparkTask[DeltaWriter[InternalRow]] with Product with Serializable
  24. case class DeltaWritingSparkTask(projections: WriteDeltaProjections) extends WritingSparkTask[DeltaWriter[InternalRow]] with Product with Serializable
  25. case class DescribeColumnExec(output: Seq[Attribute], column: Attribute, isExtended: Boolean, table: Table) extends V2CommandExec with LeafV2CommandExec with Product with Serializable
  26. case class DescribeNamespaceExec(output: Seq[Attribute], catalog: SupportsNamespaces, namespace: Seq[String], isExtended: Boolean) extends V2CommandExec with LeafV2CommandExec with Product with Serializable

    Physical plan node for describing a namespace.

  27. case class DescribeTableExec(output: Seq[Attribute], table: Table, isExtended: Boolean) extends V2CommandExec with LeafV2CommandExec with Product with Serializable
  28. case class DropIndexExec(table: SupportsIndex, indexName: String, ignoreIfNotExists: Boolean) extends V2CommandExec with LeafV2CommandExec with Product with Serializable

    Physical plan node for dropping an index.

  29. case class DropNamespaceExec(catalog: CatalogPlugin, namespace: Seq[String], ifExists: Boolean, cascade: Boolean) extends V2CommandExec with LeafV2CommandExec with Product with Serializable

    Physical plan node for dropping a namespace.

  30. case class DropPartitionExec(table: SupportsPartitionManagement, partSpecs: Seq[ResolvedPartitionSpec], ignoreIfNotExists: Boolean, purge: Boolean, refreshCache: () => Unit) extends V2CommandExec with LeafV2CommandExec with Product with Serializable

    Physical plan node for dropping partitions of table.

  31. case class DropTableExec(catalog: TableCatalog, ident: Identifier, ifExists: Boolean, purge: Boolean, invalidateCache: () => Unit) extends V2CommandExec with LeafV2CommandExec with Product with Serializable

    Physical plan node for dropping a table.

  32. class EmptyPartitionReader[T] extends PartitionReader[T]

    A PartitionReader with empty output.

  33. class FileBatchWrite extends BatchWrite with Logging
  34. trait FileDataSourceV2 extends TableProvider with DataSourceRegister

    A base interface for data source v2 implementations of the built-in file-based data sources.

  35. class FilePartitionReader[T] extends PartitionReader[T] with Logging
  36. abstract class FilePartitionReaderFactory extends PartitionReaderFactory
  37. trait FileScan extends Scan with Batch with SupportsReportStatistics with SupportsMetadata with Logging
  38. abstract class FileScanBuilder extends ScanBuilder with SupportsPushDownRequiredColumns with SupportsPushDownCatalystFilters
  39. abstract class FileTable extends Table with SupportsRead with SupportsWrite
  40. trait FileWrite extends Write
  41. case class FileWriterFactory(description: WriteJobDescription, committer: FileCommitProtocol) extends DataWriterFactory with Product with Serializable
  42. trait LeafV2CommandExec extends V2CommandExec with LeafLike[SparkPlan]
  43. case class MergeRowsExec(isSourceRowPresent: Expression, isTargetRowPresent: Expression, matchedInstructions: Seq[Instruction], notMatchedInstructions: Seq[Instruction], notMatchedBySourceInstructions: Seq[Instruction], checkCardinality: Boolean, output: Seq[Attribute], child: SparkPlan) extends SparkPlan with UnaryExecNode with Product with Serializable
  44. case class MicroBatchScanExec(output: Seq[Attribute], scan: Scan, stream: MicroBatchStream, start: Offset, end: Offset, keyGroupedPartitioning: Option[Seq[Expression]] = None, ordering: Option[Seq[SortOrder]] = None) extends SparkPlan with DataSourceV2ScanExecBase with Product with Serializable

    Physical plan node for scanning a micro-batch of data from a data source.

  45. case class OverwriteByExpressionExec(query: SparkPlan, refreshCache: () => Unit, write: Write) extends V2CommandExec with V2ExistingTableWriteExec with Product with Serializable

    Physical plan node for overwrite into a v2 table.

    Physical plan node for overwrite into a v2 table.

    Overwrites data in a table matched by a set of filters. Rows matching all of the filters will be deleted and rows in the output data set are appended.

    This plan is used to implement SaveMode.Overwrite. The behavior of SaveMode.Overwrite is to truncate the table -- delete all rows -- and append the output data set. This uses the filter AlwaysTrue to delete all rows.

  46. case class OverwriteByExpressionExecV1(table: SupportsWrite, plan: LogicalPlan, refreshCache: () => Unit, write: V1Write) extends V2CommandExec with V1FallbackWriters with Product with Serializable

    Physical plan node for overwrite into a v2 table with V1 write interfaces.

    Physical plan node for overwrite into a v2 table with V1 write interfaces. Note that when this interface is used, the atomicity of the operation depends solely on the target data source.

    Overwrites data in a table matched by a set of filters. Rows matching all of the filters will be deleted and rows in the output data set are appended.

    This plan is used to implement SaveMode.Overwrite. The behavior of SaveMode.Overwrite is to truncate the table -- delete all rows -- and append the output data set. This uses the filter AlwaysTrue to delete all rows.

  47. case class OverwritePartitionsDynamicExec(query: SparkPlan, refreshCache: () => Unit, write: Write) extends V2CommandExec with V2ExistingTableWriteExec with Product with Serializable

    Physical plan node for dynamic partition overwrite into a v2 table.

    Physical plan node for dynamic partition overwrite into a v2 table.

    Dynamic partition overwrite is the behavior of Hive INSERT OVERWRITE ... PARTITION queries, and Spark INSERT OVERWRITE queries when spark.sql.sources.partitionOverwriteMode=dynamic. Each partition in the output data set replaces the corresponding existing partition in the table or creates a new partition. Existing partitions for which there is no data in the output data set are not modified.

  48. class PartitionReaderFromIterator[InternalRow] extends PartitionReader[InternalRow]
  49. class PartitionReaderWithPartitionValues extends PartitionReader[InternalRow]

    A wrapper reader that always appends partition values to InternalRows produced by the input reader fileReader.

  50. class PartitionRecordReader[T] extends PartitionReader[T]
  51. case class PushedDownOperators(aggregation: Option[Aggregation], sample: Option[TableSampleInfo], limit: Option[Int], offset: Option[Int], sortValues: Seq[SortOrder], pushedPredicates: Seq[Predicate]) extends Product with Serializable

    Pushed down operators

  52. case class RefreshTableExec(catalog: TableCatalog, ident: Identifier, refreshCache: () => Unit) extends V2CommandExec with LeafV2CommandExec with Product with Serializable
  53. case class RenamePartitionExec(table: SupportsPartitionManagement, from: ResolvedPartitionSpec, to: ResolvedPartitionSpec, refreshCache: () => Unit) extends V2CommandExec with LeafV2CommandExec with Product with Serializable

    Physical plan node for renaming a table partition.

  54. case class RenameTableExec(catalog: TableCatalog, oldIdent: Identifier, newIdent: Identifier, invalidateCache: () => Option[StorageLevel], cacheTable: (SparkSession, LogicalPlan, Option[String], StorageLevel) => Unit) extends V2CommandExec with LeafV2CommandExec with Product with Serializable

    Physical plan node for renaming a table.

  55. case class ReplaceDataExec(query: SparkPlan, refreshCache: () => Unit, write: Write) extends V2CommandExec with V2ExistingTableWriteExec with Product with Serializable

    Physical plan node to replace data in existing tables.

  56. case class ReplaceTableAsSelectExec(catalog: TableCatalog, ident: Identifier, partitioning: Seq[Transform], query: LogicalPlan, tableSpec: TableSpec, writeOptions: Map[String, String], orCreate: Boolean, invalidateCache: (TableCatalog, Table, Identifier) => Unit) extends V2CommandExec with V2CreateTableAsSelectBaseExec with Product with Serializable

    Physical plan node for v2 replace table as select when the catalog does not support staging table replacement.

    Physical plan node for v2 replace table as select when the catalog does not support staging table replacement.

    A new table will be created using the schema of the query, and rows from the query are appended. If the table exists, its contents and schema should be replaced with the schema and the contents of the query. This is a non-atomic implementation that drops the table and then runs non-atomic CTAS. For an atomic implementation for catalogs with the appropriate support, see ReplaceTableAsSelectStagingExec.

  57. case class ReplaceTableExec(catalog: TableCatalog, ident: Identifier, columns: Array[connector.catalog.Column], partitioning: Seq[Transform], tableSpec: TableSpec, orCreate: Boolean, invalidateCache: (TableCatalog, Table, Identifier) => Unit) extends V2CommandExec with LeafV2CommandExec with Product with Serializable
  58. case class ScanBuilderHolder(output: Seq[AttributeReference], relation: DataSourceV2Relation, builder: ScanBuilder) extends LogicalPlan with LeafNode with Product with Serializable
  59. case class SetCatalogAndNamespaceExec(catalogManager: CatalogManager, catalogName: Option[String], namespace: Option[Seq[String]]) extends V2CommandExec with LeafV2CommandExec with Product with Serializable

    Physical plan node for setting the current catalog and/or namespace.

  60. case class ShowCreateTableExec(output: Seq[Attribute], resolvedTable: ResolvedTable) extends V2CommandExec with LeafExecNode with Product with Serializable

    Physical plan node for show create table.

  61. case class ShowFunctionsExec(output: Seq[Attribute], catalog: FunctionCatalog, namespace: Seq[String], userScope: Boolean, systemScope: Boolean, pattern: Option[String]) extends V2CommandExec with LeafExecNode with Product with Serializable

    Physical plan node for showing functions.

  62. case class ShowNamespacesExec(output: Seq[Attribute], catalog: SupportsNamespaces, namespace: Seq[String], pattern: Option[String]) extends V2CommandExec with LeafExecNode with Product with Serializable

    Physical plan node for showing namespaces.

  63. case class ShowPartitionsExec(output: Seq[Attribute], catalog: TableCatalog, table: SupportsPartitionManagement, partitionSpec: Option[ResolvedPartitionSpec]) extends V2CommandExec with LeafExecNode with Product with Serializable

    Physical plan node for showing partitions.

  64. case class ShowTablePropertiesExec(output: Seq[Attribute], catalogTable: Table, tableName: String, propertyKey: Option[String]) extends V2CommandExec with LeafV2CommandExec with Product with Serializable

    Physical plan node for showing table properties.

  65. case class ShowTablesExec(output: Seq[Attribute], catalog: TableCatalog, namespace: Seq[String], pattern: Option[String]) extends V2CommandExec with LeafExecNode with Product with Serializable

    Physical plan node for showing tables.

  66. case class StoragePartitionJoinParams(keyGroupedPartitioning: Option[Seq[Expression]] = None, commonPartitionValues: Option[Seq[(InternalRow, Int)]] = None, applyPartialClustering: Boolean = false, replicatePartitions: Boolean = false) extends Product with Serializable
  67. trait SupportsV1Write extends SparkPlan

    A trait that allows Tables that use V1 Writer interfaces to append data.

  68. case class TableSampleInfo(lowerBound: Double, upperBound: Double, withReplacement: Boolean, seed: Long) extends Product with Serializable
  69. abstract class TextBasedFileScan extends FileScan
  70. case class TruncatePartitionExec(table: SupportsPartitionManagement, partSpec: ResolvedPartitionSpec, refreshCache: () => Unit) extends V2CommandExec with LeafV2CommandExec with Product with Serializable

    Physical plan node for table partition truncation.

  71. case class TruncateTableExec(table: TruncatableTable, refreshCache: () => Unit) extends V2CommandExec with LeafV2CommandExec with Product with Serializable

    Physical plan node for table truncation.

  72. case class UncacheTableExec(relation: LogicalPlan, cascade: Boolean) extends V2CommandExec with LeafV2CommandExec with Product with Serializable
  73. sealed trait V1FallbackWriters extends V2CommandExec with LeafV2CommandExec with SupportsV1Write

    Some helper interfaces that use V2 write semantics through the V1 writer interface.

  74. case class V1ScanWrapper(v1Scan: V1Scan, handledFilters: Seq[Filter], pushedDownOperators: PushedDownOperators) extends Scan with Product with Serializable
  75. abstract class V2CommandExec extends SparkPlan

    A physical operator that executes run() and saves the result to prevent multiple executions.

    A physical operator that executes run() and saves the result to prevent multiple executions. Any V2 commands that do not require triggering a spark job should extend this class.

  76. trait V2ExistingTableWriteExec extends V2CommandExec with V2TableWriteExec
  77. class V2SessionCatalog extends TableCatalog with FunctionCatalog with SupportsNamespaces with SQLConfHelper

    A TableCatalog that translates calls to the v1 SessionCatalog.

  78. trait V2TableWriteExec extends V2CommandExec with UnaryExecNode

    The base physical plan for writing data into data source v2.

  79. case class WriteDeltaExec(query: SparkPlan, refreshCache: () => Unit, projections: WriteDeltaProjections, write: DeltaWrite) extends V2CommandExec with V2ExistingTableWriteExec with Product with Serializable

    Physical plan node to write a delta of rows to an existing table.

  80. case class WriteToDataSourceV2Exec(batchWrite: BatchWrite, refreshCache: () => Unit, query: SparkPlan, writeMetrics: Seq[CustomMetric]) extends V2CommandExec with V2TableWriteExec with Product with Serializable
  81. trait WritingSparkTask[W <: DataWriter[InternalRow]] extends Logging with Serializable

Deprecated Type Members

  1. case class WriteToDataSourceV2(relation: Option[DataSourceV2Relation], batchWrite: BatchWrite, query: LogicalPlan, customMetrics: Seq[CustomMetric]) extends LogicalPlan with UnaryNode with Product with Serializable

    Deprecated logical plan for writing data into data source v2.

    Deprecated logical plan for writing data into data source v2. This is being replaced by more specific logical plans, like org.apache.spark.sql.catalyst.plans.logical.AppendData.

    Annotations
    @deprecated
    Deprecated

    (Since version 2.4.0) Use specific logical plans like AppendData instead

Value Members

  1. object DataWritingSparkTask extends WritingSparkTask[DataWriter[InternalRow]]
  2. object DistributionAndOrderingUtils
  3. object FileTable
  4. object GroupBasedRowLevelOperationScanPlanning extends Rule[LogicalPlan] with PredicateHelper

    A rule that builds scans for group-based row-level operations.

    A rule that builds scans for group-based row-level operations.

    Note this rule must be run before V2ScanRelationPushDown as scans for group-based row-level operations must be planned in a special way.

  5. object OptimizeMetadataOnlyDeleteFromTable extends Rule[LogicalPlan] with PredicateHelper

    A rule that replaces a rewritten DELETE operation with a delete using filters if the data source can handle this DELETE command without executing the plan that operates on individual or groups of rows.

    A rule that replaces a rewritten DELETE operation with a delete using filters if the data source can handle this DELETE command without executing the plan that operates on individual or groups of rows.

    Note this rule must be run after expression optimization but before scan planning.

  6. object PushDownUtils
  7. object PushablePredicate

    Get the expression of DS V2 to represent catalyst predicate that can be pushed down.

  8. object TableCapabilityCheck extends (LogicalPlan) => Unit

    Checks the capabilities of Data Source V2 tables, and fail problematic queries earlier.

  9. object V2ColumnUtils
  10. object V2ScanPartitioningAndOrdering extends Rule[LogicalPlan] with SQLConfHelper with Logging

    Extracts DataSourceV2ScanRelation from the input logical plan, converts any V2 partitioning and ordering reported by data sources to their catalyst counterparts.

    Extracts DataSourceV2ScanRelation from the input logical plan, converts any V2 partitioning and ordering reported by data sources to their catalyst counterparts. Then, annotates the plan with the partitioning and ordering result.

  11. object V2ScanRelationPushDown extends Rule[LogicalPlan] with PredicateHelper
  12. object V2Writes extends Rule[LogicalPlan] with PredicateHelper

    A rule that constructs logical writes.

Ungrouped