trait FileDataSourceV2 extends TableProvider with DataSourceRegister
A base interface for data source v2 implementations of the built-in file-based data sources.
Linear Supertypes
Known Subclasses
Ordering
- Alphabetic
- By Inheritance
Inherited
- FileDataSourceV2
- DataSourceRegister
- TableProvider
- AnyRef
- Any
- Hide All
- Show All
Visibility
- Public
- Protected
Abstract Value Members
- abstract def fallbackFileFormat: Class[_ <: FileFormat]
Returns a V1 FileFormat class of the same file data source.
Returns a V1 FileFormat class of the same file data source. This is a solution for the following cases: 1. File datasource V2 implementations cause regression. Users can disable the problematic data source via SQL configuration and fall back to FileFormat. 2. Catalog support is required, which is still under development for data source V2.
- abstract def getTable(options: CaseInsensitiveStringMap): Table
- Attributes
- protected
- abstract def shortName(): String
The string that represents the format that this data source provider uses.
The string that represents the format that this data source provider uses. This is overridden by children to provide a nice alias for the data source. For example:
override def shortName(): String = "parquet"
- Definition Classes
- DataSourceRegister
- Since
1.5.0
Concrete Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @native()
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable])
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def getOptionsWithoutPaths(map: CaseInsensitiveStringMap): CaseInsensitiveStringMap
- Attributes
- protected
- def getPaths(map: CaseInsensitiveStringMap): Seq[String]
- Attributes
- protected
- def getTable(schema: StructType, partitioning: Array[Transform], properties: Map[String, String]): Table
- Definition Classes
- FileDataSourceV2 → TableProvider
- def getTable(options: CaseInsensitiveStringMap, schema: StructType): Table
- Attributes
- protected
- def getTableName(map: CaseInsensitiveStringMap, paths: Seq[String]): String
- Attributes
- protected
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def inferPartitioning(options: CaseInsensitiveStringMap): Array[Transform]
- Definition Classes
- FileDataSourceV2 → TableProvider
- def inferSchema(options: CaseInsensitiveStringMap): StructType
- Definition Classes
- FileDataSourceV2 → TableProvider
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- lazy val sparkSession: SparkSession
- def supportsExternalMetadata(): Boolean
- Definition Classes
- FileDataSourceV2 → TableProvider
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()