abstract class TahoeFileIndex extends FileIndex with SupportsRowIndexFilters with SnapshotDescriptor
A FileIndex that generates the list of files managed by the Tahoe protocol.
- Alphabetic
- By Inheritance
- TahoeFileIndex
- SnapshotDescriptor
- SupportsRowIndexFilters
- FileIndex
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Abstract Value Members
- abstract def inputFiles: Array[String]
- Definition Classes
- FileIndex
- abstract def matchingFiles(partitionFilters: Seq[Expression], dataFilters: Seq[Expression]): Seq[AddFile]
Returns all matching/valid files by the given
partitionFiltersanddataFilters.Returns all matching/valid files by the given
partitionFiltersanddataFilters. Implementations may avoid evaluating data filters when doing so would be expensive, but *must* evaluate the partition filters; wrong results will be produced if AddFile entries which don't match the partition filters are returned. - abstract def metadata: Metadata
- Definition Classes
- SnapshotDescriptor
- abstract def numOfFilesIfKnown: Option[Long]
- Attributes
- protected[delta]
- Definition Classes
- SnapshotDescriptor
- abstract def protocol: Protocol
- Definition Classes
- SnapshotDescriptor
- abstract def refresh(): Unit
- Definition Classes
- FileIndex
- abstract def sizeInBytes: Long
- Definition Classes
- FileIndex
- abstract def sizeInBytesIfKnown: Option[Long]
- Attributes
- protected[delta]
- Definition Classes
- SnapshotDescriptor
- abstract def version: Long
- Definition Classes
- SnapshotDescriptor
Concrete Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- def absolutePath(child: String): Path
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @native()
- val deltaLog: DeltaLog
- Definition Classes
- TahoeFileIndex → SnapshotDescriptor
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- def fileStatusWithMetadataFromAddFile(addFile: AddFile): FileStatusWithMetadata
Generates a FileStatusWithMetadata using data extracted from a given AddFile.
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable])
- def getBasePath(filePath: Path): Option[Path]
Returns the path of the base directory of the given file path (i.e.
Returns the path of the base directory of the given file path (i.e. its parent directory with all the partition directories stripped off).
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def getPartitionValuesRow(partitionValues: Map[String, String]): GenericInternalRow
- Attributes
- protected
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- def listFiles(partitionFilters: Seq[Expression], dataFilters: Seq[Expression]): Seq[PartitionDirectory]
- Definition Classes
- TahoeFileIndex → FileIndex
- def listPartitionsAsAddFiles(partitionFilters: Seq[Expression], dataFilters: Seq[Expression]): (Seq[(InternalRow, Seq[AddFile])], Seq[AddFile])
Returns (i) tuples of partition directories to their respective AddFile actions and (ii) a collection of matched AddFiles.
Returns (i) tuples of partition directories to their respective AddFile actions and (ii) a collection of matched AddFiles. The matched AddFiles are those that meet the criteria set by the partition and data filters. Essentially, this is a collection of all the files associated with the identified partitions.
- def makePartitionDirectories(partitionValuesToFiles: Seq[(InternalRow, Seq[AddFile])]): Seq[PartitionDirectory]
- def metadataOpsTimeNs: Option[Long]
- Definition Classes
- FileIndex
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- def partitionSchema: StructType
- Definition Classes
- TahoeFileIndex → FileIndex
- val path: Path
- def rootPaths: Seq[Path]
- Definition Classes
- TahoeFileIndex → FileIndex
- def rowIndexFilters: Option[Map[String, RowIndexFilterType]]
If we know a-priori which exact rows we want to read (e.g., from a previous scan) find the per-file filter here, which must be passed down to the appropriate reader.
If we know a-priori which exact rows we want to read (e.g., from a previous scan) find the per-file filter here, which must be passed down to the appropriate reader.
- returns
a mapping from file names to the row index filter for that file.
- Definition Classes
- SupportsRowIndexFilters
- def schema: StructType
- Definition Classes
- SnapshotDescriptor
- val spark: SparkSession
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- TahoeFileIndex → FileIndex → AnyRef → Any
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()