package storage
- Alphabetic
- Public
- All
Type Members
-
class
AzureLogStore extends HadoopFileSystemLogStore
LogStore implementation for Azure.
LogStore implementation for Azure.
We assume the following from Azure's FileSystem implementations: - Rename without overwrite is atomic. - List-after-write is consistent.
Regarding file creation, this implementation: - Uses atomic rename when overwrite is false; if the destination file exists or the rename fails, throws an exception. - Uses create-with-overwrite when overwrite is true. This does not make the file atomically visible and therefore the caller must handle partial files.
- trait ClosableIterator[T] extends Iterator[T] with Closeable
-
class
DelegatingLogStore extends LogStore with DeltaLogging
A delegating LogStore used to dynamically resolve LogStore implementation based on the scheme of paths.
-
case class
FileMetadata(length: Long, modificationTime: Long) extends Product with Serializable
The file metadata to be stored in the cache.
-
class
HDFSLogStore extends HadoopFileSystemLogStore with Logging
The LogStore implementation for HDFS, which uses Hadoop FileContext API's to provide the necessary atomic and durability guarantees:
The LogStore implementation for HDFS, which uses Hadoop FileContext API's to provide the necessary atomic and durability guarantees:
1. Atomic visibility of files:
FileContext.renameis used write files which is atomic for HDFS.2. Consistent file listing: HDFS file listing is consistent.
-
abstract
class
HadoopFileSystemLogStore extends LogStore
Default implementation of LogStore for Hadoop FileSystem implementations.
-
class
LineClosableIterator extends ClosableIterator[String]
Turn a
ReadertoClosableIteratorwhich can be read on demand.Turn a
ReadertoClosableIteratorwhich can be read on demand. Each element is a trimmed line. -
class
LocalLogStore extends HadoopFileSystemLogStore
Default LogStore implementation (should be used for testing only!).
Default LogStore implementation (should be used for testing only!).
Production users should specify the appropriate implementation in Spark properties.
We assume the following from org.apache.hadoop.fs.FileSystem implementations: - Rename without overwrite is atomic. - List-after-write is consistent.
Regarding file creation, this implementation: - Uses atomic rename when overwrite is false; if the destination file exists or the rename fails, throws an exception. - Uses create-with-overwrite when overwrite is true. This does not make the file atomically visible and therefore the caller must handle partial files.
-
trait
LogStore extends AnyRef
General interface for all critical file system operations required to read and write the DeltaLog.
General interface for all critical file system operations required to read and write the DeltaLog. The correctness of the DeltaLog is predicated on the atomicity and durability guarantees of the implementation of this interface. Specifically,
1. Atomic visibility of files: Any file written through this store must be made visible atomically. In other words, this should not generate partial files.
2. Mutual exclusion: Only one writer must be able to create (or rename) a file at the final destination.
3. Consistent listing: Once a file has been written in a directory, all future listings for that directory must return that file.
-
class
LogStoreAdaptor extends LogStore
An adaptor from the new public LogStore API to the old private LogStore API.
An adaptor from the new public LogStore API to the old private LogStore API. The old LogStore API is still used in most places. Before we move all of them to the new API, adapting from the new API to the old API is a cheap way to ensure that implementations of both APIs work.
- class LogStoreInverseAdaptor extends io.delta.storage.LogStore
- trait LogStoreProvider extends AnyRef
-
class
S3SingleDriverLogStore extends HadoopFileSystemLogStore
Single Spark-driver/JVM LogStore implementation for S3.
Single Spark-driver/JVM LogStore implementation for S3.
We assume the following from S3's FileSystem implementations: - File writing on S3 is all-or-nothing, whether overwrite or not. - List-after-write can be inconsistent.
Regarding file creation, this implementation: - Opens a stream to write to S3 (regardless of the overwrite option). - Failures during stream write may leak resources, but may never result in partial writes.
Regarding directory listing, this implementation: - returns a list by merging the files listed from S3 and recently-written files from the cache.
- trait SupportsRewinding[T] extends Iterator[T]
Value Members
- object ClosableIterator
- object DelegatingLogStore
- object LogStore extends LogStoreProvider with Logging
- object LogStoreInverseAdaptor
- object S3SingleDriverLogStore