Type Members

1.  class AzureLogStore extends HadoopFileSystemLogStore

   LogStore implementation for Azure.

   LogStore implementation for Azure.

   We assume the following from Azure's FileSystem implementations: - Rename without overwrite is atomic. - List-after-write is consistent.

   Regarding file creation, this implementation: - Uses atomic rename when overwrite is false; if the destination file exists or the rename fails, throws an exception. - Uses create-with-overwrite when overwrite is true. This does not make the file atomically visible and therefore the caller must handle partial files.

2.  case class FileMetadata(length: Long, modificationTime: Long) extends Product with Serializable

   The file metadata to be stored in the cache.

3.  class HDFSLogStore extends HadoopFileSystemLogStore with Logging

   The LogStore implementation for HDFS, which uses Hadoop FileContext API's to provide the necessary atomic and durability guarantees:

   The LogStore implementation for HDFS, which uses Hadoop FileContext API's to provide the necessary atomic and durability guarantees:

   1. Atomic visibility of files: FileContext.rename is used write files which is atomic for HDFS.

   2. Consistent file listing: HDFS file listing is consistent.

4.  abstract class HadoopFileSystemLogStore extends LogStore

   Default implementation of LogStore for Hadoop FileSystem implementations.

5.  class LocalLogStore extends HadoopFileSystemLogStore

   Default LogStore implementation (should be used for testing only!).

   Default LogStore implementation (should be used for testing only!).

   Production users should specify the appropriate implementation in Spark properties.

   We assume the following from org.apache.hadoop.fs.FileSystem implementations: - Rename without overwrite is atomic. - List-after-write is consistent.

   Regarding file creation, this implementation: - Uses atomic rename when overwrite is false; if the destination file exists or the rename fails, throws an exception. - Uses create-with-overwrite when overwrite is true. This does not make the file atomically visible and therefore the caller must handle partial files.

6.  trait LogStore extends AnyRef

   General interface for all critical file system operations required to read and write the DeltaLog.

   General interface for all critical file system operations required to read and write the DeltaLog. The correctness of the DeltaLog is predicated on the atomicity and durability guarantees of the implementation of this interface. Specifically,

   1. Atomic visibility of files: Any file written through this store must be made visible atomically. In other words, this should not generate partial files.

   2. Mutual exclusion: Only one writer must be able to create (or rename) a file at the final destination.

   3. Consistent listing: Once a file has been written in a directory, all future listings for that directory must return that file.

7.  trait LogStoreProvider extends AnyRef
8.  class S3SingleDriverLogStore extends HadoopFileSystemLogStore

   Single Spark-driver/JVM LogStore implementation for S3.

   Single Spark-driver/JVM LogStore implementation for S3.

   We assume the following from S3's FileSystem implementations: - File writing on S3 is all-or-nothing, whether overwrite or not. - List-after-write can be inconsistent.

   Regarding file creation, this implementation: - Opens a stream to write to S3 (regardless of the overwrite option). - Failures during stream write may leak resources, but may never result in partial writes.

   Regarding directory listing, this implementation: - returns a list by merging the files listed from S3 and recently-written files from the cache.