io.delta.storage

Class S3SingleDriverLogStore

java.lang.Object

io.delta.storage.LogStore

io.delta.storage.HadoopFileSystemLogStore

io.delta.storage.S3SingleDriverLogStore

public class S3SingleDriverLogStore
extends HadoopFileSystemLogStore

Single Spark-driver/JVM LogStore implementation for S3.

We assume the following from S3's FileSystem implementations:

• File writing on S3 is all-or-nothing, whether overwrite or not.
• List-after-write can be inconsistent.

Regarding file creation, this implementation:

• Opens a stream to write to S3 (regardless of the overwrite option).
• Failures during stream write may leak resources, but may never result in partial writes.

Regarding directory listing, this implementation:

• returns a list by merging the files listed from S3 and recently-written files from the cache.

Constructor Summary

Constructors

Constructor and Description
S3SingleDriverLogStore(org.apache.hadoop.conf.Configuration hadoopConf)

Method Summary

Modifier and Type	Method and Description
java.lang.Boolean	isPartialWriteVisible(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration hadoopConf) :: DeveloperApi :: Whether a partial write is visible for the underlying file system of `path`.
java.util.Iterator<org.apache.hadoop.fs.FileStatus>	listFrom(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration hadoopConf) :: DeveloperApi :: List the paths in the same directory that are lexicographically greater or equal to (UTF-8 sorting) the given `path`.
void	write(org.apache.hadoop.fs.Path path, java.util.Iterator<java.lang.String> actions, java.lang.Boolean overwrite, org.apache.hadoop.conf.Configuration hadoopConf) :: DeveloperApi :: Write the given `actions` to the given `path` with or without overwrite as indicated.

Methods inherited from class io.delta.storage.HadoopFileSystemLogStore

createTempPath, read, resolvePathOnPhysicalStorage, writeWithRename

Methods inherited from class io.delta.storage.LogStore

initHadoopConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

S3SingleDriverLogStore

public S3SingleDriverLogStore(org.apache.hadoop.conf.Configuration hadoopConf)

Method Detail

write

public void write(org.apache.hadoop.fs.Path path,
                  java.util.Iterator<java.lang.String> actions,
                  java.lang.Boolean overwrite,
                  org.apache.hadoop.conf.Configuration hadoopConf)
           throws java.io.IOException

Description copied from class: LogStore

:: DeveloperApi :: Write the given `actions` to the given `path` with or without overwrite as indicated. Implementation must throw FileAlreadyExistsException exception if the file already exists and overwrite = false. Furthermore, if isPartialWriteVisible returns false, implementation must ensure that the entire file is made visible atomically, that is, it should not generate partial files.

Specified by:

write in class LogStore

Throws:

java.io.IOException - if there's an issue resolving the FileSystem

java.nio.file.FileAlreadyExistsException - if the file already exists and overwrite is false

listFrom

public java.util.Iterator<org.apache.hadoop.fs.FileStatus> listFrom(org.apache.hadoop.fs.Path path,
                                                                    org.apache.hadoop.conf.Configuration hadoopConf)
                                                             throws java.io.IOException

Description copied from class: LogStore

:: DeveloperApi :: List the paths in the same directory that are lexicographically greater or equal to (UTF-8 sorting) the given `path`. The result should also be sorted by the file name.

Overrides:

listFrom in class HadoopFileSystemLogStore

Throws:

java.io.IOException - if there's an issue resolving the FileSystem

java.nio.file.FileAlreadyExistsException - if path directory can't be found

isPartialWriteVisible

public java.lang.Boolean isPartialWriteVisible(org.apache.hadoop.fs.Path path,
                                               org.apache.hadoop.conf.Configuration hadoopConf)

Description copied from class: LogStore

:: DeveloperApi :: Whether a partial write is visible for the underlying file system of `path`.

Specified by:

isPartialWriteVisible in class LogStore