io.delta.storage

Class LocalLogStore

Object

io.delta.storage.LogStore

io.delta.storage.HadoopFileSystemLogStore

io.delta.storage.LocalLogStore

public class LocalLogStore
extends HadoopFileSystemLogStore

Default LogStore implementation (should be used for testing only!). Production users should specify the appropriate LogStore implementation in Spark properties.

We assume the following from FileSystem implementations:

• Rename without overwrite is atomic.
• List-after-write is consistent.

Regarding file creation, this implementation:

• Uses atomic rename when overwrite is false; if the destination file exists or the rename fails, throws an exception.
• Uses create-with-overwrite when overwrite is true. This does not make the file atomically visible and therefore the caller must handle partial files.

Constructor Summary

Constructors

Constructor and Description
LocalLogStore(org.apache.hadoop.conf.Configuration hadoopConf)

Method Summary

Modifier and Type	Method and Description
Boolean	isPartialWriteVisible(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration hadoopConf) :: DeveloperApi :: Whether a partial write is visible for the underlying file system of `path`.
void	write(org.apache.hadoop.fs.Path path, java.util.Iterator<String> actions, Boolean overwrite, org.apache.hadoop.conf.Configuration hadoopConf) This write implementation needs to wrap `writeWithRename` with `synchronized` as rename() for RawLocalFileSystem doesn't throw an exception when the target file exists.

Methods inherited from class io.delta.storage.HadoopFileSystemLogStore

listFrom, read, resolvePathOnPhysicalStorage

Methods inherited from class io.delta.storage.LogStore

initHadoopConf

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

LocalLogStore

public LocalLogStore(org.apache.hadoop.conf.Configuration hadoopConf)

Method Detail

write

public void write(org.apache.hadoop.fs.Path path,
                  java.util.Iterator<String> actions,
                  Boolean overwrite,
                  org.apache.hadoop.conf.Configuration hadoopConf)
           throws java.io.IOException

This write implementation needs to wrap `writeWithRename` with `synchronized` as rename() for RawLocalFileSystem doesn't throw an exception when the target file exists. Hence, we must make sure `exists + rename` in `writeWithRename` is atomic in our tests.

Specified by:

write in class LogStore

Throws:

java.io.IOException - if there's an issue resolving the FileSystem

java.nio.file.FileAlreadyExistsException - if the file already exists and overwrite is false

isPartialWriteVisible

public Boolean isPartialWriteVisible(org.apache.hadoop.fs.Path path,
                                     org.apache.hadoop.conf.Configuration hadoopConf)
                              throws java.io.IOException

Description copied from class: LogStore

:: DeveloperApi :: Whether a partial write is visible for the underlying file system of `path`.

Specified by:

isPartialWriteVisible in class LogStore

Throws:

java.io.IOException - if there's an issue resolving the FileSystem