io.delta.storage

Class AzureLogStore

java.lang.Object

io.delta.storage.LogStore

io.delta.storage.HadoopFileSystemLogStore

io.delta.storage.AzureLogStore

public class AzureLogStore
extends HadoopFileSystemLogStore

LogStore implementation for Azure.

We assume the following from Azure's [[FileSystem]] implementations:

• Rename without overwrite is atomic.
• List-after-write is consistent.

Regarding file creation, this implementation:

• Uses atomic rename when overwrite is false; if the destination file exists or the rename fails, throws an exception.
• Uses create-with-overwrite when overwrite is true. This does not make the file atomically visible and therefore the caller must handle partial files.

Constructor Summary

Constructors

Constructor and Description
AzureLogStore(org.apache.hadoop.conf.Configuration hadoopConf)

Method Summary

Modifier and Type	Method and Description
java.lang.Boolean	isPartialWriteVisible(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration hadoopConf) :: DeveloperApi :: Whether a partial write is visible for the underlying file system of `path`.
void	write(org.apache.hadoop.fs.Path path, java.util.Iterator<java.lang.String> actions, java.lang.Boolean overwrite, org.apache.hadoop.conf.Configuration hadoopConf) :: DeveloperApi :: Write the given `actions` to the given `path` with or without overwrite as indicated.

Methods inherited from class io.delta.storage.HadoopFileSystemLogStore

createTempPath, listFrom, read, resolvePathOnPhysicalStorage, writeWithRename

Methods inherited from class io.delta.storage.LogStore

initHadoopConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AzureLogStore

public AzureLogStore(org.apache.hadoop.conf.Configuration hadoopConf)

Method Detail

write

public void write(org.apache.hadoop.fs.Path path,
                  java.util.Iterator<java.lang.String> actions,
                  java.lang.Boolean overwrite,
                  org.apache.hadoop.conf.Configuration hadoopConf)
           throws java.io.IOException

Description copied from class: LogStore

:: DeveloperApi :: Write the given `actions` to the given `path` with or without overwrite as indicated. Implementation must throw FileAlreadyExistsException exception if the file already exists and overwrite = false. Furthermore, if isPartialWriteVisible returns false, implementation must ensure that the entire file is made visible atomically, that is, it should not generate partial files.

Specified by:

write in class LogStore

Throws:

java.io.IOException - if there's an issue resolving the FileSystem

java.nio.file.FileAlreadyExistsException - if the file already exists and overwrite is false

isPartialWriteVisible

public java.lang.Boolean isPartialWriteVisible(org.apache.hadoop.fs.Path path,
                                               org.apache.hadoop.conf.Configuration hadoopConf)

Description copied from class: LogStore

:: DeveloperApi :: Whether a partial write is visible for the underlying file system of `path`.

Specified by:

isPartialWriteVisible in class LogStore