io.delta.storage

Class HDFSLogStore

Object

io.delta.storage.LogStore

io.delta.storage.HadoopFileSystemLogStore

io.delta.storage.HDFSLogStore

public class HDFSLogStore
extends HadoopFileSystemLogStore

The LogStore implementation for HDFS, which uses Hadoop FileContext API's to provide the necessary atomic and durability guarantees:

1. Atomic visibility of files: `FileContext.rename` is used write files which is atomic for HDFS.
2. Consistent file listing: HDFS file listing is consistent.

Field Summary

Fields

Modifier and Type	Field and Description
static String	NO_ABSTRACT_FILE_SYSTEM_EXCEPTION_MESSAGE

Constructor Summary

Constructors

Constructor and Description
HDFSLogStore(org.apache.hadoop.conf.Configuration hadoopConf)

Method Summary

Modifier and Type	Method and Description
Boolean	isPartialWriteVisible(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration hadoopConf) :: DeveloperApi :: Whether a partial write is visible for the underlying file system of `path`.
void	write(org.apache.hadoop.fs.Path path, java.util.Iterator<String> actions, Boolean overwrite, org.apache.hadoop.conf.Configuration hadoopConf) :: DeveloperApi :: Write the given `actions` to the given `path` with or without overwrite as indicated.

Methods inherited from class io.delta.storage.HadoopFileSystemLogStore

listFrom, read, resolvePathOnPhysicalStorage

Methods inherited from class io.delta.storage.LogStore

initHadoopConf

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

NO_ABSTRACT_FILE_SYSTEM_EXCEPTION_MESSAGE

public static final String NO_ABSTRACT_FILE_SYSTEM_EXCEPTION_MESSAGE

See Also:

Constant Field Values

Constructor Detail

HDFSLogStore

public HDFSLogStore(org.apache.hadoop.conf.Configuration hadoopConf)

Method Detail

write

public void write(org.apache.hadoop.fs.Path path,
                  java.util.Iterator<String> actions,
                  Boolean overwrite,
                  org.apache.hadoop.conf.Configuration hadoopConf)
           throws java.io.IOException

Description copied from class: LogStore

:: DeveloperApi :: Write the given `actions` to the given `path` with or without overwrite as indicated. Implementation must throw FileAlreadyExistsException exception if the file already exists and overwrite = false. Furthermore, if isPartialWriteVisible returns false, implementation must ensure that the entire file is made visible atomically, that is, it should not generate partial files.

Specified by:

write in class LogStore

Throws:

java.io.IOException - if there's an issue resolving the FileSystem

java.nio.file.FileAlreadyExistsException - if the file already exists and overwrite is false

isPartialWriteVisible

public Boolean isPartialWriteVisible(org.apache.hadoop.fs.Path path,
                                     org.apache.hadoop.conf.Configuration hadoopConf)

Description copied from class: LogStore

:: DeveloperApi :: Whether a partial write is visible for the underlying file system of `path`.

Specified by:

isPartialWriteVisible in class LogStore