io.delta.storage

Class HadoopFileSystemLogStore

java.lang.Object

io.delta.storage.LogStore

io.delta.storage.HadoopFileSystemLogStore

Direct Known Subclasses:

AzureLogStore, GCSLogStore, HDFSLogStore, LocalLogStore, S3SingleDriverLogStore

public abstract class HadoopFileSystemLogStore
extends LogStore

Default implementation of LogStore for Hadoop FileSystem implementations.

Constructor Summary

Constructors

Constructor and Description
HadoopFileSystemLogStore(org.apache.hadoop.conf.Configuration hadoopConf)

Method Summary

Modifier and Type	Method and Description
protected org.apache.hadoop.fs.Path	createTempPath(org.apache.hadoop.fs.Path path) Create a temporary path (to be used as a copy) for the input path
java.util.Iterator<org.apache.hadoop.fs.FileStatus>	listFrom(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration hadoopConf) :: DeveloperApi :: List the paths in the same directory that are lexicographically greater or equal to (UTF-8 sorting) the given `path`.
CloseableIterator<java.lang.String>	read(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration hadoopConf) :: DeveloperApi :: Load the given file and return an `Iterator` of lines, with line breaks removed from each line.
org.apache.hadoop.fs.Path	resolvePathOnPhysicalStorage(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration hadoopConf) :: DeveloperApi :: Resolve the fully qualified path for the given `path`.
protected void	writeWithRename(org.apache.hadoop.fs.Path path, java.util.Iterator<java.lang.String> actions, java.lang.Boolean overwrite, org.apache.hadoop.conf.Configuration hadoopConf) An internal write implementation that uses FileSystem.rename().

Methods inherited from class io.delta.storage.LogStore

initHadoopConf, isPartialWriteVisible, write

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HadoopFileSystemLogStore

public HadoopFileSystemLogStore(org.apache.hadoop.conf.Configuration hadoopConf)

Method Detail

read

public CloseableIterator<java.lang.String> read(org.apache.hadoop.fs.Path path,
                                                org.apache.hadoop.conf.Configuration hadoopConf)
                                         throws java.io.IOException

Description copied from class: LogStore

:: DeveloperApi :: Load the given file and return an `Iterator` of lines, with line breaks removed from each line. Callers of this function are responsible to close the iterator if they are done with it.

Specified by:

read in class LogStore

Throws:

java.io.IOException - if there's an issue resolving the FileSystem

listFrom

public java.util.Iterator<org.apache.hadoop.fs.FileStatus> listFrom(org.apache.hadoop.fs.Path path,
                                                                    org.apache.hadoop.conf.Configuration hadoopConf)
                                                             throws java.io.IOException

Description copied from class: LogStore

:: DeveloperApi :: List the paths in the same directory that are lexicographically greater or equal to (UTF-8 sorting) the given `path`. The result should also be sorted by the file name.

Specified by:

listFrom in class LogStore

Throws:

java.io.IOException - if there's an issue resolving the FileSystem

java.nio.file.FileAlreadyExistsException - if path directory can't be found

resolvePathOnPhysicalStorage

public org.apache.hadoop.fs.Path resolvePathOnPhysicalStorage(org.apache.hadoop.fs.Path path,
                                                              org.apache.hadoop.conf.Configuration hadoopConf)
                                                       throws java.io.IOException

Description copied from class: LogStore

:: DeveloperApi :: Resolve the fully qualified path for the given `path`.

Specified by:

resolvePathOnPhysicalStorage in class LogStore

Throws:

java.io.IOException - if there's an issue resolving the FileSystem

writeWithRename

protected void writeWithRename(org.apache.hadoop.fs.Path path,
                               java.util.Iterator<java.lang.String> actions,
                               java.lang.Boolean overwrite,
                               org.apache.hadoop.conf.Configuration hadoopConf)
                        throws java.io.IOException

An internal write implementation that uses FileSystem.rename().

This implementation should only be used for the underlying file systems that support atomic renames, e.g., Azure is OK but HDFS is not.

Throws:

java.io.IOException

createTempPath

protected org.apache.hadoop.fs.Path createTempPath(org.apache.hadoop.fs.Path path)

Create a temporary path (to be used as a copy) for the input path