io.delta.storage

Class HadoopFileSystemLogStore

Object

io.delta.storage.LogStore

io.delta.storage.HadoopFileSystemLogStore

Direct Known Subclasses:

AzureLogStore, GCSLogStore, HDFSLogStore, LocalLogStore, S3SingleDriverLogStore

public abstract class HadoopFileSystemLogStore
extends LogStore

Default implementation of LogStore for Hadoop FileSystem implementations.

Constructor Summary

Constructors

Constructor and Description
HadoopFileSystemLogStore(org.apache.hadoop.conf.Configuration hadoopConf)

Method Summary

Modifier and Type	Method and Description
java.util.Iterator<org.apache.hadoop.fs.FileStatus>	listFrom(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration hadoopConf) :: DeveloperApi :: List the paths in the same directory that are lexicographically greater or equal to (UTF-8 sorting) the given `path`.
CloseableIterator<String>	read(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration hadoopConf) :: DeveloperApi :: Load the given file and return an `Iterator` of lines, with line breaks removed from each line.
org.apache.hadoop.fs.Path	resolvePathOnPhysicalStorage(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration hadoopConf) :: DeveloperApi :: Resolve the fully qualified path for the given `path`.

Methods inherited from class io.delta.storage.LogStore

initHadoopConf, isPartialWriteVisible, write

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HadoopFileSystemLogStore

public HadoopFileSystemLogStore(org.apache.hadoop.conf.Configuration hadoopConf)

Method Detail

read

public CloseableIterator<String> read(org.apache.hadoop.fs.Path path,
                                      org.apache.hadoop.conf.Configuration hadoopConf)
                               throws java.io.IOException

Description copied from class: LogStore

:: DeveloperApi :: Load the given file and return an `Iterator` of lines, with line breaks removed from each line. Callers of this function are responsible to close the iterator if they are done with it.

Specified by:

read in class LogStore

Throws:

java.io.IOException - if there's an issue resolving the FileSystem

listFrom

public java.util.Iterator<org.apache.hadoop.fs.FileStatus> listFrom(org.apache.hadoop.fs.Path path,
                                                                    org.apache.hadoop.conf.Configuration hadoopConf)
                                                             throws java.io.IOException

Description copied from class: LogStore

:: DeveloperApi :: List the paths in the same directory that are lexicographically greater or equal to (UTF-8 sorting) the given `path`. The result should also be sorted by the file name.

Specified by:

listFrom in class LogStore

Throws:

java.io.IOException - if there's an issue resolving the FileSystem

java.nio.file.FileAlreadyExistsException - if path directory can't be found

resolvePathOnPhysicalStorage

public org.apache.hadoop.fs.Path resolvePathOnPhysicalStorage(org.apache.hadoop.fs.Path path,
                                                              org.apache.hadoop.conf.Configuration hadoopConf)
                                                       throws java.io.IOException

Description copied from class: LogStore

:: DeveloperApi :: Resolve the fully qualified path for the given `path`.

Specified by:

resolvePathOnPhysicalStorage in class LogStore

Throws:

java.io.IOException - if there's an issue resolving the FileSystem