Package io.delta.storage

Class HadoopFileSystemLogStore

Object
io.delta.storage.LogStore
io.delta.storage.HadoopFileSystemLogStore
Direct Known Subclasses:
AzureLogStore, GCSLogStore, HDFSLogStore, LocalLogStore, S3SingleDriverLogStore
public abstract class HadoopFileSystemLogStore extends LogStore
Default implementation of LogStore for Hadoop FileSystem implementations.

  • Constructor Summary

    Constructors
    Constructor
    Description
    HadoopFileSystemLogStore(org.apache.hadoop.conf.Configuration hadoopConf)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    Iterator<org.apache.hadoop.fs.FileStatus>
    listFrom(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration hadoopConf)
    :: DeveloperApi :: List the paths in the same directory that are lexicographically greater or equal to (UTF-8 sorting) the given `path`.
    CloseableIterator<String>
    read(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration hadoopConf)
    :: DeveloperApi :: Load the given file and return an `Iterator` of lines, with line breaks removed from each line.
    org.apache.hadoop.fs.Path
    resolvePathOnPhysicalStorage(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration hadoopConf)
    :: DeveloperApi :: Resolve the fully qualified path for the given `path`.

    Methods inherited from class io.delta.storage.LogStore

    initHadoopConf, isPartialWriteVisible, write

    Methods inherited from class java.lang.Object

    equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • HadoopFileSystemLogStore

      public HadoopFileSystemLogStore(org.apache.hadoop.conf.Configuration hadoopConf)

  • Method Details

    • read

      public CloseableIterator<String> read(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration hadoopConf) throws IOException
      Description copied from class: LogStore
      :: DeveloperApi :: Load the given file and return an `Iterator` of lines, with line breaks removed from each line. Callers of this function are responsible to close the iterator if they are done with it.
      Specified by:
      read in class LogStore
      Throws:
      IOException - if there's an issue resolving the FileSystem

    • listFrom

      public Iterator<org.apache.hadoop.fs.FileStatus> listFrom(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration hadoopConf) throws IOException
      Description copied from class: LogStore
      :: DeveloperApi :: List the paths in the same directory that are lexicographically greater or equal to (UTF-8 sorting) the given `path`. The result should also be sorted by the file name.
      Specified by:
      listFrom in class LogStore
      Throws:
      IOException - if there's an issue resolving the FileSystem
      FileAlreadyExistsException - if path directory can't be found

    • resolvePathOnPhysicalStorage

      public org.apache.hadoop.fs.Path resolvePathOnPhysicalStorage(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration hadoopConf) throws IOException
      Description copied from class: LogStore
      :: DeveloperApi :: Resolve the fully qualified path for the given `path`.
      Specified by:
      resolvePathOnPhysicalStorage in class LogStore
      Throws:
      IOException - if there's an issue resolving the FileSystem