A C D F G H I L N R S T W

A

AzureLogStore - Class in io.delta.storage

LogStore implementation for Azure.

AzureLogStore(Configuration) - Constructor for class io.delta.storage.AzureLogStore

C

close() - Method in class io.delta.storage.LineCloseableIterator

CloseableIterator<T> - Interface in io.delta.storage

:: DeveloperApi :: An iterator that may contain resources which should be released after use.

createTempPath(Path) - Method in class io.delta.storage.HadoopFileSystemLogStore

Create a temporary path (to be used as a copy) for the input path

D

deltaFile(Path, long) - Static method in class io.delta.storage.internal.FileNameUtils

Returns the delta (json format) path for a given delta file.

deltaVersion(Path) - Static method in class io.delta.storage.internal.FileNameUtils

Returns the version for the given delta path.

F

FileNameUtils - Class in io.delta.storage.internal

Helper for misc functions relating to file names for delta commits.

FileNameUtils() - Constructor for class io.delta.storage.internal.FileNameUtils

G

GCSLogStore - Class in io.delta.storage

The LogStore implementation for GCS, which uses gcs-connector to provide the necessary atomic and durability guarantees: Atomic Visibility: Read/read-after-metadata-update/delete are strongly consistent for GCS. Consistent Listing: GCS guarantees strong consistency for both object and bucket listing operations.

GCSLogStore(Configuration) - Constructor for class io.delta.storage.GCSLogStore

H

HadoopFileSystemLogStore - Class in io.delta.storage

Default implementation of LogStore for Hadoop FileSystem implementations.

HadoopFileSystemLogStore(Configuration) - Constructor for class io.delta.storage.HadoopFileSystemLogStore

hasNext() - Method in class io.delta.storage.LineCloseableIterator

HDFSLogStore - Class in io.delta.storage

The LogStore implementation for HDFS, which uses Hadoop FileContext API's to provide the necessary atomic and durability guarantees: Atomic visibility of files: `FileContext.rename` is used write files which is atomic for HDFS. Consistent file listing: HDFS file listing is consistent.

HDFSLogStore(Configuration) - Constructor for class io.delta.storage.HDFSLogStore

I

incorrectLogStoreImplementationException(Throwable) - Static method in class io.delta.storage.internal.LogStoreErrors

initHadoopConf() - Method in class io.delta.storage.LogStore

:: DeveloperApi :: Hadoop configuration that should only be used during initialization of LogStore.

io.delta.storage - package io.delta.storage

io.delta.storage.internal - package io.delta.storage.internal

isDeltaFile(Path) - Static method in class io.delta.storage.internal.FileNameUtils

Returns true if the given path is a delta file, else false.

isNonFatal(Throwable) - Static method in class io.delta.storage.internal.LogStoreErrors

Returns true if the provided Throwable is to be considered non-fatal, or false if it is to be considered fatal

isPartialWriteVisible(Path, Configuration) - Method in class io.delta.storage.AzureLogStore

isPartialWriteVisible(Path, Configuration) - Method in class io.delta.storage.GCSLogStore

isPartialWriteVisible(Path, Configuration) - Method in class io.delta.storage.HDFSLogStore

isPartialWriteVisible(Path, Configuration) - Method in class io.delta.storage.LocalLogStore

isPartialWriteVisible(Path, Configuration) - Method in class io.delta.storage.LogStore

:: DeveloperApi :: Whether a partial write is visible for the underlying file system of `path`.

isPartialWriteVisible(Path, Configuration) - Method in class io.delta.storage.S3SingleDriverLogStore

L

LineCloseableIterator - Class in io.delta.storage

Turn a Reader to CloseableIterator which can be read on demand.

LineCloseableIterator(Reader) - Constructor for class io.delta.storage.LineCloseableIterator

listFrom(Path, Configuration) - Method in class io.delta.storage.HadoopFileSystemLogStore

listFrom(Path, Configuration) - Method in class io.delta.storage.LogStore

:: DeveloperApi :: List the paths in the same directory that are lexicographically greater or equal to (UTF-8 sorting) the given `path`.

listFrom(Path, Configuration) - Method in class io.delta.storage.S3SingleDriverLogStore

LocalLogStore - Class in io.delta.storage

Default LogStore implementation (should be used for testing only!).

LocalLogStore(Configuration) - Constructor for class io.delta.storage.LocalLogStore

LogStore - Class in io.delta.storage

:: DeveloperApi ::

LogStore(Configuration) - Constructor for class io.delta.storage.LogStore

LogStoreErrors - Class in io.delta.storage.internal

LogStoreErrors() - Constructor for class io.delta.storage.internal.LogStoreErrors

N

next() - Method in class io.delta.storage.LineCloseableIterator

NO_ABSTRACT_FILE_SYSTEM_EXCEPTION_MESSAGE - Static variable in class io.delta.storage.HDFSLogStore

R

read(Path, Configuration) - Method in class io.delta.storage.HadoopFileSystemLogStore

read(Path, Configuration) - Method in class io.delta.storage.LogStore

:: DeveloperApi :: Load the given file and return an `Iterator` of lines, with line breaks removed from each line.

resolvePathOnPhysicalStorage(Path, Configuration) - Method in class io.delta.storage.HadoopFileSystemLogStore

resolvePathOnPhysicalStorage(Path, Configuration) - Method in class io.delta.storage.LogStore

:: DeveloperApi :: Resolve the fully qualified path for the given `path`.

runInNewThread(String, boolean, Callable<T>) - Static method in class io.delta.storage.internal.ThreadUtils

Based on Apache Spark's ThreadUtils.runInNewThread Run a piece of code in a new thread and return the result.

S

S3SingleDriverLogStore - Class in io.delta.storage

Single Spark-driver/JVM LogStore implementation for S3.

S3SingleDriverLogStore(Configuration) - Constructor for class io.delta.storage.S3SingleDriverLogStore

T

ThreadUtils - Class in io.delta.storage.internal

ThreadUtils() - Constructor for class io.delta.storage.internal.ThreadUtils

W

write(Path, Iterator<String>, Boolean, Configuration) - Method in class io.delta.storage.AzureLogStore

write(Path, Iterator<String>, Boolean, Configuration) - Method in class io.delta.storage.GCSLogStore

write(Path, Iterator<String>, Boolean, Configuration) - Method in class io.delta.storage.HDFSLogStore

write(Path, Iterator<String>, Boolean, Configuration) - Method in class io.delta.storage.LocalLogStore

This write implementation needs to wrap `writeWithRename` with `synchronized` as rename() for RawLocalFileSystem doesn't throw an exception when the target file exists.

write(Path, Iterator<String>, Boolean, Configuration) - Method in class io.delta.storage.LogStore

:: DeveloperApi :: Write the given `actions` to the given `path` with or without overwrite as indicated.

write(Path, Iterator<String>, Boolean, Configuration) - Method in class io.delta.storage.S3SingleDriverLogStore

writeWithRename(Path, Iterator<String>, Boolean, Configuration) - Method in class io.delta.storage.HadoopFileSystemLogStore

An internal write implementation that uses FileSystem.rename().

A C D F G H I L N R S T W