Package io.delta.storage

Class GCSLogStore

Object

io.delta.storage.LogStore

io.delta.storage.HadoopFileSystemLogStore

io.delta.storage.GCSLogStore

public class GCSLogStore
extends HadoopFileSystemLogStore

The LogStore implementation for GCS, which uses gcs-connector to provide the necessary atomic and durability guarantees:

1. Atomic Visibility: Read/read-after-metadata-update/delete are strongly consistent for GCS.
2. Consistent Listing: GCS guarantees strong consistency for both object and bucket listing operations. https://cloud.google.com/storage/docs/consistency
3. Mutual Exclusion: Preconditions are used to handle race conditions.

Regarding file creation, this implementation:

• Opens a stream to write to GCS otherwise.
• Throws [[FileAlreadyExistsException]] if file exists and overwrite is false.
• Assumes file writing to be all-or-nothing, irrespective of overwrite option.

This class is not meant for direct access but for configuration based on storage system. See https://docs.delta.io/latest/delta-storage.html for details.

Constructor Summary

Constructors

Constructor	Description
GCSLogStore(org.apache.hadoop.conf.Configuration hadoopConf)

Method Summary

Modifier and Type	Method	Description
Boolean	isPartialWriteVisible(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration hadoopConf)	:: DeveloperApi :: Whether a partial write is visible for the underlying file system of `path`.
void	write(org.apache.hadoop.fs.Path path, Iterator<String> actions, Boolean overwrite, org.apache.hadoop.conf.Configuration hadoopConf)	:: DeveloperApi :: Write the given `actions` to the given `path` with or without overwrite as indicated.

Methods inherited from class io.delta.storage.HadoopFileSystemLogStore

listFrom, read, resolvePathOnPhysicalStorage

Methods inherited from class io.delta.storage.LogStore

initHadoopConf

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

GCSLogStore

public GCSLogStore(org.apache.hadoop.conf.Configuration hadoopConf)

Method Details

write

public void write(org.apache.hadoop.fs.Path path,
 Iterator<String> actions,
 Boolean overwrite,
 org.apache.hadoop.conf.Configuration hadoopConf)
           throws IOException

Description copied from class: LogStore

:: DeveloperApi :: Write the given `actions` to the given `path` with or without overwrite as indicated. Implementation must throw FileAlreadyExistsException exception if the file already exists and overwrite = false. Furthermore, if isPartialWriteVisible returns false, implementation must ensure that the entire file is made visible atomically, that is, it should not generate partial files.

Specified by:

write in class LogStore

Throws:

IOException - if there's an issue resolving the FileSystem

FileAlreadyExistsException - if the file already exists and overwrite is false

isPartialWriteVisible

public Boolean isPartialWriteVisible(org.apache.hadoop.fs.Path path,
 org.apache.hadoop.conf.Configuration hadoopConf)
                              throws IOException

Description copied from class: LogStore

:: DeveloperApi :: Whether a partial write is visible for the underlying file system of `path`.

Specified by:

isPartialWriteVisible in class LogStore

Throws:

IOException - if there's an issue resolving the FileSystem