io.delta.storage.commit

Class CoordinatedCommitsUtils

Object

io.delta.storage.commit.CoordinatedCommitsUtils

public class CoordinatedCommitsUtils
extends Object

Method Summary

Modifier and Type	Method and Description
static org.apache.hadoop.fs.Path	commitDirPath(org.apache.hadoop.fs.Path logPath) Returns path to the directory which holds the unbackfilled commits
static org.apache.hadoop.fs.Path	generateUnbackfilledDeltaFilePath(org.apache.hadoop.fs.Path logPath, long version) Creates a new unbackfilled delta file path for the given commit version.
static org.apache.hadoop.fs.Path	getBackfilledDeltaFilePath(org.apache.hadoop.fs.Path logPath, Long version) Returns the path to the backfilled delta file for the given commit version.
static java.util.Map<String,String>	getCoordinatorConf(AbstractMetadata abstractMetadata) Get the coordinated commits owner configuration from the provided abstract metadata.
static java.util.Optional<String>	getCoordinatorName(AbstractMetadata metadata) Retrieves the coordinator name from the provided abstract metadata.
static java.util.Map<String,String>	getTableConf(AbstractMetadata abstractMetadata) Get the coordinated commits table configuration from the provided abstract metadata.
static org.apache.hadoop.fs.Path	getTablePath(org.apache.hadoop.fs.Path logPath) Get the table path from the provided log path.
static org.apache.hadoop.fs.Path	getUnbackfilledDeltaFile(org.apache.hadoop.fs.Path logPath, long version, java.util.Optional<String> uuidString) Returns the un-backfilled uuid formatted delta (json format) path for a given version.
static boolean	isCoordinatedCommitsToFSConversion(Long commitVersion, UpdatedActions updatedActions) Returns true if the commit is a coordinated commits to filesystem conversion.
static org.apache.hadoop.fs.Path	logDirPath(org.apache.hadoop.fs.Path tablePath) Returns path to the directory which holds the delta log
static org.apache.hadoop.fs.FileStatus	writeUnbackfilledCommitFile(LogStore logStore, org.apache.hadoop.conf.Configuration hadoopConf, String logPath, long commitVersion, java.util.Iterator<String> actions, String uuid) Write a UUID-based commit file for the specified version to the table at logPath.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

generateUnbackfilledDeltaFilePath

public static org.apache.hadoop.fs.Path generateUnbackfilledDeltaFilePath(org.apache.hadoop.fs.Path logPath,
                                                                          long version)

Creates a new unbackfilled delta file path for the given commit version. The path is of the form `tablePath/_delta_log/_commits/00000000000000000001.uuid.json`.

getBackfilledDeltaFilePath

public static org.apache.hadoop.fs.Path getBackfilledDeltaFilePath(org.apache.hadoop.fs.Path logPath,
                                                                   Long version)

Returns the path to the backfilled delta file for the given commit version. The path is of the form `tablePath/_delta_log/00000000000000000001.json`.

isCoordinatedCommitsToFSConversion

public static boolean isCoordinatedCommitsToFSConversion(Long commitVersion,
                                                         UpdatedActions updatedActions)

Returns true if the commit is a coordinated commits to filesystem conversion.

getTablePath

public static org.apache.hadoop.fs.Path getTablePath(org.apache.hadoop.fs.Path logPath)

Get the table path from the provided log path.

getUnbackfilledDeltaFile

public static org.apache.hadoop.fs.Path getUnbackfilledDeltaFile(org.apache.hadoop.fs.Path logPath,
                                                                 long version,
                                                                 java.util.Optional<String> uuidString)

Returns the un-backfilled uuid formatted delta (json format) path for a given version.

Parameters:

logPath - The root path of the delta log.

version - The version of the delta file.

Returns:

The path to the un-backfilled delta file: logPath/_commits/version.uuid.json

writeUnbackfilledCommitFile

public static org.apache.hadoop.fs.FileStatus writeUnbackfilledCommitFile(LogStore logStore,
                                                                          org.apache.hadoop.conf.Configuration hadoopConf,
                                                                          String logPath,
                                                                          long commitVersion,
                                                                          java.util.Iterator<String> actions,
                                                                          String uuid)
                                                                   throws java.io.IOException

Write a UUID-based commit file for the specified version to the table at logPath.

Throws:

java.io.IOException

logDirPath

public static org.apache.hadoop.fs.Path logDirPath(org.apache.hadoop.fs.Path tablePath)

Returns path to the directory which holds the delta log

commitDirPath

public static org.apache.hadoop.fs.Path commitDirPath(org.apache.hadoop.fs.Path logPath)

Returns path to the directory which holds the unbackfilled commits

getCoordinatorName

public static java.util.Optional<String> getCoordinatorName(AbstractMetadata metadata)

Retrieves the coordinator name from the provided abstract metadata. If no coordinator is set, an empty optional is returned.

Parameters:

metadata - The abstract metadata from which to retrieve the coordinator name.

Returns:

The coordinator name if set, otherwise an empty optional.

getCoordinatorConf

public static java.util.Map<String,String> getCoordinatorConf(AbstractMetadata abstractMetadata)

Get the coordinated commits owner configuration from the provided abstract metadata.

getTableConf

public static java.util.Map<String,String> getTableConf(AbstractMetadata abstractMetadata)

Get the coordinated commits table configuration from the provided abstract metadata.