io.delta.storage.commit

Interface CommitCoordinatorClient

public interface CommitCoordinatorClient

The CommitCoordinatorClient is responsible for communicating with the commit coordinator and backfilling commits. It has four main APIs that need to be implemented

• Commit a new version of the table. See commit(io.delta.storage.LogStore, org.apache.hadoop.conf.Configuration, io.delta.storage.commit.TableDescriptor, long, java.util.Iterator<java.lang.String>, io.delta.storage.commit.UpdatedActions).
• Ensure that commits are backfilled if/when needed. See backfillToVersion(io.delta.storage.LogStore, org.apache.hadoop.conf.Configuration, io.delta.storage.commit.TableDescriptor, long, java.lang.Long)
• Tracks and returns unbackfilled commits. See getCommits(io.delta.storage.commit.TableDescriptor, java.lang.Long, java.lang.Long).
• Determine the table config during commit coordinator registration. See registerTable(org.apache.hadoop.fs.Path, java.util.Optional<io.delta.storage.commit.TableIdentifier>, long, io.delta.storage.commit.actions.AbstractMetadata, io.delta.storage.commit.actions.AbstractProtocol)

Method Summary

Modifier and Type	Method and Description
void	backfillToVersion(LogStore logStore, org.apache.hadoop.conf.Configuration hadoopConf, TableDescriptor tableDescriptor, long version, Long lastKnownBackfilledVersion) API to ask the commit coordinator client to backfill all commits up to version and notify the commit coordinator.
CommitResponse	commit(LogStore logStore, org.apache.hadoop.conf.Configuration hadoopConf, TableDescriptor tableDescriptor, long commitVersion, java.util.Iterator<String> actions, UpdatedActions updatedActions) API to commit the given set of actions to the table represented by logPath at the given commitVersion.
GetCommitsResponse	getCommits(TableDescriptor tableDescriptor, Long startVersion, Long endVersion) API to get the unbackfilled commits for the table represented by the given logPath.
java.util.Map<String,String>	registerTable(org.apache.hadoop.fs.Path logPath, java.util.Optional<TableIdentifier> tableIdentifier, long currentVersion, AbstractMetadata currentMetadata, AbstractProtocol currentProtocol) API to register the table represented by the given `logPath` at the provided currentTableVersion with the commit coordinator this commit coordinator client represents.
boolean	semanticEquals(CommitCoordinatorClient other) Determines whether this CommitCoordinatorClient is semantically equal to another CommitCoordinatorClient.

Method Detail

registerTable

java.util.Map<String,String> registerTable(org.apache.hadoop.fs.Path logPath,
                                           java.util.Optional<TableIdentifier> tableIdentifier,
                                           long currentVersion,
                                           AbstractMetadata currentMetadata,
                                           AbstractProtocol currentProtocol)

API to register the table represented by the given `logPath` at the provided currentTableVersion with the commit coordinator this commit coordinator client represents. This API is called when the table is being converted from a file system table to a coordinated-commit table. When a new coordinated-commit table is being created, the currentTableVersion will be -1 and the upgrade commit needs to be a file system commit which will write the backfilled file directly.

Parameters:

logPath - The path to the delta log of the table that should be converted

tableIdentifier - The optional tableIdentifier for the table. Some commit coordinators may choose to make this compulsory and error out if this is not provided.

currentVersion - The currentTableVersion is the version of the table just before conversion. currentTableVersion + 1 represents the commit that will do the conversion. This must be backfilled atomically. currentTableVersion + 2 represents the first commit after conversion. This will go through the CommitCoordinatorClient and the client is free to choose when it wants to backfill this commit.

currentMetadata - The metadata of the table at currentTableVersion

currentProtocol - The protocol of the table at currentTableVersion

Returns:

A map of key-value pairs which is issued by the commit coordinator to identify the table. This should be stored in the table's metadata. This information needs to be passed to the commit(io.delta.storage.LogStore, org.apache.hadoop.conf.Configuration, io.delta.storage.commit.TableDescriptor, long, java.util.Iterator<java.lang.String>, io.delta.storage.commit.UpdatedActions), getCommits(io.delta.storage.commit.TableDescriptor, java.lang.Long, java.lang.Long), and backfillToVersion(io.delta.storage.LogStore, org.apache.hadoop.conf.Configuration, io.delta.storage.commit.TableDescriptor, long, java.lang.Long) APIs to identify the table.

commit

CommitResponse commit(LogStore logStore,
                      org.apache.hadoop.conf.Configuration hadoopConf,
                      TableDescriptor tableDescriptor,
                      long commitVersion,
                      java.util.Iterator<String> actions,
                      UpdatedActions updatedActions)
               throws CommitFailedException

API to commit the given set of actions to the table represented by logPath at the given commitVersion.

Parameters:

logStore - The log store to use for writing the commit file.

hadoopConf - The Hadoop configuration required to access the file system.

tableDescriptor - The descriptor for the table.

commitVersion - The version of the commit that is being committed.

actions - The actions that need to be committed.

updatedActions - The commit info and any metadata or protocol changes that are made as part of this commit.

Returns:

CommitResponse which contains the file status of the committed commit file. If the commit is already backfilled, then the file status could be omitted from the response and the client could retrieve the information by itself.

Throws:

CommitFailedException - if the commit operation fails.

getCommits

GetCommitsResponse getCommits(TableDescriptor tableDescriptor,
                              Long startVersion,
                              Long endVersion)

API to get the unbackfilled commits for the table represented by the given logPath. Commits older than startVersion or newer than endVersion (if given) are ignored. The returned commits are contiguous and in ascending version order. Note that the first version returned by this API may not be equal to startVersion. This happens when some versions starting from startVersion have already been backfilled and so the commit coordinator may have stopped tracking them. The returned latestTableVersion is the maximum commit version ratified by the commit coordinator. Note that returning latestTableVersion as -1 is acceptable only if the commit coordinator never ratified any version, i.e. it never accepted any unbackfilled commit.

Parameters:

tableDescriptor - The descriptor for the table.

startVersion - The minimum version of the commit that should be returned. Can be null.

endVersion - The maximum version of the commit that should be returned. Can be null.

Returns:

GetCommitsResponse which has a list of Commits and the latestTableVersion which is tracked by CommitCoordinatorClient.

backfillToVersion

void backfillToVersion(LogStore logStore,
                       org.apache.hadoop.conf.Configuration hadoopConf,
                       TableDescriptor tableDescriptor,
                       long version,
                       Long lastKnownBackfilledVersion)
                throws java.io.IOException

API to ask the commit coordinator client to backfill all commits up to version and notify the commit coordinator. If this API returns successfully, that means the backfill must have been completed, although the commit coordinator may not be aware of it yet.

Parameters:

logStore - The log store to use for writing the backfilled commits.

hadoopConf - The Hadoop configuration required to access the file system.

tableDescriptor - The descriptor for the table.

version - The version till which the commit coordinator client should backfill.

lastKnownBackfilledVersion - The last known version that was backfilled before this API was called. If it is None or invalid, then the commit coordinator client should backfill from the beginning of the table. Can be null.

Throws:

java.io.IOException - if there is an IO error while backfilling the commits.

semanticEquals

boolean semanticEquals(CommitCoordinatorClient other)

Determines whether this CommitCoordinatorClient is semantically equal to another CommitCoordinatorClient. Semantic equality is determined by each CommitCoordinatorClient implementation based on whether the two instances can be used interchangeably when invoking any of the CommitCoordinatorClient APIs, such as commit(io.delta.storage.LogStore, org.apache.hadoop.conf.Configuration, io.delta.storage.commit.TableDescriptor, long, java.util.Iterator<java.lang.String>, io.delta.storage.commit.UpdatedActions), getCommits(io.delta.storage.commit.TableDescriptor, java.lang.Long, java.lang.Long), etc. For example, both instances might be pointing to the same underlying endpoint.