io.delta.kernel.internal.replay

Class LogReplay

Object

io.delta.kernel.internal.replay.LogReplay

public class LogReplay
extends Object

Replays a history of actions, resolving them to produce the current state of the table. The protocol for resolution is as follows: - The most recent AddFile and accompanying metadata for any `(path, dv id)` tuple wins. - RemoveFile deletes a corresponding AddFile. A RemoveFile "corresponds" to the AddFile that matches both the parquet file URI *and* the deletion vector's URI (if any). - The most recent Metadata wins. - The most recent Protocol version wins. - For each `(path, dv id)` tuple, this class should always output only one FileAction (either AddFile or RemoveFile) This class exposes the following public APIs - getProtocol(): latest non-null Protocol - getMetadata(): latest non-null Metadata - getAddFilesAsColumnarBatches(boolean): return all active (not tombstoned) AddFiles as ColumnarBatchs

Field Summary

Fields

Modifier and Type	Field and Description
static int	ADD_FILE_DV_ORDINAL
static int	ADD_FILE_ORDINAL
static int	ADD_FILE_PATH_ORDINAL
static StructType	PROTOCOL_METADATA_READ_SCHEMA Read schema when searching for the latest Protocol and Metadata.
static int	REMOVE_FILE_DV_ORDINAL
static int	REMOVE_FILE_ORDINAL
static int	REMOVE_FILE_PATH_ORDINAL
static StructType	SET_TRANSACTION_READ_SCHEMA Read schema when searching for just the transaction identifiers

Constructor Summary

Constructors

Constructor and Description
LogReplay(Path logPath, Path dataPath, long snapshotVersion, TableClient tableClient, LogSegment logSegment, java.util.Optional<SnapshotHint> snapshotHint)

Method Summary

Modifier and Type	Method and Description
CloseableIterator<FilteredColumnarBatch>	getAddFilesAsColumnarBatches(boolean shouldReadStats) Returns an iterator of FilteredColumnarBatch representing all the active AddFiles in the table.
static StructType	getAddRemoveReadSchema(boolean shouldReadStats) Read schema when searching for all the active AddFiles
java.util.Optional<Long>	getLatestTransactionIdentifier(String applicationId)
Metadata	getMetadata()
Protocol	getProtocol()

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PROTOCOL_METADATA_READ_SCHEMA

public static final StructType PROTOCOL_METADATA_READ_SCHEMA

Read schema when searching for the latest Protocol and Metadata.

SET_TRANSACTION_READ_SCHEMA

public static final StructType SET_TRANSACTION_READ_SCHEMA

Read schema when searching for just the transaction identifiers

ADD_FILE_ORDINAL

public static int ADD_FILE_ORDINAL

ADD_FILE_PATH_ORDINAL

public static int ADD_FILE_PATH_ORDINAL

ADD_FILE_DV_ORDINAL

public static int ADD_FILE_DV_ORDINAL

REMOVE_FILE_ORDINAL

public static int REMOVE_FILE_ORDINAL

REMOVE_FILE_PATH_ORDINAL

public static int REMOVE_FILE_PATH_ORDINAL

REMOVE_FILE_DV_ORDINAL

public static int REMOVE_FILE_DV_ORDINAL

Constructor Detail

LogReplay

public LogReplay(Path logPath,
                 Path dataPath,
                 long snapshotVersion,
                 TableClient tableClient,
                 LogSegment logSegment,
                 java.util.Optional<SnapshotHint> snapshotHint)

Method Detail

getAddRemoveReadSchema

public static StructType getAddRemoveReadSchema(boolean shouldReadStats)

Read schema when searching for all the active AddFiles

getProtocol

public Protocol getProtocol()

getMetadata

public Metadata getMetadata()

getLatestTransactionIdentifier

public java.util.Optional<Long> getLatestTransactionIdentifier(String applicationId)

getAddFilesAsColumnarBatches

public CloseableIterator<FilteredColumnarBatch> getAddFilesAsColumnarBatches(boolean shouldReadStats)

Returns an iterator of FilteredColumnarBatch representing all the active AddFiles in the table.

Statistics are conditionally read for the AddFiles based on shouldReadStats. The returned batches have schema:

1. name: add

   type: AddFile.SCHEMA_WITH_STATS if shouldReadStats=true, otherwise AddFile.SCHEMA_WITHOUT_STATS