Type Members

1.  sealed trait Action extends AnyRef

   Represents a single change to the state of a Delta table.

   Represents a single change to the state of a Delta table. An order sequence of actions can be replayed using InMemoryLogReplay to derive the state of the table at a given point in time.

2.  case class AddCDCFile(path: String, partitionValues: Map[String, String], size: Long, tags: Map[String, String] = null) extends FileAction with Product with Serializable

   A change file containing CDC data for the Delta version it's within.

   A change file containing CDC data for the Delta version it's within. Non-CDC readers should ignore this, CDC readers should scan all ChangeFiles in a version rather than computing changes from AddFile and RemoveFile actions.

   path is URL-encoded.

3.  case class AddFile(path: String, partitionValues: Map[String, String], size: Long, modificationTime: Long, dataChange: Boolean, stats: String = null, tags: Map[String, String] = null, deletionVector: DeletionVectorDescriptor = null, baseRowId: Option[Long] = None, defaultRowCommitVersion: Option[Long] = None, clusteringProvider: Option[String] = None) extends FileAction with HasNumRecords with Product with Serializable

   Adds a new file to the table.

   Adds a new file to the table. When multiple AddFile file actions are seen with the same path only the metadata from the last one is kept.

   path is URL-encoded.

4.  case class CheckpointMetadata(version: Long, tags: Map[String, String] = null) extends CheckpointOnlyAction with Product with Serializable

   Holds information about the Delta Checkpoint.

   Holds information about the Delta Checkpoint. This action will only be part of checkpoints.

   version

   version of the checkpoint

   tags

   attributes of the checkpoint, defaults to null (which is semantically same as an empty Map). This is kept null to ensure that the field is not present in the generated json.

5.  sealed trait CheckpointOnlyAction extends Action

   A trait to represent actions which can only be part of Checkpoint

6.  case class CommitInfo(version: Option[Long], inCommitTimestamp: Option[Long], timestamp: Timestamp, userId: Option[String], userName: Option[String], operation: String, operationParameters: Map[String, String], job: Option[JobInfo], notebook: Option[NotebookInfo], clusterId: Option[String], readVersion: Option[Long], isolationLevel: Option[String], isBlindAppend: Option[Boolean], operationMetrics: Option[Map[String, String]], userMetadata: Option[String], tags: Option[Map[String, String]], engineInfo: Option[String], txnId: Option[String]) extends Action with CommitMarker with AbstractCommitInfo with Product with Serializable

   Holds provenance information about changes to the table.

   Holds provenance information about changes to the table. This Action is not stored in the checkpoint and has reduced compatibility guarantees. Information stored in it is best effort (i.e. can be falsified by the writer).

   inCommitTimestamp

   A monotonically increasing timestamp that represents the time since epoch in milliseconds when the commit write was started. This should only be set when the feature inCommitTimestamps is enabled.

   isBlindAppend

   Whether this commit has blindly appended without caring about existing files

   engineInfo

   The information for the engine that makes the commit. If a commit is made by Delta Lake 1.1.0 or above, it will be Apache-Spark/x.y.z Delta-Lake/x.y.z.

7.  trait CommitMarker extends AnyRef

   Interface for objects that represents the information for a commit.

   Interface for objects that represents the information for a commit. Commits can be referred to using a version and timestamp. The timestamp of a commit comes from the remote storage lastModifiedTime, and can be adjusted for clock skew. Hence we have the method withTimestamp.

8.  case class DeletionVectorDescriptor(storageType: String, pathOrInlineDv: String, offset: Option[Int] = None, sizeInBytes: Int, cardinality: Long, maxRowIndex: Option[Long] = None) extends Product with Serializable

   Information about a deletion vector attached to a file action.

9.  case class DomainMetadata(domain: String, configuration: String, removed: Boolean) extends Action with Product with Serializable

   The domain metadata action contains a configuration (string-string map) for a named metadata domain.

   The domain metadata action contains a configuration (string-string map) for a named metadata domain. Two overlapping transactions conflict if they both contain a domain metadata action for the same metadata domain.

   domain: A string used to identify a specific feature. configuration: A string containing configuration options for the conflict domain. removed: If it is true it serves as a tombstone to logically delete a DomainMetadata action.

10.  sealed trait FileAction extends Action

    Actions pertaining to the addition and removal of files.

11.  case class Format(provider: String = "parquet", options: Map[String, String] = Map.empty) extends Product with Serializable
12.  trait HasNumRecords extends AnyRef

    Common trait for AddFile and RemoveFile actions providing methods for the computation of logical, physical and deleted number of records based on the statistics and the Deletion Vector of the file.

13.  class InMemoryLogReplay extends LogReplay

    Replays a history of actions, resolving them to produce the current state of the table.

    Replays a history of actions, resolving them to produce the current state of the table. The protocol for resolution is as follows:

    • The most recent AddFile and accompanying metadata for any (path, dv id) tuple wins.
    • RemoveFile deletes a corresponding AddFile and is retained as a tombstone until minFileRetentionTimestamp has passed. A RemoveFile "corresponds" to the AddFile that matches both the parquet file URI *and* the deletion vector's URI (if any).
    • The most recent version for any appId in a SetTransaction wins.
    • The most recent Metadata wins.
    • The most recent Protocol version wins.
    • For each (path, dv id) tuple, this class should always output only one FileAction (either AddFile or RemoveFile)

    This class is not thread safe.

14.  case class JobInfo(jobId: String, jobName: String, jobRunId: String, runId: String, jobOwnerId: String, triggerType: String) extends Product with Serializable
15.  class JsonMapSerializer extends JsonSerializer[Map[String, String]]

    Serializes Maps containing JSON strings without extra escaping.

16.  trait LogReplay extends AnyRef

    Replays a history of actions, resolving them to produce the current state of the table.

17.  case class Metadata(id: String = ..., name: String = null, description: String = null, format: Format = Format(), schemaString: String = null, partitionColumns: Seq[String] = Nil, configuration: Map[String, String] = Map.empty, createdTime: Option[Long] = None) extends Action with AbstractMetadata with Product with Serializable

    Updates the metadata of the table.

    Updates the metadata of the table. Only the last update to the Metadata of a table is kept. It is the responsibility of the writer to ensure that any data already present in the table is still valid after any change.

18.  case class NotebookInfo(notebookId: String) extends Product with Serializable
19.  case class ParsedStatsFields(numLogicalRecords: Option[Long], tightBounds: Option[Boolean]) extends Product with Serializable
20.  case class Protocol extends Action with AbstractProtocol with TableFeatureSupport with Product with Serializable

    Used to block older clients from reading or writing the log when backwards incompatible changes are made to the protocol.

    Used to block older clients from reading or writing the log when backwards incompatible changes are made to the protocol. Readers and writers are responsible for checking that they meet the minimum versions before performing any other operations.

    This action allows us to explicitly block older clients in the case of a breaking change to the protocol. Absent a protocol change, Clients MUST silently ignore messages and fields that they do not understand.

    Note: Please initialize this class using the companion object's apply method, which will assign correct values (Set() vs None) to readerFeatures and writerFeatures.

21.  case class RemoveFile(path: String, deletionTimestamp: Option[Long], dataChange: Boolean = true, extendedFileMetadata: Option[Boolean] = None, partitionValues: Map[String, String] = null, size: Option[Long] = None, tags: Map[String, String] = null, deletionVector: DeletionVectorDescriptor = null, baseRowId: Option[Long] = None, defaultRowCommitVersion: Option[Long] = None, stats: String = null) extends FileAction with HasNumRecords with Product with Serializable

    Logical removal of a given file from the reservoir.

    Logical removal of a given file from the reservoir. Acts as a tombstone before a file is deleted permanently.

    Note that for protocol compatibility reasons, the fields partitionValues, size, and tags are only present when the extendedFileMetadata flag is true. New writers should generally be setting this flag, but old writers (and FSCK) won't, so readers must check this flag before attempting to consume those values.

    Since old tables would not have extendedFileMetadata and size field, we should make them nullable by setting their type Option.

    path is URL-encoded.

22.  case class SetTransaction(appId: String, version: Long, lastUpdated: Option[Long]) extends Action with Product with Serializable

    Sets the committed version for a given application.

    Sets the committed version for a given application. Used to make operations like streaming append idempotent.

23.  case class SidecarFile(path: String, sizeInBytes: Long, modificationTime: Long, tags: Map[String, String] = null) extends CheckpointOnlyAction with Product with Serializable

    An Action containing the information about a sidecar file.

    An Action containing the information about a sidecar file.

    path

    - sidecar path relative to _delta_log/_sidecar directory

    sizeInBytes

    - size in bytes for the sidecar file

    modificationTime

    - modification time of the sidecar file

    tags

    - attributes of the sidecar file, defaults to null (which is semantically same as an empty Map). This is kept null to ensure that the field is not present in the generated json.

24.  case class SingleAction(txn: SetTransaction = null, add: AddFile = null, remove: RemoveFile = null, metaData: Metadata = null, protocol: Protocol = null, cdc: AddCDCFile = null, checkpointMetadata: CheckpointMetadata = null, sidecar: SidecarFile = null, domainMetadata: DomainMetadata = null, commitInfo: CommitInfo = null) extends Product with Serializable

    A serialization helper to create a common action envelope.

25.  trait TableFeatureSupport extends AnyRef

    Trait to be mixed into the Protocol case class to enable Table Features.

    Trait to be mixed into the Protocol case class to enable Table Features.

    Protocol reader version 3 and writer version 7 start to support reader and writer table features. Reader version 3 supports only reader-writer features in an explicit way, by adding its name to readerFeatures. Similarly, writer version 7 supports only writer-only or reader-writer features in an explicit way, by adding its name to writerFeatures. When reading or writing a table, clients MUST respect all supported features.

    See also the document of TableFeature for feature-specific terminologies.