Packages

p

org.apache.spark.sql.delta

managedcommit

package managedcommit

Ordering
  1. Alphabetic
Visibility
  1. Public
  2. All

Type Members

  1. trait AbstractBatchBackfillingCommitOwnerClient extends CommitOwnerClient with Logging

    An abstract CommitOwnerClient which triggers backfills every n commits.

    An abstract CommitOwnerClient which triggers backfills every n commits. - every commit version which satisfies commitVersion % batchSize == 0 will trigger a backfill.

  2. case class Commit(version: Long, fileStatus: FileStatus, commitTimestamp: Long) extends Product with Serializable

    Representation of a commit file

  3. case class CommitFailedException(retryable: Boolean, conflict: Boolean, message: String) extends Exception with Product with Serializable

    Exception raised by CommitOwnerClient.commit method.

    Exception raised by CommitOwnerClient.commit method. | retryable | conflict | meaning | | no | no | something bad happened (e.g. auth failure) | | no | yes | permanent transaction conflict (e.g. multi-table commit failed) | | yes | no | transient error (e.g. network hiccup) | | yes | yes | physical conflict (allowed to rebase and retry) |

  4. trait CommitOwnerBuilder extends AnyRef

    A builder interface for CommitOwnerClient

  5. trait CommitOwnerClient extends AnyRef

    CommitOwnerClient is responsible for managing commits for a managed-commit delta table.

    CommitOwnerClient is responsible for managing commits for a managed-commit delta table. 1. It provides API to commit a new version of the table. See CommitOwnerClient.commit API. 2. It makes sure that commits are backfilled if/when needed 3. It also tracks and returns unbackfilled commits. See CommitOwnerClient.getCommits API.

  6. case class CommitResponse(commit: Commit) extends Product with Serializable

    Response container for CommitOwnerClient.commit API

  7. case class GetCommitsResponse(commits: Seq[Commit], latestTableVersion: Long) extends Product with Serializable

    Response container for CommitOwnerClient.getCommits API

  8. class InMemoryCommitOwner extends AbstractBatchBackfillingCommitOwnerClient
  9. case class InMemoryCommitOwnerBuilder(batchSize: Long) extends CommitOwnerBuilder with Product with Serializable

    The InMemoryCommitOwnerBuilder class is responsible for creating singleton instances of InMemoryCommitOwner with the specified batchSize.

  10. case class TableCommitOwnerClient(commitOwnerClient: CommitOwnerClient, logPath: Path, tableConf: Map[String, String], hadoopConf: Configuration, logStore: LogStore) extends Product with Serializable

    A wrapper around CommitOwnerClient that provides a more user-friendly API for committing / accessing commits to a specific table.

    A wrapper around CommitOwnerClient that provides a more user-friendly API for committing / accessing commits to a specific table. This class takes care of passing the table specific configuration to the underlying CommitOwnerClient e.g. logPath / logStore / managedCommitTableConf / hadoopConf.

    commitOwnerClient

    the underlying CommitOwnerClient

    logPath

    the path to the log directory

    tableConf

    the table specific managed-commit configuration

    hadoopConf

    hadoop configuration

    logStore

    the log store

  11. case class UpdatedActions(commitInfo: CommitInfo, newMetadata: Metadata, newProtocol: Protocol, oldMetadata: Metadata, oldProtocol: Protocol) extends Product with Serializable

    A container class to inform the CommitOwnerClient about any changes in Protocol/Metadata

Value Members

  1. object CommitOwner
  2. object CommitOwnerClient
  3. object CommitOwnerProvider

    Factory to get the correct CommitOwnerClient for a table

  4. object ManagedCommitUtils
  5. object TableCommitOwnerClient extends Serializable

Ungrouped