Packages

c

io.delta.dynamodbcommitcoordinator

DynamoDBCommitCoordinatorClient

class DynamoDBCommitCoordinatorClient extends CommitCoordinatorClient

A commit coordinator client that uses DynamoDB as the commit coordinator. The table schema is as follows: tableId: String --- The unique identifier for the table. This is a UUID. path: String --- The fully qualified path of the table in the file system. e.g. s3://bucket/path. acceptingCommits: Boolean --- Whether the commit coordinator is accepting new commits. This will only be set to false when the table is converted from coordinated commits to file system commits. tableVersion: Number --- The version of the latest commit. tableTimestamp: Number --- The inCommitTimestamp of the latest commit. schemaVersion: Number --- The version of the schema used to store the data. hasAcceptedCommits: Boolean --- Whether any actual commits have been accepted by this commit coordinator after registerTable. commits: --- The list of unbackfilled commits. version: Number --- The version of the commit. inCommitTimestamp: Number --- The inCommitTimestamp of the commit. fsName: String --- The name of the unbackfilled file. fsLength: Number --- The length of the unbackfilled file. fsTimestamp: Number --- The modification time of the unbackfilled file.

Linear Supertypes
CommitCoordinatorClient, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DynamoDBCommitCoordinatorClient
  2. CommitCoordinatorClient
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new DynamoDBCommitCoordinatorClient(coordinatedCommitsTableName: String, endpoint: String, client: AmazonDynamoDB, backfillBatchSize: Long, readCapacityUnits: Long, writeCapacityUnits: Long, skipPathCheck: Boolean)
  2. new DynamoDBCommitCoordinatorClient(coordinatedCommitsTableName: String, endpoint: String, client: AmazonDynamoDB, backfillBatchSize: Long)

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def backfillToVersion(logStore: LogStore, hadoopConf: Configuration, tableDesc: TableDescriptor, version: Long, lastKnownBackfilledVersion: Long): Unit

    Backfills all the unbackfilled commits returned by the commit coordinator and notifies the commit owner of the backfills.

    Backfills all the unbackfilled commits returned by the commit coordinator and notifies the commit owner of the backfills. The version parameter is ignored in this implementation and all the unbackfilled commits are backfilled. This method will not throw any exception if the physical backfill succeeds but the update to the commit coordinator fails.

    Definition Classes
    DynamoDBCommitCoordinatorClient → CommitCoordinatorClient
    Annotations
    @Override()
    Exceptions thrown

    IllegalArgumentException if the requested backfill version is greater than the latest version for the table.

  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  7. def commit(logStore: LogStore, hadoopConf: Configuration, tableDesc: TableDescriptor, commitVersion: Long, actions: Iterator[String], updatedActions: UpdatedActions): CommitResponse
    Definition Classes
    DynamoDBCommitCoordinatorClient → CommitCoordinatorClient
    Annotations
    @Override()
  8. def commitToCoordinator(logPath: Path, coordinatedCommitsTableConf: Map[String, String], attemptVersion: Long, commitFile: FileStatus, inCommitTimestamp: Long, isCCtoFSConversion: Boolean): CommitResponse

    Commits the given file to the commit coordinator.

    Commits the given file to the commit coordinator. A conditional write is performed to the DynamoDB table entry associated with this Delta table. If the conditional write goes through, the filestatus of the UUID delta file will be appended to the list of unbackfilled commits, and other updates like setting the latest table version to attemptVersion will be performed.

    For the conditional write to go through, the following conditions must be met right before the write is performed: 1. The latest table version in DynamoDB is equal to attemptVersion - 1. 2. The commit coordinator is accepting new commits. 3. The schema version of the commit coordinator matches the schema version of the client. 4. The table path stored in DynamoDB matches the path of the table. This check is skipped if skipPathCheck is set to true. If the conditional write fails, we retrieve the current entry in DynamoDB to figure out which condition failed. (DynamoDB does not tell us which condition failed in the rejection.) If any of (2), (3), or (4) fail, an unretryable CommitFailedException will be thrown. For (1): If the retrieved latest table version is greater than or equal to attemptVersion, a retryable CommitFailedException will be thrown. If the retrieved latest table version is less than attemptVersion - 1, an unretryable CommitFailedException will be thrown.

    Attributes
    protected[dynamodbcommitcoordinator]
  9. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  10. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  11. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  12. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  13. def getCommits(tableDesc: TableDescriptor, startVersion: Long, endVersion: Long): GetCommitsResponse
    Definition Classes
    DynamoDBCommitCoordinatorClient → CommitCoordinatorClient
    Annotations
    @Override()
  14. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  15. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  16. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  17. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  18. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  19. def registerTable(logPath: Path, tableIdentifier: Optional[TableIdentifier], currentVersion: Long, currentMetadata: AbstractMetadata, currentProtocol: AbstractProtocol): Map[String, String]
    Definition Classes
    DynamoDBCommitCoordinatorClient → CommitCoordinatorClient
    Annotations
    @Override()
  20. def semanticEquals(other: CommitCoordinatorClient): Boolean
    Definition Classes
    DynamoDBCommitCoordinatorClient → CommitCoordinatorClient
    Annotations
    @Override()
  21. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  22. def toString(): String
    Definition Classes
    AnyRef → Any
  23. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  24. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  25. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()

Inherited from CommitCoordinatorClient

Inherited from AnyRef

Inherited from Any

Ungrouped