Packages

trait SnapshotManagement extends AnyRef

Manages the creation, computation, and access of Snapshot's for Delta tables. Responsibilities include:

  • Figuring out the set of files that are required to compute a specific version of a table
  • Updating and exposing the latest snapshot of the Delta table in a thread-safe manner
Self Type
DeltaLog
Linear Supertypes
AnyRef, Any
Known Subclasses
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SnapshotManagement
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. def createSnapshot(segment: LogSegment, minFileRetentionTimestamp: Long, timestamp: Long): Snapshot
    Attributes
    protected
  7. val currentSnapshot: Snapshot
    Attributes
    protected
    Annotations
    @volatile()
  8. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  9. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  10. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  11. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. def getLogSegmentForVersion(startCheckpoint: Option[Long], versionToLoad: Option[Long] = None): LogSegment

    Get a list of files that can be used to compute a Snapshot at version versionToLoad, If versionToLoad is not provided, will generate the list of files that are needed to load the latest version of the Delta table.

    Get a list of files that can be used to compute a Snapshot at version versionToLoad, If versionToLoad is not provided, will generate the list of files that are needed to load the latest version of the Delta table. This method also performs checks to ensure that the delta files are contiguous.

    startCheckpoint

    A potential start version to perform the listing of the DeltaLog, typically that of a known checkpoint. If this version's not provided, we will start listing from version 0.

    versionToLoad

    A specific version to load. Typically used with time travel and the Delta streaming source. If not provided, we will try to load the latest version of the table.

    returns

    Some LogSegment to build a Snapshot if files do exist after the given startCheckpoint. None, if there are no new files after startCheckpoint.

    Attributes
    protected
  13. def getLogSegmentFrom(startingCheckpoint: Option[CheckpointMetaData]): LogSegment

    Get the LogSegment that will help in computing the Snapshot of the table at DeltaLog initialization.

    Get the LogSegment that will help in computing the Snapshot of the table at DeltaLog initialization.

    startingCheckpoint

    A checkpoint that we can start our listing from

    Attributes
    protected
  14. def getSnapshotAt(version: Long, commitTimestamp: Option[Long] = None, lastCheckpointHint: Option[CheckpointInstance] = None): Snapshot

    Get the snapshot at version.

  15. def getSnapshotAtInit: Snapshot

    Load the Snapshot for this Delta table at initialization.

    Load the Snapshot for this Delta table at initialization. This method uses the lastCheckpoint file as a hint on where to start listing the transaction log directory. If the _delta_log directory doesn't exist, this method will return an InitialSnapshot.

    Attributes
    protected
  16. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  17. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  18. val lastUpdateTimestamp: Long

    The timestamp when the last successful update action is finished.

    The timestamp when the last successful update action is finished.

    Attributes
    protected
    Annotations
    @volatile()
  19. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  20. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  21. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  22. def snapshot: Snapshot

    Returns the current snapshot.

    Returns the current snapshot. Note this does not automatically update().

  23. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  24. def toString(): String
    Definition Classes
    AnyRef → Any
  25. def update(stalenessAcceptable: Boolean = false): Snapshot

    Update ActionLog by applying the new delta files if any.

    Update ActionLog by applying the new delta files if any.

    stalenessAcceptable

    Whether we can accept working with a stale version of the table. If the table has surpassed our staleness tolerance, we will update to the latest state of the table synchronously. If staleness is acceptable, and the table hasn't passed the staleness tolerance, we will kick off a job in the background to update the table state, and can return a stale snapshot in the meantime.

  26. def updateInternal(isAsync: Boolean): Snapshot

    Queries the store for new delta files and applies them to the current state.

    Queries the store for new delta files and applies them to the current state. Note: the caller should hold deltaLogLock before calling this method.

    Attributes
    protected
  27. def verifyDeltaVersions(versions: Array[Long]): Unit

    Verify the versions are contiguous.

    Verify the versions are contiguous.

    Attributes
    protected
  28. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  29. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  30. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from AnyRef

Inherited from Any

Ungrouped