object CheckpointProtectionTableFeature extends WriterFeature with RemovableFeature
Writer feature that enforces writers to cleanup metadata iff metadata can be cleaned up to requireCheckpointProtectionBeforeVersion in one go. This means that a single cleanup operation should truncate up to requireCheckpointProtectionBeforeVersion as opposed to several cleanup operations truncating in chunks.
The are two exceptions to this rule. If any of the two holds, the rule above can be ignored:
a) The writer verifies it supports all protocols between [start, min(requireCheckpointProtectionBeforeVersion, targetCleanupVersion)] versions it intends to truncate. b) The writer does not create any checkpoints during history cleanup and does not erase any checkpoints after the truncation version.
The CheckpointProtectionTableFeature can only be removed if history is truncated up to at least requireCheckpointProtectionBeforeVersion.
- Alphabetic
- By Inheritance
- CheckpointProtectionTableFeature
- RemovableFeature
- WriterFeature
- TableFeature
- Serializable
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- def actionUsesFeature(action: Action): Boolean
The feature uses the
requireCheckpointProtectionBeforeVersionproperty.The feature uses the
requireCheckpointProtectionBeforeVersionproperty. This is removed when dropping the feature but we allow it to exist in the history. This is to allow history truncation at the boundary of requireCheckpointProtectionBeforeVersion rather than the last 24 hours. Otherwise, dropping the feature would always require 24 hour waiting time.- Definition Classes
- CheckpointProtectionTableFeature → RemovableFeature
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @native()
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable])
- def getCheckpointProtectionVersion(snapshot: Snapshot): Long
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def historyContainsFeature(spark: SparkSession, downgradeTxnReadSnapshot: Snapshot): Boolean
Examines all historical commits for traces of the removableFeature.
Examines all historical commits for traces of the removableFeature. This is achieved as follows:
1) We find the earliest valid checkpoint, recreate a snapshot at that version and we check whether there any traces of the feature-to-remove. 2) We check all commits that exist between version 0 and the current version. This includes the versions we validated the snapshots. This is because a commit might include information that is not available in the snapshot. Examples include CommitInfo, CDCInfo etc. Note, there can still be valid log commit files with versions prior the earliest checkpoint version. 3) We do not need to recreate a snapshot at the current version because this is already being handled by validateRemoval.
Note, this is a slow process.
- spark
The SparkSession.
- downgradeTxnReadSnapshot
The read snapshot of the protocol downgrade transaction.
- returns
True if the history contains any trace of the feature.
- Definition Classes
- RemovableFeature
- def historyPriorToCheckpointProtectionVersionIsTruncated(snapshot: Snapshot): Boolean
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- def isLegacyFeature: Boolean
Determine if this feature is a legacy feature.
Determine if this feature is a legacy feature. See the documentation of TableFeature for more information.
- Definition Classes
- TableFeature
- def isReaderWriterFeature: Boolean
Determine if this feature applies to both readers and writers.
Determine if this feature applies to both readers and writers.
- Definition Classes
- TableFeature
- def isRemovable: Boolean
True if this feature can be removed.
True if this feature can be removed.
- Definition Classes
- TableFeature
- def metadataWithCheckpointProtection(metadata: Metadata, version: Long): Metadata
- def minProtocolVersion: Protocol
Get a Protocol object stating the minimum reader and writer versions this feature requires.
Get a Protocol object stating the minimum reader and writer versions this feature requires. For a feature that can only be explicitly supported, this method returns a protocol version that supports table features, either
(0,7)or(3,7)depending on the feature is writer-only or reader-writer. For a legacy feature that can be implicitly supported, this method returns the first protocol version which introduced the said feature.For all features, if the table's protocol version does not support table features, then the minimum protocol version is enough. However, if the protocol version supports table features for the feature type (writer-only or reader-writer), then the minimum protocol version is not enough to support a feature. In this case the feature must also be explicitly listed in the appropriate feature sets in the Protocol.
- Definition Classes
- TableFeature
- val minReaderVersion: Int
- Definition Classes
- TableFeature
- val minWriterVersion: Int
- Definition Classes
- TableFeature
- val name: String
- Definition Classes
- TableFeature
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- def preDowngradeCommand(table: DeltaTableV2): PreDowngradeTableFeatureCommand
- Definition Classes
- CheckpointProtectionTableFeature → RemovableFeature
- def requiredFeatures: Set[TableFeature]
Set of table features that this table feature depends on.
Set of table features that this table feature depends on. I.e. the set of features that need to be enabled if this table feature is enabled.
- Definition Classes
- TableFeature
- def requiresHistoryProtection: Boolean
This is a special feature in the sense that it requires history truncation but implements it as part of its downgrade process.
This is a special feature in the sense that it requires history truncation but implements it as part of its downgrade process. This is implemented like this for 2 reasons:
- It allows us to remove the feature table property after the clean up in the preDowngrade command is successful. 2. It does not require to scan the history for features traces as long as all history before requireCheckpointProtectionBeforeVersion is truncated.
- Definition Classes
- CheckpointProtectionTableFeature → RemovableFeature
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- def validateRemoval(snapshot: Snapshot): Boolean
Returns true if table property is absent.
Returns true if table property is absent.
- Definition Classes
- CheckpointProtectionTableFeature → RemovableFeature
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()