object CheckpointProtectionTableFeature extends WriterFeature with RemovableFeature
Writer feature that enforces writers to cleanup metadata iff metadata can be cleaned up to requireCheckpointProtectionBeforeVersion in one go. This means that a single cleanup operation should truncate up to requireCheckpointProtectionBeforeVersion as opposed to several cleanup operations truncating in chunks.
The are two exceptions to this rule. If any of the two holds, the rule above can be ignored:
a) The writer verifies it supports all protocols between [start, min(requireCheckpointProtectionBeforeVersion, targetCleanupVersion)] versions it intends to truncate. b) The writer does not create any checkpoints during history cleanup and does not erase any checkpoints after the truncation version.
The CheckpointProtectionTableFeature can only be removed if history is truncated up to at least requireCheckpointProtectionBeforeVersion.
- Alphabetic
- By Inheritance
- CheckpointProtectionTableFeature
- RemovableFeature
- WriterFeature
- TableFeature
- Serializable
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
actionUsesFeature(action: Action): Boolean
The feature uses the
requireCheckpointProtectionBeforeVersionproperty.The feature uses the
requireCheckpointProtectionBeforeVersionproperty. This is removed when dropping the feature but we allow it to exist in the history. This is to allow history truncation at the boundary of requireCheckpointProtectionBeforeVersion rather than the last 24 hours. Otherwise, dropping the feature would always require 24 hour waiting time.- Definition Classes
- CheckpointProtectionTableFeature → RemovableFeature
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
- def getCheckpointProtectionVersion(snapshot: Snapshot): Long
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
historyContainsFeature(spark: SparkSession, downgradeTxnReadSnapshot: Snapshot): Boolean
Examines all historical commits for traces of the removableFeature.
Examines all historical commits for traces of the removableFeature. This is achieved as follows:
1) We find the earliest valid checkpoint, recreate a snapshot at that version and we check whether there any traces of the feature-to-remove. 2) We check all commits that exist between version 0 and the current version. This includes the versions we validated the snapshots. This is because a commit might include information that is not available in the snapshot. Examples include CommitInfo, CDCInfo etc. Note, there can still be valid log commit files with versions prior the earliest checkpoint version. 3) We do not need to recreate a snapshot at the current version because this is already being handled by validateRemoval.
Note, this is a slow process.
- spark
The SparkSession.
- downgradeTxnReadSnapshot
The read snapshot of the protocol downgrade transaction.
- returns
True if the history contains any trace of the feature.
- Definition Classes
- RemovableFeature
- def historyPriorToCheckpointProtectionVersionIsTruncated(snapshot: Snapshot): Boolean
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
def
isLegacyFeature: Boolean
Determine if this feature is a legacy feature.
Determine if this feature is a legacy feature. See the documentation of TableFeature for more information.
- Definition Classes
- TableFeature
-
def
isReaderWriterFeature: Boolean
Determine if this feature applies to both readers and writers.
Determine if this feature applies to both readers and writers.
- Definition Classes
- TableFeature
-
def
isRemovable: Boolean
True if this feature can be removed.
True if this feature can be removed.
- Definition Classes
- TableFeature
- def metadataWithCheckpointProtection(metadata: Metadata, version: Long): Metadata
-
def
minProtocolVersion: Protocol
Get a Protocol object stating the minimum reader and writer versions this feature requires.
Get a Protocol object stating the minimum reader and writer versions this feature requires. For a feature that can only be explicitly supported, this method returns a protocol version that supports table features, either
(0,7)or(3,7)depending on the feature is writer-only or reader-writer. For a legacy feature that can be implicitly supported, this method returns the first protocol version which introduced the said feature.For all features, if the table's protocol version does not support table features, then the minimum protocol version is enough. However, if the protocol version supports table features for the feature type (writer-only or reader-writer), then the minimum protocol version is not enough to support a feature. In this case the feature must also be explicitly listed in the appropriate feature sets in the Protocol.
- Definition Classes
- TableFeature
-
val
minReaderVersion: Int
- Definition Classes
- TableFeature
-
val
minWriterVersion: Int
- Definition Classes
- TableFeature
-
val
name: String
- Definition Classes
- TableFeature
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
def
preDowngradeCommand(table: DeltaTableV2): PreDowngradeTableFeatureCommand
- Definition Classes
- CheckpointProtectionTableFeature → RemovableFeature
-
def
requiredFeatures: Set[TableFeature]
Set of table features that this table feature depends on.
Set of table features that this table feature depends on. I.e. the set of features that need to be enabled if this table feature is enabled.
- Definition Classes
- TableFeature
-
def
requiresHistoryProtection: Boolean
This is a special feature in the sense that it requires history truncation but implements it as part of its downgrade process.
This is a special feature in the sense that it requires history truncation but implements it as part of its downgrade process. This is implemented like this for 2 reasons:
- It allows us to remove the feature table property after the clean up in the preDowngrade command is successful. 2. It does not require to scan the history for features traces as long as all history before requireCheckpointProtectionBeforeVersion is truncated.
- Definition Classes
- CheckpointProtectionTableFeature → RemovableFeature
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
def
validateRemoval(snapshot: Snapshot): Boolean
Returns true if table property is absent.
Returns true if table property is absent.
- Definition Classes
- CheckpointProtectionTableFeature → RemovableFeature
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()