sealed trait RemovableFeature extends AnyRef
A trait indicating a feature can be removed. Classes that extend the trait need to implement the following four functions:
a) preDowngradeCommand. This is where all required actions for removing the feature are implemented. For example, to remove the DVs feature we need to remove metadata config and purge all DVs from table. This action takes place before the protocol downgrade in separate commit(s). Note, the command needs to be implemented in a way concurrent transactions do not nullify the effect. For example, disabling DVs on a table before purging will stop concurrent transactions from adding DVs. During protocol downgrade we perform a validation in validateRemoval to make sure all invariants still hold.
b) validateRemoval. Add any feature-specific checks before proceeding to the protocol downgrade. This function is guaranteed to be called at the latest version before the protocol downgrade is committed to the table. When the protocol downgrade txn conflicts, the validation is repeated against the winning txn snapshot. As soon as the protocol downgrade succeeds, all subsequent interleaved txns are aborted. The implementation should return true if there are no feature traces in the latest version. False otherwise.
c) requiresHistoryProtection. It indicates whether the feature leaves traces in the table history that may result in incorrect behaviour if the table is read/written by a client that does not support the feature. This is by default true for all reader+writer features and false for writer features. WARNING: Disabling requiresHistoryProtection for relevant features could result in incorrect snapshot reconstruction.
d) actionUsesFeature. For features that require history truncation we verify whether past versions contain any traces of the removed feature. This is achieved by calling actionUsesFeature for every action of every reachable commit version in the log. Note, a feature may leave traces in both data and metadata. Depending on the feature, we need to check several types of actions such as Metadata, AddFile, RemoveFile etc.
WARNING: actionUsesFeature should not check Protocol actions for the feature being removed, because at the time actionUsesFeature is invoked the protocol downgrade did not happen yet. Thus, the feature-to-remove is still active. As a result, any unrelated operations that produce a protocol action (while we are waiting for the retention period to expire) will "carry" the feature-to-remove. Checking protocol for that feature would result in an unnecessary failure during the history validation of the next DROP FEATURE call. Note, while the feature-to-remove is supported in the protocol we cannot generate a legit protocol action that adds support for that feature since it is already supported.
- Self Type
- RemovableFeature with TableFeature
- Alphabetic
- By Inheritance
- RemovableFeature
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Abstract Value Members
- abstract def actionUsesFeature(action: Action): Boolean
- abstract def preDowngradeCommand(table: DeltaTableV2): PreDowngradeTableFeatureCommand
- abstract def validateRemoval(snapshot: Snapshot): Boolean
Concrete Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
historyContainsFeature(spark: SparkSession, downgradeTxnReadSnapshot: Snapshot): Boolean
Examines all historical commits for traces of the removableFeature.
Examines all historical commits for traces of the removableFeature. This is achieved as follows:
1) We find the earliest valid checkpoint, recreate a snapshot at that version and we check whether there any traces of the feature-to-remove. 2) We check all commits that exist between version 0 and the current version. This includes the versions we validated the snapshots. This is because a commit might include information that is not available in the snapshot. Examples include CommitInfo, CDCInfo etc. Note, there can still be valid log commit files with versions prior the earliest checkpoint version. 3) We do not need to recreate a snapshot at the current version because this is already being handled by validateRemoval.
Note, this is a slow process.
- spark
The SparkSession.
- downgradeTxnReadSnapshot
The read snapshot of the protocol downgrade transaction.
- returns
True if the history contains any trace of the feature.
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- def requiresHistoryProtection: Boolean
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()