package hooks
- Alphabetic
- Public
- All
Type Members
-
trait
AutoCompactBase extends PostCommitHook with DeltaLogging
A trait for post commit hook which compacts files in a Delta table.
A trait for post commit hook which compacts files in a Delta table. This hook acts as a cheaper version of the OPTIMIZE command, by attempting to compact small files together into fewer bigger files.
Auto Compact chooses files to compact greedily by looking at partition directories which have the largest number of files that are under a certain size threshold and launches a bounded number of optimize tasks based on the capacity of the cluster.
-
case class
AutoCompactRequest(shouldCompact: Boolean, allowedPartitions: PartitionKeySet, targetPartitionsPredicate: Seq[Expression] = Nil) extends Product with Serializable
The request class that contains all information needed for Auto Compaction.
The request class that contains all information needed for Auto Compaction.
- shouldCompact
True if Auto Compact should start.
- targetPartitionsPredicate
The predicate of the target partitions of this Auto Compact request.
-
sealed
trait
AutoCompactType extends AnyRef
A trait describing the type of Auto Compaction.
- trait GenerateSymlinkManifestImpl extends PostCommitHook with DeltaLogging with Serializable
-
trait
PostCommitHook extends AnyRef
A hook which can be executed after a transaction.
A hook which can be executed after a transaction. These hooks are registered to a OptimisticTransaction, and are executed after a *successful* commit takes place.
-
case class
UpdateCatalog(table: CatalogTable) extends UpdateCatalogBase with Product with Serializable
A post-commit hook that allows us to cache the most recent schema and table properties of a Delta table in an External Catalog.
A post-commit hook that allows us to cache the most recent schema and table properties of a Delta table in an External Catalog. In addition to the schema and table properties, we also store the last commit timestamp and version for which we updated the catalog. This prevents us from updating the MetaStore with potentially stale information.
-
trait
UpdateCatalogBase extends PostCommitHook with DeltaLogging
Base trait for post commit hooks that want to update the catalog with the latest table schema and properties.
Value Members
-
object
AutoCompact extends AutoCompactBase with Product with Serializable
Post commit hook for Auto Compaction.
-
object
AutoCompactPartitionReserve
Thread-safe singleton to keep track of partitions reserved for auto-compaction.
- object AutoCompactRequest extends Serializable
- object AutoCompactType
- object AutoCompactUtils extends DeltaLogging
-
object
CheckpointHook extends PostCommitHook
Write a new checkpoint at the version committed by the txn if required.
-
object
ChecksumHook extends PostCommitHook with DeltaLogging
Write a new checksum at the version committed by the txn if possible.
-
object
GenerateSymlinkManifest extends GenerateSymlinkManifestImpl
Post commit hook to generate hive-style manifests for Delta table.
Post commit hook to generate hive-style manifests for Delta table. This is useful for compatibility with Presto / Athena.
- object GenerateSymlinkManifestUtils extends DeltaLogging
-
object
HudiConverterHook extends PostCommitHook with DeltaLogging
Write a new Hudi commit for the version committed by the txn, if required.
-
object
IcebergConverterHook extends PostCommitHook with DeltaLogging
Write a new Iceberg metadata file at the version committed by the txn, if required.
- object UpdateCatalog extends Serializable
-
object
UpdateCatalogFactory
Factory object to create an UpdateCatalog post commit hook.
Factory object to create an UpdateCatalog post commit hook. This should always be used instead of directly creating a specific hook.