package hooks
- Alphabetic
- Public
- Protected
Type Members
- trait AutoCompactBase extends PostCommitHook with DeltaLogging
A trait for post commit hook which compacts files in a Delta table.
A trait for post commit hook which compacts files in a Delta table. This hook acts as a cheaper version of the OPTIMIZE command, by attempting to compact small files together into fewer bigger files.
Auto Compact chooses files to compact greedily by looking at partition directories which have the largest number of files that are under a certain size threshold and launches a bounded number of optimize tasks based on the capacity of the cluster.
- case class AutoCompactRequest(shouldCompact: Boolean, allowedPartitions: PartitionKeySet, targetPartitionsPredicate: Seq[Expression] = Nil) extends Product with Serializable
The request class that contains all information needed for Auto Compaction.
The request class that contains all information needed for Auto Compaction.
- shouldCompact
True if Auto Compact should start.
- targetPartitionsPredicate
The predicate of the target partitions of this Auto Compact request.
- sealed trait AutoCompactType extends AnyRef
A trait describing the type of Auto Compaction.
- trait GenerateSymlinkManifestImpl extends PostCommitHook with DeltaLogging with Serializable
- trait PostCommitHook extends AnyRef
A hook which can be executed after a transaction.
A hook which can be executed after a transaction. These hooks are registered to a OptimisticTransaction, and are executed after a *successful* commit takes place.
- case class UpdateCatalog(table: CatalogTable) extends UpdateCatalogBase with Product with Serializable
A post-commit hook that allows us to cache the most recent schema and table properties of a Delta table in an External Catalog.
A post-commit hook that allows us to cache the most recent schema and table properties of a Delta table in an External Catalog. In addition to the schema and table properties, we also store the last commit timestamp and version for which we updated the catalog. This prevents us from updating the MetaStore with potentially stale information.
- trait UpdateCatalogBase extends PostCommitHook with DeltaLogging
Base trait for post commit hooks that want to update the catalog with the latest table schema and properties.
Value Members
- case object AutoCompact extends AutoCompactBase with Product with Serializable
Post commit hook for Auto Compaction.
- object AutoCompactPartitionReserve
Thread-safe singleton to keep track of partitions reserved for auto-compaction.
- object AutoCompactRequest extends Serializable
- object AutoCompactType
- object AutoCompactUtils extends DeltaLogging
- object CheckpointHook extends PostCommitHook
Write a new checkpoint at the version committed by the txn if required.
- object ChecksumHook extends PostCommitHook with DeltaLogging
Write a new checksum at the version committed by the txn if possible.
- object GenerateSymlinkManifest extends GenerateSymlinkManifestImpl
Post commit hook to generate hive-style manifests for Delta table.
Post commit hook to generate hive-style manifests for Delta table. This is useful for compatibility with Presto / Athena.
- object GenerateSymlinkManifestUtils extends DeltaLogging
- object HudiConverterHook extends PostCommitHook with DeltaLogging
Write a new Hudi commit for the version committed by the txn, if required.
- object IcebergConverterHook extends PostCommitHook with DeltaLogging
Write a new Iceberg metadata file at the version committed by the txn, if required.
- object UpdateCatalog extends Serializable
- object UpdateCatalogFactory
Factory object to create an UpdateCatalog post commit hook.
Factory object to create an UpdateCatalog post commit hook. This should always be used instead of directly creating a specific hook.