io.delta.kernel.internal.snapshot

Class MetadataCleanup

Object

io.delta.kernel.internal.snapshot.MetadataCleanup

public class MetadataCleanup
extends Object

Method Summary

Modifier and Type	Method and Description
static long	cleanupExpiredLogs(Engine engine, Clock clock, Path tablePath, long retentionMillis) Delete the Delta log files (delta and checkpoint files) that are expired according to the table metadata retention settings.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

cleanupExpiredLogs

public static long cleanupExpiredLogs(Engine engine,
                                      Clock clock,
                                      Path tablePath,
                                      long retentionMillis)
                               throws java.io.IOException

Delete the Delta log files (delta and checkpoint files) that are expired according to the table metadata retention settings. While deleting the log files, it makes sure the time travel continues to work for all unexpired table versions.

Here is algorithm:

• Initial the potential delete file list: `potentialFilesToDelete` as an empty list
• Initialize the last seen checkpoint file list: `lastSeenCheckpointFiles`. There could be one or more checkpoint files for a given version.
• List the delta log files starting with prefix "00000000000000000000." (%020d). For each file:
  • Step 1: Check if the `lastSeenCheckpointFiles` contains a complete checkpoint, then
    • Step 1.1: delete all files in `potentialFilesToDelete`. Now we know there is a checkpoint that contains the compacted Delta log up to the checkpoint version and all commit/checkpoint files before this checkpoint version are not needed.
    • Step 1.2: add `lastCheckpointFiles` to `potentialFileStoDelete` list. This checkpoint is potential candidate to delete later if we find another checkpoint
  • Step 2: If the timestamp falls within the retention period, stop
  • Step 3: If the file is a delta log file, add it to the `potentialFilesToDelete` list
  • Step 4: If the file is a checkpoint file, add it to the `lastSeenCheckpointFiles`

Parameters:

engine - Engine instance to delete the expired log files

clock - Clock instance to get the current time. Useful in testing to mock the current time.

tablePath - Table location

retentionMillis - Log file retention period in milliseconds

Returns:

number of log files deleted

Throws:

java.io.IOException - if an error occurs while deleting the log files