com.sleepycat.je.cleaner

Class UtilizationCalculator

java.lang.Object

com.sleepycat.je.cleaner.UtilizationCalculator

All Implemented Interfaces:

EnvConfigObserver

public class UtilizationCalculator
extends Object
implements EnvConfigObserver

Contains methods for calculating utilization and for selecting files to clean. Note that we do clean files that are protected from deletion by HA/DataSync. If we did not clean them and a large number of files were to become unprotected at once, a large amount of log cleaning may suddenly be necessary. Cleaning the files avoids this. Better still would be to delete the metadata, but that would require writing a log entry to indicate the file is ready to be deleted, to avoid cleaning from scratch after a crash. [#16643] [#19221] Historical note: Prior to JE 6.0, LN utilization adjustments were needed because the LN last logged size was not stored in the BIN [#18633]. Originally in JE 5, the corrected average LN size was used to adjust utilization. This was changed later in JE 5 to a correction factor since different log files may have different average LN sizes [#21106]. Then in JE 6.0 the last logged size was added to the BIN, the default for EnvironmentConfig.CLEANER_ADJUST_UTILIZATION was changed to false and a warning was added that the feature will be removed in the future [#22275]. Finally in JE 6.3 the LN adjustment code and data in CheckpointEnd were removed and the parameter was deprecated [#24090]. Unlike with LNs, we do not store the last logged size of INs, so their obsolete size is computed as an average and this has the potential to cause over/under-cleaning. This problem is not known to occur, but if there are over/under-cleaning problems we should examine the recalculated info that is output as part of the CleanerRun INFO message. === Expired Data and Utilization === Per-file histograms are calculated by the ExpirationTracker and stored in an internal database and cache by the ExpirationProfile. The histograms are used to calculate the expired bytes for a file at a particular time. Since obsolete (not expired) data can overlap with expired data, upper and lower bounds for overall utilization are determined. When the lower bound is below minUtilization, cleaning occurs. The file that has the lowest average utilization (midway between its upper and lower bounds) is selected for cleaning. If the file's upper and lower bounds are not close together (or the same), and the upper bound is above a threshold, then two-pass cleaning is performed. See EnvironmentParams.CLEANER_TWO_PASS_GAP and EnvironmentParams.CLEANER_TWO_PASS_THRESHOLD. The first pass of two-pass cleaning reads the file but doesn't do any real cleaning (no side effects). If this pass finds that the true utilization of the file is above the threshold (the same threshold as above), then cleaning does not take place and instead the histogram is updated. In this case the obsolete and expired data (in the old histogram) overlap, and it is the overlap that caused utilization to be estimated incorrectly. The new/updated histogram is calculated such that there is no overlap, improving utilization accuracy. If the first pass finds that true utilization is below the threshold, then normal cleaning (pass two) occurs. Two-pass cleaning protects against "over cleaning". The use of the overall utilization lower bound to drive cleaning is considered sufficient to protect against "under cleaning". Therefore, a disk space threshold is unnecessary. Gradual expiration is used to prevent cleaning spikes on day or hour boundaries. For purposes of driving cleaning, the utilization lower bound is calculated by distributing the bytes that expired in the current day/hour period evenly over that day/hour.

Method Summary

Modifier and Type	Method and Description
void	envConfigUpdate(DbConfigManager cm, EnvironmentMutableConfig ignore) Process notifications of mutable property changes.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

envConfigUpdate

public void envConfigUpdate(DbConfigManager cm,
                            EnvironmentMutableConfig ignore)

Process notifications of mutable property changes.

Specified by:

envConfigUpdate in interface EnvConfigObserver

Throws:

IllegalArgumentException - via FilesToMigrate ctor and parseForceCleanFiles.