com.sleepycat.je.cleaner

Class FileProtector

java.lang.Object

com.sleepycat.je.cleaner.FileProtector

public class FileProtector
extends Object

The FileProtector is primarily responsible for protecting files from being deleted due to log cleaning, when they are needed for other purposes. As such it is a gatekeeper for reading files. In addition it maintains certain metadata: - the size of each file, which is needed for calculating disk usage; - the first and last VLSNs in each reserved file, which are needed for maintaining the VLSNIndex; - the total size of active and reserved files, for disk usage statistics. All files are in three categories: + Active: readable by all. The minimum set of files needed to function. + Reserved: should be read only by feeders and low level utilities. They have been cleaned and will not become active again. They can be "condemned" and deleted if they are not being read. + Condemned: not readable and effectively invisible. They will be deleted ASAP. They will not become active or reserved again. A file is typically in the Condemned category for a very short time, while being deleted. Reserved files can be temporarily protected, i.e., prevented from being condemned and deleted. Only reserved files can be condemned and deleted. All active files are implicitly protected, but are also protected explicitly by consumers because they may become reserved while being consumed. Consumers of the File Protection Service ---------------------------------------- + DiskOrderedScanner (DiskOrderedCursor and Database.count) - Protects all currently active files. By currently active we mean active at the time they are protected. If they become reserved during the scan, they must continue to be protected. - Also protects any new files written during the scan. + DbBackup: - Protects-and-lists all currently active files, or a subset of the currently active files in the case of an incremental backup. - Provides API to remove files from protected set as they are copied, to allow file deletion. + NetworkRestore: - Protects-and-lists all currently active files using DbBackup. - Also protects the two newest reserved files at the time that the active files are protected. - Removes files from protected set as they are copied, to allow file deletion. + Syncup: - Protects all files (active and reserved) in an open ended range starting with the file of the VLSNIndex range start. Barren files (with no replicable entries) are not protected. + Feeder: - Protects all files (active and reserved) in an open ended range starting with the file of the current VLSN. Barren files (with no replicable entries) are not protected. - Advances lower bound of protected range as VLSN advances, to allow file deletion. + Cleaner: - Transforms active files into reserved files after cleaning them. - Condemns and deletes reserved files to honor disk limits. Truncates head of VLSNIndex when necessary to stay with disk thresholds. Syncup, Feeders and the VLSNIndex --------------------------------- During syncup, a ProtectedFileRange is used to protect files in the entire range of the VLSNIndex. Syncup must also prevent head truncation of the VLSNIndex itself because the file readers (used by both master and replica) use the VLSNIndex to position in the file at various times. A feeder file reader also protects all files from the current VLSN onward using a ProtectedFileRange. We rely on syncup to initialize the feeder's ProtectedFileRange safely, while the syncup ProtectedFileRange is in effect. The feeder reader will advance the lower bound of its ProtectedFileRange as it reads forward to allow files to be deleted. It also uses the VLNSIndex to skip over gaps in the file, although it is unclear whether this is really necessary. Therefore the syncup and feeder ProtectedFileRanges are special in that they also prevent head truncation of the VLSNIndex. The cleaner truncates the head of the VLSNIndex to allow deletion of files when necessary to stay within disk usage limits. This truncation must not be allowed when a syncup is in progress, and must not be allowed to remove the portion of the VLSN range used by a feeder. This is enforced using a special ProtectedFileRange (vlsnIndexRange) that protects the entire VLSNIndex range. The vlsnIndexRange is advanced when necessary to delete files that it protects, but only if those files are not protected by syncup or feeders. See checkVLSNIndexTruncation(long, com.sleepycat.je.utilint.VLSN), VLSNTracker.tryTruncateFromHead(long, com.sleepycat.je.rep.vlsn.LogItemCache) and VLSNTracker.protectRangeHead(java.lang.String). We takes pains to avoid synchronizing on FileProtector while truncating the VLSNIndex head, which is a relatively expensive operation. (The FileProtector is meant to be used by multiple threads without a lot of blocking and should perform only a fairly small amount of work while synchronized.) The following approach is used to truncate the VLSNIndex head safely: -- To prevent disk usage limit violations, Cleaner.manageDiskUsage first tries to delete reserved files without truncating the VLSNIndex. If this is not sufficient, it then tries to truncate the VLSNIndex head. If the VLSNIndex head can be truncated, then it tries again to delete reserved files, since more files should then be unprotected. -- VLSNTracker synchronization is used to protect the VLSNIndex range. The vlsnIndexRange ProtectedFileRange is advanced only while synchronized on the VLSNTracker. -- VLSNTracker.tryTruncateFromHead (which is synchronized) calls FileProtector.checkVLSNIndexTruncation to determine where to truncate the index. Reserved files can be removed from the VLSNIndex range only if they are not protected by syncup and feeders. -- The VLSNTracker range is then truncated, and the vlsnIndexRange is advanced to allow file deletion, all while synchronized on the tracker. -- When a syncup starts, it adds a ProtectedFileRange with the same startFile as the vlsnIndexRange. This is done while synchronized on the VLSNTracker and it prevents the vlsnIndexRange from advancing during the syncup. -- When a syncup is successful, on the master the Feeder is initialized and it adds a ProtectedFileRange to protect the range of the VLSNIndex that it is reading. This is done BEFORE removing the ProtectedFileRange that was added at the start of the syncup. This guarantees that the files and VLSNIndex range used by the feeder will not be truncated/deleted. Note that the special vlsnIndexRange ProtectedFileRange is excluded from LogSizeStats to avoid confusion and because this ProtectedFileRange does not ultimately prevent VLSNIndex head truncation or file deletion. Barren Files ------------ Files with no replicable entries do not need to be protected by feeders. See protectFileRange(String, long, boolean, boolean). Barren files may be created when cleaning is occurring but app writes are not, for example, when recovering from a cache size configuration error. In this situation it is important to delete the barren files to reclaim disk space. Such "barren" files are identified by having null begin/end VLSNs. The begin/end VLSNs for a file are part of the cleaner metadata that is collected when the cleaner processes a file. These VLSNs are for replicable entries only, not migrated entries that happen to contain a VLSN.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	FileProtector.ProtectedActiveFileSet A ProtectedFileSet created using protectActiveFiles(java.lang.String).
static class	FileProtector.ProtectedFileRange A ProtectedFileSet created using protectFileRange(java.lang.String, long).
static class	FileProtector.ProtectedFileSet A ProtectedFileSet is used to prevent a set of files from being deleted.

Field Summary

Fields

Modifier and Type	Field and Description
static String	BACKUP_NAME
static String	DATABASE_COUNT_NAME
static String	DISK_ORDERED_CURSOR_NAME
static String	FEEDER_NAME
static String	NETWORK_RESTORE_NAME
static String	SYNCUP_NAME
static String	VLSN_INDEX_NAME

Method Summary

Modifier and Type	Method and Description
boolean	checkVLSNIndexTruncation(Long file) Determines whether the VLSNIndex ProtectedFileRange should be advanced to unprotect the given file.
Pair<VLSN,Long>	checkVLSNIndexTruncation(long bytesNeeded, VLSN preserveVLSN) Determines whether the VLSNIndex ProtectedFileRange should be advanced to reclaim bytesNeeded.
Pair<Long,NavigableSet<Long>>	getReservedFileInfo() Returns a copy of the reserved files along with the total size.
boolean	isActiveOrNewFile(Long file) Returns whether the given file is active, including the last file whether or not it has been created on disk yet.
boolean	isReservedFile(Long file) Returns whether the given file is in the reserved file set.
FileProtector.ProtectedActiveFileSet	protectActiveFiles(String name) Calls protectActiveFiles(String, int, boolean) passing 0 for nReservedFiles and true for protectNewFiles.
FileProtector.ProtectedActiveFileSet	protectActiveFiles(String name, int nReservedFiles, boolean protectNewFiles) Returns a ProtectedActiveFileSet that protects all files currently active at the time of construction.
FileProtector.ProtectedFileRange	protectFileRange(String name, long rangeStart) Calls protectFileRange(String, long, boolean, boolean) passing false for protectVlsnIndex and true for protectBarrenFiles.
FileProtector.ProtectedFileRange	protectFileRange(String name, long rangeStart, boolean protectVlsnIndex, boolean protectBarrenFiles) Returns a ProtectedFileRange that protects files with numbers GTE a lower bound.
void	removeFileProtection(FileProtector.ProtectedFileSet pfs) Removes protection by the given ProtectedFileSet to allow files to be deleted.
void	setVLSNIndexProtectedFileRange(FileProtector.ProtectedFileRange pfs) Sets the ProtectedFileRange that protects files in VLSNIndex range from being deleted.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

BACKUP_NAME

public static final String BACKUP_NAME

See Also:

Constant Field Values

DATABASE_COUNT_NAME

public static final String DATABASE_COUNT_NAME

See Also:

Constant Field Values

DISK_ORDERED_CURSOR_NAME

public static final String DISK_ORDERED_CURSOR_NAME

See Also:

Constant Field Values

FEEDER_NAME

public static final String FEEDER_NAME

See Also:

Constant Field Values

SYNCUP_NAME

public static final String SYNCUP_NAME

See Also:

Constant Field Values

VLSN_INDEX_NAME

public static final String VLSN_INDEX_NAME

See Also:

Constant Field Values

NETWORK_RESTORE_NAME

public static final String NETWORK_RESTORE_NAME

See Also:

Constant Field Values

Method Detail

removeFileProtection

public void removeFileProtection(FileProtector.ProtectedFileSet pfs)

Removes protection by the given ProtectedFileSet to allow files to be deleted.

protectFileRange

public FileProtector.ProtectedFileRange protectFileRange(String name,
                                                         long rangeStart)

Calls protectFileRange(String, long, boolean, boolean) passing false for protectVlsnIndex and true for protectBarrenFiles.

protectFileRange

public FileProtector.ProtectedFileRange protectFileRange(String name,
                                                         long rangeStart,
                                                         boolean protectVlsnIndex,
                                                         boolean protectBarrenFiles)

Returns a ProtectedFileRange that protects files with numbers GTE a lower bound. The upper bound is open ended. The protectVlsnIndex param should be true for feeder/syncup file protection only.

Parameters:

rangeStart - is the first file to be protected in the range.

protectVlsnIndex - is whether to prevent the VLSNIndex head from advancing.

protectBarrenFiles - is whether barren files (having no replicable entries) are protected.

protectActiveFiles

public FileProtector.ProtectedActiveFileSet protectActiveFiles(String name)

Calls protectActiveFiles(String, int, boolean) passing 0 for nReservedFiles and true for protectNewFiles.

protectActiveFiles

public FileProtector.ProtectedActiveFileSet protectActiveFiles(String name,
                                                               int nReservedFiles,
                                                               boolean protectNewFiles)

Returns a ProtectedActiveFileSet that protects all files currently active at the time of construction. These files are protected even if they later become reserved. Note that this does not include the last file at the time of construction. Additional files can also be protected -- see params.

Parameters:

nReservedFiles - if greater than zero, this number of the newest (highest numbered) reserved files are also protected.

protectNewFiles - if true, the last file and any new files created later are also protected.

getReservedFileInfo

public Pair<Long,NavigableSet<Long>> getReservedFileInfo()

Returns a copy of the reserved files along with the total size.

isActiveOrNewFile

public boolean isActiveOrNewFile(Long file)

Returns whether the given file is active, including the last file whether or not it has been created on disk yet. If false is returned, the file is reserved, condemned or deleted.

isReservedFile

public boolean isReservedFile(Long file)

Returns whether the given file is in the reserved file set.

setVLSNIndexProtectedFileRange

public void setVLSNIndexProtectedFileRange(FileProtector.ProtectedFileRange pfs)

Sets the ProtectedFileRange that protects files in VLSNIndex range from being deleted. The range start is changed during VLSNIndex initialization and when the head of the index is truncated. It is changed while synchronized on VLSNTracker so that changes to the range and changes to the files it protects are made atomically. This is important for VLSNTracker.protectRangeHead(java.lang.String).

checkVLSNIndexTruncation

public Pair<VLSN,Long> checkVLSNIndexTruncation(long bytesNeeded,
                                                VLSN preserveVLSN)

Determines whether the VLSNIndex ProtectedFileRange should be advanced to reclaim bytesNeeded. This is possible if one or more reserved files are not protected by syncup and feeders. The range of files to be truncated must be at the head of the ordered set of reserved files, and the highest numbered file must contain a VLSN so we know where to truncate the VLSNIndex.

Parameters:

bytesNeeded - the number of bytes we need to free.

preserveVLSN - is the boundary above which the VLSN range may not advance. The deleteEnd returned will be less than preserveVLSN.

Returns:

{deleteEnd, deleteFileNum} pair if the protected file range should be advanced, or null if advancing is not currently possible. -- deleteEnd is the last VLSN to be truncated. -- deleteFileNum the file having deleteEnd as its last VLSN.

checkVLSNIndexTruncation

public boolean checkVLSNIndexTruncation(Long file)

Determines whether the VLSNIndex ProtectedFileRange should be advanced to unprotect the given file. This is possible if the file is not protected by syncup and feeders. The given file is assumed to contain deleteEnd VLSN, i.e., it is not a barren file.

Parameters:

file - that should be unprotected.

Returns:

whether the protected file range should be advanced.