com.sleepycat.je.incomp

Class INCompressor

java.lang.Object

com.sleepycat.je.utilint.DaemonThread

com.sleepycat.je.incomp.INCompressor

All Implemented Interfaces:

DaemonRunner, Runnable

public class INCompressor
extends DaemonThread

JE compression consists of removing BIN slots for deleted and expired records, and pruning empty IN/BINs from the tree which is also called a reverse split. One of the reasons compression is treated specially is that slot compression cannot be performed inline as part of a delete operation. When we delete an LN, a cursor is always present on the LN. The API dictates that the cursor will remain positioned on the deleted record. In addition, if the deleting transaction aborts we must restore the slot and the possibility of a split during an abort is something we wish to avoid; for this reason, compression will not occur if the slot's LSN is locked. In principle, slot compression could be performed during transaction commit, but that would be expensive because a Btree lookup would be required, and this would negatively impact operation latency. For all these reasons, slot compression is performed after the delete operation is complete and committed, and not in the thread performing the operation or transaction commit. Compression is of two types: + "Queued compression" is carried out by the INCompressor daemon thread. Both slot compression and pruning are performed. + "Lazy compression" is carried out opportunistically at various times when compression is beneficial. The use of BIN-deltas has a big impact on slot compression because dirty slots cannot be compressed until we know that a full BIN will be logged next. If a dirty slot were compressed prior to logging a BIN-delta, the record of the compression would be lost and the slot would "reappear" when the BIN is reconstituted. Normally we do not compress dirty slots when a delta would next be logged. However, there are times when we do compress dirty slots, and in that case the "prohibit next logged delta" flag is set on the BIN. Queued compression prior to logging a BIN-delta is also wasteful because the dequeued entry cannot be processed. Therefore, lazy compression is relied on when a BIN-delta will next be logged. Because, BIN-deltas are logged more often than BINs, lazy compression is used for slot compression more often than queued compression. Lazy compression is used for compressing expired slots, in addition to deleted slots. This is done opportunistically as described above. Expired slots are normally not dirty, so they can often be compressed even when a BIN will be logged next as a BIN-delta. The same is true of deleted slots that are not dirty, although these occur infrequently. Since we don't lazy-compress BIN-deltas, how can we prevent their expired slots from using space for long time periods? Currently the expired space will be reclaimed only when the delta is mutated to a full BIN and then compressed, including when the full BIN is cleaned. This is the same as reclaiming space for deleted slots, so this is acceptable for now at least. You may wonder, since lazy compression is necessary, why use queued compression for slot compression at all? Queued compression is useful for the following reasons: + If a BIN-delta will not be logged next, queued compression will cause the compression to occur sooner than with lazy compression. + When a cursor is on a BIN or a deleted entry is locked during lazy compression, we cannot compress the slot. Queuing allows it to be compressed sooner than if we waited for the next lazy compression. + The code to process a queue entry must do slot compression anyway, even if we only want to prune the BIN. We have to account for the case where all slots are deleted but not yet compressed. So the code to process the queue entry could not be simplified even if we were to decide not to queue entries for slot compression.

Field Summary

Fields inherited from class com.sleepycat.je.utilint.DaemonThread

envImpl, logger, name, nWakeupRequests, stifleExceptionChatter

Constructor Summary

Constructors

Constructor and Description
INCompressor(EnvironmentImpl env, long waitTime, String name)

Method Summary

Modifier and Type	Method and Description
void	addBinToQueue(BIN bin) Adds the BIN to the queue if the BIN is not already in the queue.
void	addMultipleBinRefsToQueue(Collection<BINReference> binRefs) Adds an entire collection of BINReferences to the queue at once.
void	doCompress() The real work to doing a compress.
boolean	exists(long nodeId)
int	getBinRefQueueSize()
void	lazyCompress(IN in, boolean compressDirtySlots) Lazily/opportunistically compress a full BIN.
StatGroup	loadStats(StatsConfig config) Return stats
protected long	nDeadlockRetries() Return the number of retries when a deadlock exception occurs.
void	onWakeup() onWakeup is synchronized to ensure that multiple invocations of the DaemonThread aren't made.
BIN	searchForBIN(DatabaseImpl db, BINReference binRef) Search the tree for the BIN that corresponds to this BINReference.
void	setBeforeFlushTrackerHook(TestHook hook)
void	verifyCursors()

Methods inherited from class com.sleepycat.je.utilint.DaemonThread

checkErrorListener, createLogger, getNWakeupRequests, getThread, isPaused, isRunning, isShutdownRequested, requestShutdown, run, runOrPause, setWaitTime, shutdown, toString, wakeup

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

INCompressor

public INCompressor(EnvironmentImpl env,
                    long waitTime,
                    String name)

Method Detail

setBeforeFlushTrackerHook

public void setBeforeFlushTrackerHook(TestHook hook)

verifyCursors

public void verifyCursors()
                   throws DatabaseException

Throws:

DatabaseException

getBinRefQueueSize

public int getBinRefQueueSize()

addBinToQueue

public void addBinToQueue(BIN bin)

Adds the BIN to the queue if the BIN is not already in the queue.

addMultipleBinRefsToQueue

public void addMultipleBinRefsToQueue(Collection<BINReference> binRefs)

Adds an entire collection of BINReferences to the queue at once. Use this to avoid latching for each add.

exists

public boolean exists(long nodeId)

loadStats

public StatGroup loadStats(StatsConfig config)

Return stats

nDeadlockRetries

protected long nDeadlockRetries()

Return the number of retries when a deadlock exception occurs.

Overrides:

nDeadlockRetries in class DaemonThread

onWakeup

public void onWakeup()
              throws DatabaseException

Description copied from class: DaemonThread

onWakeup is synchronized to ensure that multiple invocations of the DaemonThread aren't made.

Specified by:

onWakeup in class DaemonThread

Throws:

DatabaseException

doCompress

public void doCompress()
                throws DatabaseException

The real work to doing a compress. This may be called by the compressor thread or programatically.

Throws:

DatabaseException

searchForBIN

public BIN searchForBIN(DatabaseImpl db,
                        BINReference binRef)

Search the tree for the BIN that corresponds to this BINReference.

Parameters:

binRef - the BINReference that indicates the bin we want.

Returns:

the BIN that corresponds to this BINReference. The node is latched upon return. Returns null if the BIN can't be found.

lazyCompress

public void lazyCompress(IN in,
                         boolean compressDirtySlots)

Lazily/opportunistically compress a full BIN. The target IN should be latched when we enter, and it will be remain latched. If compression succeeds, does not prune empty BINs, but does queue them for pruning later. If compression fails because a record lock cannot be obtained, queues the BIN to retry later. Note that we do not bother to delete queue entries for the BIN if compression succeeds. Queue entries are normally removed quickly by the compressor. In the case where queue entries happen to exist when we do the final compression below, we rely on the compressor to clean them up later on when they are processed.