com.sleepycat.je

Interface ExtinctionFilter

public interface ExtinctionFilter

Used in conjunction with Environment.discardExtinctRecords(com.sleepycat.je.Transaction, java.util.Set<java.lang.String>, com.sleepycat.je.DatabaseEntry, com.sleepycat.je.DatabaseEntry, com.sleepycat.je.ScanFilter, java.lang.String) to discard and purge extinct records. The ExtinctionFilter is specified by calling EnvironmentConfig.setExtinctionFilter(com.sleepycat.je.ExtinctionFilter). A non-null extinction filter must be specified before calling Environment.discardExtinctRecords.

The benefit of record extinction, compared to explicit record deletion, is that an entry is not logged or replicated for each extinct record, resulting in less resource usage. Only a single entry is logged and replicated when calling Environment.discardExtinctRecords, and a small amount of cleaner metadata is written when extinct records are counted obsolete. Performance impacts to be aware of are as follows.

• A Btree scan is performed asynchronously, after calling Environment.discardExtinctRecords. This scan will have little or no impact on ongoing operations, as long as Btree nodes (BINs in particular) are in the JE cache as recommended. The scan counts the size of the specified extinct records, logs this change to JE cleaner metadata, and removes the extinct records from the Btree. The changes to the Btree are logged at the next checkpoint.
• The performance benefits of record extinction are proportional to the size of the set of extinct records. For small sets of records (1000 or less), deleting them transactionally may perform just as well. Because cleaner metadata is logged for each call to Environment.discardExtinctRecords, using record extinction is not recommended for discarding many, very small sets of records (for example, discarding millions of sets of 100 records or less).

Record extinction is intended for optimized record deletion in cases where a large set of records is no longer needed and the keys of the extinct records will not be accessed again. It is the responsibility of the application to ensure that this condition holds. If extinct records are accessed, JE does not provide many of the usual transactional guarantees:

• WARNING:If an extinct record is inserted by the application, it may or may not be discarded by JE. This could appear as a leakage of memory and disk space.
• If an attempt is made by the application to read an extinct record, the record may or may not be found. Discarding of extinct records is asynchronous, and is performed independently on each node in a replication group.
• Transactional locking guarantees are not provided for extinct records. If an extinct record is read (or written) and locked by a transaction, it may (or may not) appear to be deleted when accessed again by the same transaction.

Below are examples of applications where keys are never accessed again, and therefore could be considered extinct.

• A single database is used to contain a set of logical "tables" where the records in a table are identified by a key prefix. Once a table is dropped, its key prefix is never reused. This is the case for Oracle NoSQL DB tables and record extinction is used for dropping tables.
• A database is used as a FIFO queue and record keys are sequentially assigned integers. A range of keys at the "out" end of the queue are removed after being processed and the keys are never reused. Using record extinction for removing the processed records would be beneficial, if the batches of dropped records are large enough to get performance benefits of record extinction (see above).

Each call to Environment.discardExtinctRecords specifies a key range, along with an optional filter, in a set of one ore more databases. When the transaction passed to this method is committed (or auto-commit is used), JE will asynchronously remove the specified records from the Btree and count them obsolete to cause log cleaning, which will reclaim unused disk space over time.

The getExtinctionStatus(java.lang.String, boolean, byte[]) method in this interface, on the other hand, must return either ExtinctionFilter.ExtinctionStatus.EXTINCT or ExtinctionFilter.ExtinctionStatus.MAYBE_EXTINCT for all record keys specified in all previous calls to Environment.discardExtinctRecords. This is necessary to avoid integrity errors and is the responsibility of the application. If ExtinctionFilter.ExtinctionStatus.NOT_EXTINCT is returned for a key previously specified as extinct using Environment.discardExtinctRecords, JE integrity checks may incorrectly fail, and an EnvironmentFailureException may be thrown.

The getExtinctionStatus method should return ExtinctionFilter.ExtinctionStatus.MAYBE_EXTINCT when Environment.discardExtinctRecords may have been called for a given key, but the status of the key is temporarily unknown to the application. For example, the application may use metadata stored in a JE Database to determine the extinction status of a record. During startup when this metadata has not yet been read from the Database, ExtinctionFilter.ExtinctionStatus.MAYBE_EXTINCT should be returned.

Although not recommended, it is possible to call Environment.discardExtinctRecords and implement a getExtinctionStatus method that always returns ExtinctionFilter.ExtinctionStatus.MAYBE_EXTINCT. This will not cause integrity errors, but is likely to result in lower performance and will disable certain internal integrity checks for all records.

The reason for lowered performance when ExtinctionFilter.ExtinctionStatus.MAYBE_EXTINCT is returned is that the getExtinctionStatus method is used by the JE cleaner to quickly purge extinct records without a Btree lookup. When getExtinctionStatus returns ExtinctionFilter.ExtinctionStatus.MAYBE_EXTINCT for records that are actually extinct, performance will be less than optimal for two reasons.

• The JE cleaner must perform a Btree lookup for each extinct record before it can be discarded.
• Some extinct records may be migrated forward by the cleaner.

For a given set of extinct records, the application should perform actions in the following sequence.

1. Stop accessing the extinct records.
2. Arrange for the ExtinctionFilter.getExtinctionStatus method to return ExtinctionFilter.ExtinctionStatus.EXTINCT for the extinct records.
3. Call Environment.discardExtinctRecords for the extinct records.

If step 2 is omitted, or step 2 is performed after step 3, performance will be suboptimal as described above. If step 1 is not performed first, normal transactional guarantees may be violated as described further above.

Although record extinction is asynchronous, for testing and debugging the Environment.isRecordExtinctionActive(long) may be useful for determining when related tasks are complete on a given node.

In a replicated environment, Environment.discardExtinctRecords may not be called until all nodes have been upgraded to JE 18.1 or later, and it will throw IllegalStateException if this is not the case. The Environment.isRecordExtinctionAvailable() method can be used for determining this in advance.

Since:

18.1

See Also:

Environment.discardExtinctRecords(com.sleepycat.je.Transaction, java.util.Set<java.lang.String>, com.sleepycat.je.DatabaseEntry, com.sleepycat.je.DatabaseEntry, com.sleepycat.je.ScanFilter, java.lang.String), EnvironmentStats.getNLNsExtinct()

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	ExtinctionFilter.ExtinctionStatus Extinction status values returned by getExtinctionStatus(java.lang.String, boolean, byte[]).

Method Summary

Modifier and Type	Method and Description
ExtinctionFilter.ExtinctionStatus	getExtinctionStatus(String dbName, boolean dups, byte[] key) Implemented to determine whether a given key is extinct.

Method Detail

getExtinctionStatus

ExtinctionFilter.ExtinctionStatus getExtinctionStatus(String dbName,
                                                      boolean dups,
                                                      byte[] key)

Implemented to determine whether a given key is extinct. This method must return ExtinctionFilter.ExtinctionStatus.EXTINCT or ExtinctionFilter.ExtinctionStatus.MAYBE_EXTINCT for all extinct records identified by calls to Environment.discardExtinctRecords(com.sleepycat.je.Transaction, java.util.Set<java.lang.String>, com.sleepycat.je.DatabaseEntry, com.sleepycat.je.DatabaseEntry, com.sleepycat.je.ScanFilter, java.lang.String).

Parameters:

dbName - the name of the JE database containing the record.

dups - whether the JE database has duplicate keys, which is assumed to mean it is a secondary database.

key - is the primary key of the record. If dups is true, this is the record's data and is assumed to be a primary key.

Returns:

the non-null ExtinctionStatus.