io.aeron.cluster

Class RecordingLog

java.lang.Object

io.aeron.cluster.RecordingLog

All Implemented Interfaces:

AutoCloseable

public final class RecordingLog
extends Object
implements AutoCloseable

A log of recordings which make up the history of a Raft log across leadership terms. Entries are in order.

The log is made up of entries of leadership terms or snapshots to roll up state as of a log position within a leadership term.

The latest state is made up of the latest snapshot followed by any leadership log which follows. It is possible that a snapshot is taken midterm and therefore the latest state is the snapshot plus the log of messages which got appended to the log after the snapshot was taken.

Record layout as follows:

   0                   1                   2                   3
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                        Recording ID                           |
  |                                                               |
  +---------------------------------------------------------------+
  |                     Leadership Term ID                        |
  |                                                               |
  +---------------------------------------------------------------+
  |              Log Position at beginning of term                |
  |                                                               |
  +---------------------------------------------------------------+
  |        Log Position when entry was created or committed       |
  |                                                               |
  +---------------------------------------------------------------+
  |               Timestamp when entry was created                |
  |                                                               |
  +---------------------------------------------------------------+
  |                  Service ID when a Snapshot                   |
  +---------------------------------------------------------------+
  |R|               Entry Type (Log or Snapshot)                  |
  +---------------------------------------------------------------+
  |                 Archive Endpoint (length)                     |
  +---------------------------------------------------------------+
  |                 Archive Endpoint (length)                     |
  |                    Archive Endpoint                         ...
  ...                     (variable)                              |
  +---------------------------------------------------------------+
  |                                                               |
  |                                                              ...
 ...                Repeats to the end of the log                 |
  |                                                               |
  +---------------------------------------------------------------+
 

The reserved bit on the entry type indicates whether the entry was marked invalid.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	RecordingLog.Entry Representation of the entry in the RecordingLog.
static class	RecordingLog.Log Representation of a log entry in the RecordingLog.
static class	RecordingLog.RecoveryPlan The snapshots and steps to recover the state of a cluster.
static class	RecordingLog.Snapshot Representation of a snapshot entry in the RecordingLog.

Field Summary

Fields

Modifier and Type	Field and Description
static int	ENDPOINT_OFFSET The offset at which the endpoint of the remote snapshot is held.
static int	ENTRY_TYPE_INVALID_FLAG The flag used to determine if the entry has been marked with invalid.
static int	ENTRY_TYPE_OFFSET The offset at which the type of the entry is stored.
static int	ENTRY_TYPE_SNAPSHOT The log entry is for a recording of a snapshot of state taken as of a position in the log.
static int	ENTRY_TYPE_STANDBY_SNAPSHOT The log entry is for a recording of a snapshot of state taken as of a position in the log on another machine.
static int	ENTRY_TYPE_TERM The log entry is for a recording of messages within a leadership term to the log.
static int	LEADERSHIP_TERM_ID_OFFSET The offset at which the leadership term id for the entry is stored.
static int	LOG_POSITION_OFFSET The offset at which the log position is stored.
static int	MAX_ENTRY_LENGTH Maximum possible entry length.
static int	RECORDING_ID_OFFSET The offset at which the recording id for the entry is stored.
static String	RECORDING_LOG_FILE_NAME Filename for the history of leadership log terms and snapshot recordings.
static int	SERVICE_ID_OFFSET The offset at which the service id is recorded.
static int	TERM_BASE_LOG_POSITION_OFFSET The offset at which the log position as of the beginning of the term for the entry is stored.
static int	TIMESTAMP_OFFSET The offset at which the timestamp for the entry is stored.

Constructor Summary

Constructors

Constructor and Description
RecordingLog(File parentDir, boolean createNew) Create a log that appends to an existing log or creates a new one.

Method Summary

Modifier and Type	Method and Description
void	appendSnapshot(long recordingId, long leadershipTermId, long termBaseLogPosition, long logPosition, long timestamp, int serviceId) Append a log entry for a snapshot.
void	appendStandbySnapshot(long recordingId, long leadershipTermId, long termBaseLogPosition, long logPosition, long timestamp, int serviceId, String archiveEndpoint) Append a log entry for a snapshot.
void	appendTerm(long recordingId, long leadershipTermId, long termBaseLogPosition, long timestamp) Append a log entry for a leadership term.
void	close()
void	commitLogPosition(long leadershipTermId, long logPosition) Commit the log position reached in a leadership term.
RecordingLog.RecoveryPlan	createRecoveryPlan(AeronArchive archive, int serviceCount, long replicatedRecordingId) Create a recovery plan for the cluster so that when the steps are replayed the plan will bring the cluster back to the latest stable state.
static RecordingLog.RecoveryPlan	createRecoveryPlan(ArrayList<RecordingLog.Snapshot> snapshots) Create a recovery plan that has only snapshots.
List<RecordingLog.Entry>	entries() List of currently loaded entries.
RecordingLog.Entry	findLastTerm() Find the last leadership term in the recording log.
long	findLastTermRecordingId() Find the last recording id used for a leadership term.
RecordingLog.Entry	findTermEntry(long leadershipTermId) Find the term RecordingLog.Entry for a given leadership term id.
void	force(int fileSyncLevel) Force the file to durable storage.
RecordingLog.Entry	getLatestSnapshot(int serviceId) Get the latest snapshot RecordingLog.Entry in the log.
RecordingLog.Entry	getTermEntry(long leadershipTermId) Get the term RecordingLog.Entry for a given leadership term id.
long	getTermTimestamp(long leadershipTermId) Get the RecordingLog.Entry.timestamp for a term.
void	invalidateEntry(long leadershipTermId, int entryIndex) Invalidate an entry in the log, so it is no longer valid.
boolean	invalidateLatestSnapshot() Invalidate the last snapshot taken by the cluster so that on restart it can revert to the previous one.
boolean	isUnknown(long leadershipTermId) Is the given leadershipTermId unknown for the log?
Map<String,List<RecordingLog.Entry>>	latestStandbySnapshots(int serviceCount) Return a collection of the most recent remote snapshots grouped by the archiveEndpoint where the snapshot is stored.
int	nextEntryIndex() Get the next index to be used when appending an entry to the log.
void	reload() Reload the recording log from disk.
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

RECORDING_LOG_FILE_NAME

public static final String RECORDING_LOG_FILE_NAME

Filename for the history of leadership log terms and snapshot recordings.

See Also:

Constant Field Values

ENTRY_TYPE_TERM

public static final int ENTRY_TYPE_TERM

The log entry is for a recording of messages within a leadership term to the log.

See Also:

Constant Field Values

ENTRY_TYPE_SNAPSHOT

public static final int ENTRY_TYPE_SNAPSHOT

The log entry is for a recording of a snapshot of state taken as of a position in the log.

See Also:

Constant Field Values

ENTRY_TYPE_STANDBY_SNAPSHOT

public static final int ENTRY_TYPE_STANDBY_SNAPSHOT

The log entry is for a recording of a snapshot of state taken as of a position in the log on another machine. Entries of this time should have an endpoint for an archive associated with them.

See Also:

Constant Field Values

ENTRY_TYPE_INVALID_FLAG

public static final int ENTRY_TYPE_INVALID_FLAG

The flag used to determine if the entry has been marked with invalid.

See Also:

Constant Field Values

RECORDING_ID_OFFSET

public static final int RECORDING_ID_OFFSET

The offset at which the recording id for the entry is stored.

See Also:

Constant Field Values

LEADERSHIP_TERM_ID_OFFSET

public static final int LEADERSHIP_TERM_ID_OFFSET

The offset at which the leadership term id for the entry is stored.

See Also:

Constant Field Values

TERM_BASE_LOG_POSITION_OFFSET

public static final int TERM_BASE_LOG_POSITION_OFFSET

The offset at which the log position as of the beginning of the term for the entry is stored.

See Also:

Constant Field Values

LOG_POSITION_OFFSET

public static final int LOG_POSITION_OFFSET

The offset at which the log position is stored.

See Also:

Constant Field Values

TIMESTAMP_OFFSET

public static final int TIMESTAMP_OFFSET

The offset at which the timestamp for the entry is stored.

See Also:

Constant Field Values

SERVICE_ID_OFFSET

public static final int SERVICE_ID_OFFSET

The offset at which the service id is recorded.

See Also:

Constant Field Values

ENTRY_TYPE_OFFSET

public static final int ENTRY_TYPE_OFFSET

The offset at which the type of the entry is stored.

See Also:

Constant Field Values

ENDPOINT_OFFSET

public static final int ENDPOINT_OFFSET

The offset at which the endpoint of the remote snapshot is held.

See Also:

Constant Field Values

MAX_ENTRY_LENGTH

public static final int MAX_ENTRY_LENGTH

Maximum possible entry length. Include the entry plus a variable length endpoint string.

Constructor Detail

RecordingLog

public RecordingLog(File parentDir,
                    boolean createNew)

Create a log that appends to an existing log or creates a new one.

Parameters:

parentDir - in which the log will be created.

createNew - create a new recording log if one does not exist.

Method Detail

close

public void close()

Specified by:

close in interface AutoCloseable

force

public void force(int fileSyncLevel)

Force the file to durable storage. Same as calling FileChannel.force(boolean) with true.

Parameters:

fileSyncLevel - as defined by ConsensusModule.Configuration.FILE_SYNC_LEVEL_PROP_NAME.

entries

public List<RecordingLog.Entry> entries()

List of currently loaded entries.

Returns:

the list of currently loaded entries.

nextEntryIndex

public int nextEntryIndex()

Get the next index to be used when appending an entry to the log.

Returns:

the next index to be used when appending an entry to the log.

reload

public void reload()

Reload the recording log from disk.

findLastTermRecordingId

public long findLastTermRecordingId()

Find the last recording id used for a leadership term. If not found then RecordingPos.NULL_RECORDING_ID.

Returns:

the last leadership term recording id or RecordingPos.NULL_RECORDING_ID if not found.

findLastTerm

public RecordingLog.Entry findLastTerm()

Find the last leadership term in the recording log.

Returns:

the last leadership term in the recording log.

getTermEntry

public RecordingLog.Entry getTermEntry(long leadershipTermId)

Get the term RecordingLog.Entry for a given leadership term id.

Parameters:

leadershipTermId - to get RecordingLog.Entry for.

Returns:

the RecordingLog.Entry if found or throw IllegalArgumentException if no entry exists for term.

findTermEntry

public RecordingLog.Entry findTermEntry(long leadershipTermId)

Find the term RecordingLog.Entry for a given leadership term id.

Parameters:

leadershipTermId - to get RecordingLog.Entry for.

Returns:

the RecordingLog.Entry if found or null if not found.

getLatestSnapshot

public RecordingLog.Entry getLatestSnapshot(int serviceId)

Get the latest snapshot RecordingLog.Entry in the log.

Parameters:

serviceId - for the snapshot.

Returns:

the latest snapshot RecordingLog.Entry in the log or null if no snapshot exists.

invalidateLatestSnapshot

public boolean invalidateLatestSnapshot()

Invalidate the last snapshot taken by the cluster so that on restart it can revert to the previous one.

Returns:

true if the latest snapshot was found and marked as invalid, so it will not be reloaded.

getTermTimestamp

public long getTermTimestamp(long leadershipTermId)

Get the RecordingLog.Entry.timestamp for a term.

Parameters:

leadershipTermId - to get RecordingLog.Entry.timestamp for.

Returns:

the timestamp or Aeron.NULL_VALUE if not found.

createRecoveryPlan

public RecordingLog.RecoveryPlan createRecoveryPlan(AeronArchive archive,
                                                    int serviceCount,
                                                    long replicatedRecordingId)

Create a recovery plan for the cluster so that when the steps are replayed the plan will bring the cluster back to the latest stable state.

Parameters:

archive - to lookup recording descriptors.

serviceCount - of services that may have snapshots.

replicatedRecordingId - leader's recordingId used for replicating

Returns:

a new RecordingLog.RecoveryPlan for the cluster.

createRecoveryPlan

public static RecordingLog.RecoveryPlan createRecoveryPlan(ArrayList<RecordingLog.Snapshot> snapshots)

Create a recovery plan that has only snapshots. Used for dynamicJoin snapshot load.

Parameters:

snapshots - to construct plan from.

Returns:

a new RecordingLog.RecoveryPlan for the cluster.

isUnknown

public boolean isUnknown(long leadershipTermId)

Is the given leadershipTermId unknown for the log?

Parameters:

leadershipTermId - to check.

Returns:

true if term has not yet been appended otherwise false.

appendTerm

public void appendTerm(long recordingId,
                       long leadershipTermId,
                       long termBaseLogPosition,
                       long timestamp)

Append a log entry for a leadership term. Terms must be appended in ascending order.

Parameters:

recordingId - of the log in the archive.

leadershipTermId - for the appended term.

termBaseLogPosition - for the beginning of the term.

timestamp - at the beginning of the appended term.

appendSnapshot

public void appendSnapshot(long recordingId,
                           long leadershipTermId,
                           long termBaseLogPosition,
                           long logPosition,
                           long timestamp,
                           int serviceId)

Append a log entry for a snapshot. Snapshots must be for the current term.

Parameters:

recordingId - in the archive for the snapshot.

leadershipTermId - for the current term

termBaseLogPosition - at the beginning of the leadership term.

logPosition - within the current term or accumulated length for the log.

timestamp - at which the snapshot was taken.

serviceId - for which the snapshot is recorded.

appendStandbySnapshot

public void appendStandbySnapshot(long recordingId,
                                  long leadershipTermId,
                                  long termBaseLogPosition,
                                  long logPosition,
                                  long timestamp,
                                  int serviceId,
                                  String archiveEndpoint)

Append a log entry for a snapshot. Snapshots must be for the current term.

Parameters:

recordingId - in the archive for the snapshot.

leadershipTermId - for the current term

termBaseLogPosition - at the beginning of the leadership term.

logPosition - within the current term or accumulated length for the log.

timestamp - at which the snapshot was taken.

serviceId - for which the snapshot is recorded.

archiveEndpoint - endpoint for the archive where

commitLogPosition

public void commitLogPosition(long leadershipTermId,
                              long logPosition)

Commit the log position reached in a leadership term.

Parameters:

leadershipTermId - for committing the term position reached.

logPosition - reached in the leadership term.

invalidateEntry

public void invalidateEntry(long leadershipTermId,
                            int entryIndex)

Invalidate an entry in the log, so it is no longer valid. Be careful that the recording log is not left in an invalid state for recovery.

Parameters:

leadershipTermId - to match for validation.

entryIndex - reached in the leadership term.

See Also:

invalidateLatestSnapshot()

latestStandbySnapshots

public Map<String,List<RecordingLog.Entry>> latestStandbySnapshots(int serviceCount)

Return a collection of the most recent remote snapshots grouped by the archiveEndpoint where the snapshot is stored.

Parameters:

serviceCount - to ensure that we have a complete set of snapshots.

Returns:

collection of snapshots.

toString

public String toString()

Overrides:

toString in class Object