ai.eloquent.raft

Class Theseus

java.lang.Object

ai.eloquent.raft.Theseus

public class Theseus
extends java.lang.Object

This is the high-level user-facing API for RAFT. It's designed to appear simple to end-users. The key reason we need this as opposed to an off-the-shelf implementation is that we want it to be embedded in an application that runs on Kubernetes, and that imposes a key difference in assumptions over the original RAFT algorithm: boxes that fail will be replaced, not rebooted. We want this to be embedded so that our implementation for distributed locks will automatically release the lock of a box that disconnects from the cluster. Our core design goals: - Use EloquentChannel as the transport layer, to avoid flaky networking on Kubernetes - Don't persist state to disk, since boxes that fail will be replaced, not rebooted. - Keep it simple

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	Theseus.LongLivedLock This holds a lock that is expected to live longer than the duration of a single method call.

Field Summary

Fields

Modifier and Type	Field and Description
RaftLifecycle	lifecycle The RaftLifecycle that governs this Theseus
EloquentRaftNode	node The actual Raft node.
java.lang.String	serverName The name of our node.

Constructor Summary

Constructors

Constructor and Description
Theseus(int targetQuorumSize) Create a new dynamically resizing Raft cluster, with the given number of nodes as the target quorum size.
Theseus(RaftAlgorithm algo, RaftTransport transport, RaftLifecycle lifecycle) The constructor takes three arguments: a cluster name (for discovery), a server name (for identifying ourselves, must be unique within the cluster), and a reference to the lifecycle object that governs this Theseus (so that tests can pass different RaftLifecycle objects to different Raft instances).
Theseus(java.lang.String serverName, java.util.Collection<java.lang.String> quorum) Create a Raft cluster with a fixed quorum.
Theseus(java.lang.String serverName, RaftTransport transport, java.util.Collection<java.lang.String> initialMembership, RaftLifecycle lifecycle) Create a new fixed-size raft with the default algorithm, using the given transport.
Theseus(java.lang.String serverName, RaftTransport transport, int targetClusterSize, RaftLifecycle lifecycle) Create a new auto-resizing raft with the default algorithm, using the given transport.

Method Summary

Modifier and Type	Method and Description
void	addChangeListener(KeyValueStateMachine.ChangeListener changeListener) This registers a listener that will be called whenever the key-value store changes.
void	addErrorListener(RaftErrorListener errorListener) Raft keeps track of an additional error listener Errors are thrown from KeyValueStateMachine and EloquentRaftNode by default but users are free to attach their own error listeners Usage: RaftErrorHandler errorListener = (debugMessage, stackTrace) -> { // Do something with the debug message / stackTrace // Eg.
boolean	bootstrap() Bootstrap this cluster, if there are no leaders.
boolean	bootstrap(boolean force) Bootstrap this cluster, if there are no leaders.
void	clearErrorListeners() Remove all error listeners from Raft.
void	close() Stop this raft node.
java.util.List<java.lang.String>	errors() Return any errors Raft has encountered.
java.util.Set<java.lang.String>	getConfiguration() This returns the current understanding of the cluster membership on this node.
java.util.Optional<byte[]>	getElement(java.lang.String elementName) This grabs the current state of an element, if it's present in the system.
java.util.Collection<java.lang.String>	getKeys() This returns the current keys in the state machine.
java.util.Map<java.lang.String,java.lang.String>	getLocks() Get the set of locks that are held by the state machine, and the server that holds them.
java.util.Map<java.lang.String,byte[]>	getMap() This returns a snapshot of the current values in the state machine.
boolean	isLeader() If true, this node is the leader of the Raft cluster.
void	registerFailsafe(RaftFailsafe failsafe) Register a new failsafe to run occasionally on this raft node.
java.util.concurrent.CompletableFuture<java.lang.Boolean>	releaseLock(java.lang.String lockName) Release a Raft lock
void	removeChangeListener(KeyValueStateMachine.ChangeListener changeListener) This removes a listener that will be called whenever the key-value store changes.
java.util.concurrent.CompletableFuture<java.lang.Boolean>	removeElementAsync(java.lang.String elementName, java.time.Duration timeout) This removes an element from the Raft key-value store.
java.util.concurrent.CompletableFuture<java.lang.Boolean>	removeElementsAsync(java.util.Set<java.lang.String> elementName, java.time.Duration timeout) This removes a set of elements from the Raft key-value store.
void	removeErrorListener(RaftErrorListener errorListener) Remove an error listener from Raft.
java.util.concurrent.CompletableFuture<java.lang.Boolean>	setElementAsync(java.lang.String elementName, byte[] value, boolean permanent, java.time.Duration timeout) THIS IS DANGEROUS TO USE! People can clobber each other's writes, and there are tons of race conditions if you use this call in conjunction with getElement() with no outside synchronization mechanism.
RaftState	state() Get the current Raft state.
java.util.Optional<Theseus.LongLivedLock>	tryLock(java.lang.String lockName, java.time.Duration safetyReleaseWindow) This will try to acquire a lock.
java.util.concurrent.CompletableFuture<java.util.Optional<Theseus.LongLivedLock>>	tryLockAsync(java.lang.String lockName, java.time.Duration safetyReleaseWindow) This will try to acquire a lock.
java.util.concurrent.CompletableFuture<java.lang.Boolean>	withDistributedLockAsync(java.lang.String lockName, java.lang.Runnable runnable)
java.util.concurrent.CompletableFuture<java.lang.Boolean>	withDistributedLockAsync(java.lang.String lockName, java.util.function.Supplier<java.util.concurrent.CompletableFuture<java.lang.Boolean>> runnable) This runs a function while holding a distributed lock.
java.util.concurrent.CompletableFuture<java.lang.Boolean>	withElementAsync(java.lang.String elementName, java.util.function.Function<byte[],byte[]> mutator, java.util.function.Supplier<byte[]> createNew, boolean permanent) This is the safest way to do a withElement() update.
java.util.concurrent.CompletableFuture<java.lang.Boolean>	withElementUnlockedAsync(java.lang.String elementName, java.util.function.Function<byte[],byte[]> mutator, java.util.function.Supplier<byte[]> createNew, boolean permanent) Like withElementAsync(String, Function, Supplier, boolean), but without the safety of taking the lock on the element beforehand.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

serverName

public final java.lang.String serverName

The name of our node. This can also be gotten from node.EloquentRaftNode.algorithm.EloquentRaftAlgorithm.state.RaftState.serverName. But, from reading that path, you can tell why there's a helper here.

node

public final EloquentRaftNode node

The actual Raft node. This encapuslates a Raft algorithm with a transport.

lifecycle

public final RaftLifecycle lifecycle

The RaftLifecycle that governs this Theseus

Constructor Detail

Theseus

public Theseus(RaftAlgorithm algo,
               RaftTransport transport,
               RaftLifecycle lifecycle)

The constructor takes three arguments: a cluster name (for discovery), a server name (for identifying ourselves, must be unique within the cluster), and a reference to the lifecycle object that governs this Theseus (so that tests can pass different RaftLifecycle objects to different Raft instances).

Parameters:

algo - The Raft algorithm to use. Defaults to EloquentRaftAlgorithm.

transport - The type of transport to use for this Raft cluster.

lifecycle - The governing RaftLifecycle for this Theseus, so that we can pass mock ones in inside tests

Theseus

public Theseus(java.lang.String serverName,
               RaftTransport transport,
               int targetClusterSize,
               RaftLifecycle lifecycle)

Create a new auto-resizing raft with the default algorithm, using the given transport.

Parameters:

serverName - The name of this server in the cluster.

transport - The transport to use to communicate with the cluster.

targetClusterSize - The target quorum size we try to maintain with auto-resizing

lifecycle - The governing RaftLifecycle for this Theseus, so that we can pass mock ones in inside tests

Theseus

public Theseus(java.lang.String serverName,
               RaftTransport transport,
               java.util.Collection<java.lang.String> initialMembership,
               RaftLifecycle lifecycle)

Create a new fixed-size raft with the default algorithm, using the given transport.

Parameters:

serverName - The name of this server in the cluster.

initialMembership - The initial cluster membership.

lifecycle - The governing EloquentLifecycle for this Theseus, so that we can pass mock ones in inside tests

Theseus

public Theseus(java.lang.String serverName,
               java.util.Collection<java.lang.String> quorum)
        throws java.io.IOException

Create a Raft cluster with a fixed quorum.

Parameters:

serverName - The server name for this Raft node.

quorum - The fixed quorum for the cluster. This is a set of server names

Throws:

java.io.IOException - Thrown if we could not create the underlying transport.

Theseus

public Theseus(int targetQuorumSize)
        throws java.io.IOException

Create a new dynamically resizing Raft cluster, with the given number of nodes as the target quorum size. We will shrink the cluster if we have more than this number, and grow it if we have less.

Parameters:

targetQuorumSize - The target number of nodes in the quorum.

Throws:

java.io.IOException - Thrown if we could not create the underlying transport.

Method Detail

close

public void close()

Stop this raft node.

bootstrap

public boolean bootstrap(boolean force)

Bootstrap this cluster, if there are no leaders.

Parameters:

force - If true, attempt to take leadership by force. That is, massively increase the term number.

Returns:

True if the cluster was successfully bootstrapped

bootstrap

public boolean bootstrap()

Bootstrap this cluster, if there are no leaders.

Returns:

True if the cluster was successfully bootstrapped

state

public RaftState state()

Get the current Raft state. Note that this is a copy -- it may not be up to date, but it's safe to change

errors

public java.util.List<java.lang.String> errors()

Return any errors Raft has encountered.

withDistributedLockAsync

public java.util.concurrent.CompletableFuture<java.lang.Boolean> withDistributedLockAsync(java.lang.String lockName,
                                                                                          java.lang.Runnable runnable)

See Also:

This wraps a runnable with an instantly complete future.

withDistributedLockAsync

public java.util.concurrent.CompletableFuture<java.lang.Boolean> withDistributedLockAsync(java.lang.String lockName,
                                                                                          java.util.function.Supplier<java.util.concurrent.CompletableFuture<java.lang.Boolean>> runnable)

This runs a function while holding a distributed lock. The lock is global across the cluster, so you're safe to run from anywhere without fear of contention. This is similar to taking a SQL lock, but is much faster and is therefore appropriate for much lower latency scenarios.

Parameters:

lockName - the name of the lock -- in a global flat namespace

runnable - the runnable to execute while holding the lock

tryLock

public java.util.Optional<Theseus.LongLivedLock> tryLock(java.lang.String lockName,
                                                         java.time.Duration safetyReleaseWindow)

This will try to acquire a lock. It will block for network IO, but will not block waiting for the lock if the lock is not immediately acquired upon request. It returns a LongLivedLock object that is a reference to this lock, and contains a method to release it. The LongLivedLock object will also do its best to recover from situations where users fail/forget to release the lock, though it will yell at you if it has to clean up after you.

Parameters:

lockName - the name of the lock to attempt to acquire

safetyReleaseWindow - this is the duration of time after which we will automatically release the lock if it hasn't already been released. This is just a safety check, you should set it to much longer than you expect to hold the lock for.

Returns:

if successful, a LongLivedLock object that can be kept around as a reference to this lock, with a method to release it. If we were unable to acquire the lock, returns empty.

tryLockAsync

public java.util.concurrent.CompletableFuture<java.util.Optional<Theseus.LongLivedLock>> tryLockAsync(java.lang.String lockName,
                                                                                                      java.time.Duration safetyReleaseWindow)

This will try to acquire a lock. It will block for network IO, but will not block waiting for the lock if the lock is not immediately acquired upon request. It returns a LongLivedLock object that is a reference to this lock, and contains a method to release it. The LongLivedLock object will also do its best to recover from situations where users fail/forget to release the lock, though it will yell at you if it has to clean up after you.

Parameters:

lockName - the name of the lock to attempt to acquire

safetyReleaseWindow - this is the duration of time after which we will automatically release the lock if it hasn't already been released. This is just a safety check, you should set it to much longer than you expect to hold the lock for.

Returns:

if successful, a LongLivedLock object that can be kept around as a reference to this lock, with a method to release it. If we were unable to acquire the lock, returns empty.

releaseLock

public java.util.concurrent.CompletableFuture<java.lang.Boolean> releaseLock(java.lang.String lockName)

Release a Raft lock

Parameters:

lockName - The name of the lock we are releasing

Returns:

A completable future for whether the lock was released

withElementAsync

public java.util.concurrent.CompletableFuture<java.lang.Boolean> withElementAsync(java.lang.String elementName,
                                                                                  java.util.function.Function<byte[],byte[]> mutator,
                                                                                  @Nullable
                                                                                  java.util.function.Supplier<byte[]> createNew,
                                                                                  boolean permanent)

This is the safest way to do a withElement() update. We take a global lock, and then manipulate the item, and set the result. This means the mutator() can have side effects and not be idempotent, since it will only be called once. This safety comes at a performance cost, since this requires two writes to the raft cluster. This is in contrast to withElementIdempotent(), which uses a single compare-and-swap instead, to make it faster. Since we don't trust the Java serializer, which is used by Atomix for everything, we use a byte[] interface when interacting with the distributed storage. That way we can store elements that have a proto serialization interface, and still use proto.

Parameters:

elementName - the name of the element -- in a global flat namespace

mutator - the function to call, while holding a lock on the element, to mutate the element (doesn't have to actually change anything, but can).

createNew - a function to supply a new element, if none is present in the map already

permanent - if false, this element will be automatically removed when we disconnect from the cluster, if we're the last people to have edited the element.

Returns:

true on success, false if something went wrong

Throws:

java.util.NoSuchElementException - if we didn't supply a creator, and the object does not exist in Raft.

withElementUnlockedAsync

public java.util.concurrent.CompletableFuture<java.lang.Boolean> withElementUnlockedAsync(java.lang.String elementName,
                                                                                          java.util.function.Function<byte[],byte[]> mutator,
                                                                                          @Nullable
                                                                                          java.util.function.Supplier<byte[]> createNew,
                                                                                          boolean permanent)

Like withElementAsync(String, Function, Supplier, boolean), but without the safety of taking the lock on the element beforehand. This is a bit of a dangerous method, as it can open the caller up to race conditions, and should be used sparingly.

Parameters:

elementName - the name of the element -- in a global flat namespace

mutator - the function to call, while holding a lock on the element, to mutate the element (doesn't have to actually change anything, but can).

createNew - a function to supply a new element, if none is present in the map already

permanent - if false, this element will be automatically removed when we disconnect from the cluster, if we're the last people to have edited the element.

Returns:

true on success, false if something went wrong

Throws:

java.util.NoSuchElementException - if we didn't supply a creator, and the object does not exist in Raft.

setElementAsync

public java.util.concurrent.CompletableFuture<java.lang.Boolean> setElementAsync(java.lang.String elementName,
                                                                                 byte[] value,
                                                                                 boolean permanent,
                                                                                 java.time.Duration timeout)

THIS IS DANGEROUS TO USE! People can clobber each other's writes, and there are tons of race conditions if you use this call in conjunction with getElement() with no outside synchronization mechanism. Much safer, if you haven't thought about it much, is to use withElement(). Only use setElement() if you're really certain that you are the only one in the cluster writing, or you don't mind being clobbered.

Parameters:

elementName - the name of the element to write

value - the value to set the element to

permanent - if false, the element will be automatically removed when we disconnect from the cluster, if we're the last people to have edited the element. If true, the element will stick around forever.

removeElementAsync

public java.util.concurrent.CompletableFuture<java.lang.Boolean> removeElementAsync(java.lang.String elementName,
                                                                                    java.time.Duration timeout)

This removes an element from the Raft key-value store. It's a no-op if the value isn't already in the database.

Parameters:

elementName - the name of the element to remove

removeElementsAsync

public java.util.concurrent.CompletableFuture<java.lang.Boolean> removeElementsAsync(java.util.Set<java.lang.String> elementName,
                                                                                     java.time.Duration timeout)

This removes a set of elements from the Raft key-value store. It's a no-op if the value isn't already in the database.

Parameters:

elementName - the name of the element to remove

getElement

public java.util.Optional<byte[]> getElement(java.lang.String elementName)

This grabs the current state of an element, if it's present in the system. This may be out of date, since no lock is held on the item when retrieving it. If a lock is desired then use withElement() instead. This is non-blocking and fast.

Parameters:

elementName - the name of the element -- in a global flat namespace

getConfiguration

public java.util.Set<java.lang.String> getConfiguration()

This returns the current understanding of the cluster membership on this node.

getMap

public java.util.Map<java.lang.String,byte[]> getMap()

This returns a snapshot of the current values in the state machine. This is passed by value, not by reference, so this is safe to hold on to.

getKeys

public java.util.Collection<java.lang.String> getKeys()

This returns the current keys in the state machine. It's impossible to hold some sort of lock while fetching these, so these will be an eventually-consistent set, not an immediately consistent set.

getLocks

public java.util.Map<java.lang.String,java.lang.String> getLocks()

Get the set of locks that are held by the state machine, and the server that holds them. The keys are locks, mapped to the server that holds it.

addChangeListener

public void addChangeListener(KeyValueStateMachine.ChangeListener changeListener)

This registers a listener that will be called whenever the key-value store changes.

Parameters:

changeListener - the listener to register

removeChangeListener

public void removeChangeListener(KeyValueStateMachine.ChangeListener changeListener)

This removes a listener that will be called whenever the key-value store changes.

Parameters:

changeListener - the listener to deregister

addErrorListener

public void addErrorListener(RaftErrorListener errorListener)

Raft keeps track of an additional error listener Errors are thrown from KeyValueStateMachine and EloquentRaftNode by default but users are free to attach their own error listeners Usage: RaftErrorHandler errorListener = (debugMessage, stackTrace) -> { // Do something with the debug message / stackTrace // Eg. Logging, or alerting via PagerDuty } addErrorListener(errorListener); // Later in the code where there is an error throwRaftError(incident_key, debug_message);

Parameters:

errorListener - The error listener to add.

removeErrorListener

public void removeErrorListener(RaftErrorListener errorListener)

Remove an error listener from Raft.

Parameters:

errorListener - The error listener to remove.

clearErrorListeners

public void clearErrorListeners()

Remove all error listeners from Raft.

isLeader

public boolean isLeader()

If true, this node is the leader of the Raft cluster.

registerFailsafe

public void registerFailsafe(RaftFailsafe failsafe)

Register a new failsafe to run occasionally on this raft node.

Parameters:

failsafe - the one to register