org.apache.hadoop.yarn.server.resourcemanager.recovery
Class RMStateStore

java.lang.Object
  org.apache.hadoop.service.AbstractService
      org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore

All Implemented Interfaces:

Closeable, org.apache.hadoop.service.Service

Direct Known Subclasses:

FileSystemRMStateStore, MemoryRMStateStore, NullRMStateStore, ZKRMStateStore

@InterfaceAudience.Private
@InterfaceStability.Unstable
public abstract class RMStateStore
extends org.apache.hadoop.service.AbstractService

Nested Class Summary

static class	RMStateStore.ApplicationAttemptState State of an application attempt
static class	RMStateStore.ApplicationState State of an application application
static class	RMStateStore.RMDTSecretManagerState
static class	RMStateStore.RMState State of the ResourceManager

Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

org.apache.hadoop.service.Service.STATE

Field Summary

static org.apache.hadoop.io.Text	AM_CLIENT_TOKEN_MASTER_KEY_NAME
static org.apache.hadoop.io.Text	AM_RM_TOKEN_SERVICE
protected static String	AMRMTOKEN_SECRET_MANAGER_ROOT
protected static String	DELEGATION_KEY_PREFIX
protected static String	DELEGATION_TOKEN_PREFIX
protected static String	DELEGATION_TOKEN_SEQUENCE_NUMBER_PREFIX
protected static String	EPOCH_NODE
static org.apache.commons.logging.Log	LOG
protected static String	RM_APP_ROOT
protected static String	RM_DT_SECRET_MANAGER_ROOT
protected static String	VERSION_NODE

Constructor Summary

RMStateStore()

Method Summary

void	checkVersion() 1) Versioning scheme: major.minor.
protected abstract void	closeInternal() Derived classes close themselves using this method.
abstract void	deleteStore() Derived classes must implement this method to delete the state store
abstract long	getAndIncrementEpoch() Get the current epoch of RM and increment the value.
org.apache.hadoop.security.Credentials	getCredentialsFromAppAttempt(RMAppAttempt appAttempt)
protected abstract org.apache.hadoop.yarn.server.records.Version	getCurrentVersion() Get the current version of the underlying state store.
protected void	handleStoreEvent(RMStateStoreEvent event)
protected abstract void	initInternal(org.apache.hadoop.conf.Configuration conf) Derived classes initialize themselves using this method.
abstract RMStateStore.RMState	loadState() Blocking API The derived class must recover state from the store and return a new RMState object populated with that state This must not be called on the dispatcher thread
protected abstract org.apache.hadoop.yarn.server.records.Version	loadVersion() Derived class use this method to load the version information from state store.
protected void	notifyStoreOperationFailed(Exception failureCause)
void	removeApplication(RMApp app) Non-blocking API ResourceManager services call this to remove an application from the state store This does not block the dispatcher threads There is no notification of completion for this operation.
protected abstract void	removeApplicationStateInternal(RMStateStore.ApplicationState appState) Blocking API Derived classes must implement this method to remove the state of an application and its attempts
void	removeRMDelegationToken(org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier rmDTIdentifier, int sequenceNumber) RMDTSecretManager call this to remove the state of a delegation token
protected abstract void	removeRMDelegationTokenState(org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier rmDTIdentifier) Blocking API Derived classes must implement this method to remove the state of RMDelegationToken
void	removeRMDTMasterKey(org.apache.hadoop.security.token.delegation.DelegationKey delegationKey) RMDTSecretManager call this to remove the state of a master key
protected abstract void	removeRMDTMasterKeyState(org.apache.hadoop.security.token.delegation.DelegationKey delegationKey) Blocking API Derived classes must implement this method to remove the state of DelegationToken Master Key
protected void	serviceInit(org.apache.hadoop.conf.Configuration conf)
protected void	serviceStart()
protected void	serviceStop()
void	setResourceManager(ResourceManager rm)
void	setRMDispatcher(org.apache.hadoop.yarn.event.Dispatcher dispatcher) Dispatcher used to send state operation completion events to ResourceManager services
protected abstract void	startInternal() Derived classes start themselves using this method.
protected abstract void	storeApplicationAttemptStateInternal(org.apache.hadoop.yarn.api.records.ApplicationAttemptId attemptId, ApplicationAttemptStateData attemptStateData) Blocking API Derived classes must implement this method to store the state of an application attempt
protected abstract void	storeApplicationStateInternal(org.apache.hadoop.yarn.api.records.ApplicationId appId, ApplicationStateData appStateData) Blocking API Derived classes must implement this method to store the state of an application.
void	storeNewApplication(RMApp app) Non-Blocking API ResourceManager services use this to store the application's state This does not block the dispatcher threads RMAppStoredEvent will be sent on completion to notify the RMApp
void	storeNewApplicationAttempt(RMAppAttempt appAttempt)
abstract void	storeOrUpdateAMRMTokenSecretManagerState(AMRMTokenSecretManagerState amrmTokenSecretManagerState, boolean isUpdate) Blocking API Derived classes must implement this method to store or update the state of AMRMToken Master Key
void	storeRMDelegationTokenAndSequenceNumber(org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier rmDTIdentifier, Long renewDate, int latestSequenceNumber) RMDTSecretManager call this to store the state of a delegation token and sequence number
protected abstract void	storeRMDelegationTokenAndSequenceNumberState(org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier rmDTIdentifier, Long renewDate, int latestSequenceNumber) Blocking API Derived classes must implement this method to store the state of RMDelegationToken and sequence number
void	storeRMDTMasterKey(org.apache.hadoop.security.token.delegation.DelegationKey delegationKey) RMDTSecretManager call this to store the state of a master key
protected abstract void	storeRMDTMasterKeyState(org.apache.hadoop.security.token.delegation.DelegationKey delegationKey) Blocking API Derived classes must implement this method to store the state of DelegationToken Master Key
protected abstract void	storeVersion() Derived class use this method to store the version information.
void	updateApplicationAttemptState(RMStateStore.ApplicationAttemptState attemptState)
protected abstract void	updateApplicationAttemptStateInternal(org.apache.hadoop.yarn.api.records.ApplicationAttemptId attemptId, ApplicationAttemptStateData attemptStateData)
void	updateApplicationState(RMStateStore.ApplicationState appState)
protected abstract void	updateApplicationStateInternal(org.apache.hadoop.yarn.api.records.ApplicationId appId, ApplicationStateData appStateData)
void	updateRMDelegationTokenAndSequenceNumber(org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier rmDTIdentifier, Long renewDate, int latestSequenceNumber) RMDTSecretManager call this to update the state of a delegation token and sequence number
protected abstract void	updateRMDelegationTokenAndSequenceNumberInternal(org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier rmDTIdentifier, Long renewDate, int latestSequenceNumber) Blocking API Derived classes must implement this method to update the state of RMDelegationToken and sequence number

Methods inherited from class org.apache.hadoop.service.AbstractService

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

RM_APP_ROOT

protected static final String RM_APP_ROOT

See Also:

Constant Field Values

RM_DT_SECRET_MANAGER_ROOT

protected static final String RM_DT_SECRET_MANAGER_ROOT

See Also:

Constant Field Values

DELEGATION_KEY_PREFIX

protected static final String DELEGATION_KEY_PREFIX

See Also:

Constant Field Values

DELEGATION_TOKEN_PREFIX

protected static final String DELEGATION_TOKEN_PREFIX

See Also:

Constant Field Values

DELEGATION_TOKEN_SEQUENCE_NUMBER_PREFIX

protected static final String DELEGATION_TOKEN_SEQUENCE_NUMBER_PREFIX

See Also:

Constant Field Values

AMRMTOKEN_SECRET_MANAGER_ROOT

protected static final String AMRMTOKEN_SECRET_MANAGER_ROOT

See Also:

Constant Field Values

VERSION_NODE

protected static final String VERSION_NODE

See Also:

Constant Field Values

EPOCH_NODE

protected static final String EPOCH_NODE

See Also:

Constant Field Values

LOG

public static final org.apache.commons.logging.Log LOG

AM_RM_TOKEN_SERVICE

public static final org.apache.hadoop.io.Text AM_RM_TOKEN_SERVICE

AM_CLIENT_TOKEN_MASTER_KEY_NAME

public static final org.apache.hadoop.io.Text AM_CLIENT_TOKEN_MASTER_KEY_NAME

Constructor Detail

RMStateStore

public RMStateStore()

Method Detail

setRMDispatcher

public void setRMDispatcher(org.apache.hadoop.yarn.event.Dispatcher dispatcher)

Dispatcher used to send state operation completion events to ResourceManager services

serviceInit

protected void serviceInit(org.apache.hadoop.conf.Configuration conf)
                    throws Exception

Overrides:

serviceInit in class org.apache.hadoop.service.AbstractService

Throws:

Exception

serviceStart

protected void serviceStart()
                     throws Exception

Overrides:

serviceStart in class org.apache.hadoop.service.AbstractService

Throws:

Exception

initInternal

protected abstract void initInternal(org.apache.hadoop.conf.Configuration conf)
                              throws Exception

Derived classes initialize themselves using this method.

Throws:

Exception

startInternal

protected abstract void startInternal()
                               throws Exception

Derived classes start themselves using this method. The base class is started and the event dispatcher is ready to use at this point

Throws:

Exception

serviceStop

protected void serviceStop()
                    throws Exception

Overrides:

serviceStop in class org.apache.hadoop.service.AbstractService

Throws:

Exception

closeInternal

protected abstract void closeInternal()
                               throws Exception

Derived classes close themselves using this method. The base class will be closed and the event dispatcher will be shutdown after this

Throws:

Exception

checkVersion

public void checkVersion()
                  throws Exception

1) Versioning scheme: major.minor. For e.g. 1.0, 1.1, 1.2...1.25, 2.0 etc. 2) Any incompatible change of state-store is a major upgrade, and any compatible change of state-store is a minor upgrade. 3) If theres's no version, treat it as 1.0. 4) Within a minor upgrade, say 1.1 to 1.2: overwrite the version info and proceed as normal. 5) Within a major upgrade, say 1.2 to 2.0: throw exception and indicate user to use a separate upgrade tool to upgrade RM state.

Throws:

Exception

loadVersion

protected abstract org.apache.hadoop.yarn.server.records.Version loadVersion()
                                                                      throws Exception

Derived class use this method to load the version information from state store.

Throws:

Exception

storeVersion

protected abstract void storeVersion()
                              throws Exception

Derived class use this method to store the version information.

Throws:

Exception

getCurrentVersion

protected abstract org.apache.hadoop.yarn.server.records.Version getCurrentVersion()

Get the current version of the underlying state store.

getAndIncrementEpoch

public abstract long getAndIncrementEpoch()
                                   throws Exception

Get the current epoch of RM and increment the value.

Throws:

Exception

loadState

public abstract RMStateStore.RMState loadState()
                                        throws Exception

Blocking API The derived class must recover state from the store and return a new RMState object populated with that state This must not be called on the dispatcher thread

Throws:

Exception

storeNewApplication

public void storeNewApplication(RMApp app)

Non-Blocking API ResourceManager services use this to store the application's state This does not block the dispatcher threads RMAppStoredEvent will be sent on completion to notify the RMApp

updateApplicationState

public void updateApplicationState(RMStateStore.ApplicationState appState)

storeApplicationStateInternal

protected abstract void storeApplicationStateInternal(org.apache.hadoop.yarn.api.records.ApplicationId appId,
                                                      ApplicationStateData appStateData)
                                               throws Exception

Blocking API Derived classes must implement this method to store the state of an application.

Throws:

Exception

updateApplicationStateInternal

protected abstract void updateApplicationStateInternal(org.apache.hadoop.yarn.api.records.ApplicationId appId,
                                                       ApplicationStateData appStateData)
                                                throws Exception

Throws:

Exception

storeNewApplicationAttempt

public void storeNewApplicationAttempt(RMAppAttempt appAttempt)

updateApplicationAttemptState

public void updateApplicationAttemptState(RMStateStore.ApplicationAttemptState attemptState)

storeApplicationAttemptStateInternal

protected abstract void storeApplicationAttemptStateInternal(org.apache.hadoop.yarn.api.records.ApplicationAttemptId attemptId,
                                                             ApplicationAttemptStateData attemptStateData)
                                                      throws Exception

Blocking API Derived classes must implement this method to store the state of an application attempt

Throws:

Exception

updateApplicationAttemptStateInternal

protected abstract void updateApplicationAttemptStateInternal(org.apache.hadoop.yarn.api.records.ApplicationAttemptId attemptId,
                                                              ApplicationAttemptStateData attemptStateData)
                                                       throws Exception

Throws:

Exception

storeRMDelegationTokenAndSequenceNumber

public void storeRMDelegationTokenAndSequenceNumber(org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier rmDTIdentifier,
                                                    Long renewDate,
                                                    int latestSequenceNumber)

RMDTSecretManager call this to store the state of a delegation token and sequence number

storeRMDelegationTokenAndSequenceNumberState

protected abstract void storeRMDelegationTokenAndSequenceNumberState(org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier rmDTIdentifier,
                                                                     Long renewDate,
                                                                     int latestSequenceNumber)
                                                              throws Exception

Blocking API Derived classes must implement this method to store the state of RMDelegationToken and sequence number

Throws:

Exception

removeRMDelegationToken

public void removeRMDelegationToken(org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier rmDTIdentifier,
                                    int sequenceNumber)

RMDTSecretManager call this to remove the state of a delegation token

removeRMDelegationTokenState

protected abstract void removeRMDelegationTokenState(org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier rmDTIdentifier)
                                              throws Exception

Blocking API Derived classes must implement this method to remove the state of RMDelegationToken

Throws:

Exception

updateRMDelegationTokenAndSequenceNumber

public void updateRMDelegationTokenAndSequenceNumber(org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier rmDTIdentifier,
                                                     Long renewDate,
                                                     int latestSequenceNumber)

RMDTSecretManager call this to update the state of a delegation token and sequence number

updateRMDelegationTokenAndSequenceNumberInternal

protected abstract void updateRMDelegationTokenAndSequenceNumberInternal(org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier rmDTIdentifier,
                                                                         Long renewDate,
                                                                         int latestSequenceNumber)
                                                                  throws Exception

Blocking API Derived classes must implement this method to update the state of RMDelegationToken and sequence number

Throws:

Exception

storeRMDTMasterKey

public void storeRMDTMasterKey(org.apache.hadoop.security.token.delegation.DelegationKey delegationKey)

RMDTSecretManager call this to store the state of a master key

storeRMDTMasterKeyState

protected abstract void storeRMDTMasterKeyState(org.apache.hadoop.security.token.delegation.DelegationKey delegationKey)
                                         throws Exception

Blocking API Derived classes must implement this method to store the state of DelegationToken Master Key

Throws:

Exception

removeRMDTMasterKey

public void removeRMDTMasterKey(org.apache.hadoop.security.token.delegation.DelegationKey delegationKey)

RMDTSecretManager call this to remove the state of a master key

removeRMDTMasterKeyState

protected abstract void removeRMDTMasterKeyState(org.apache.hadoop.security.token.delegation.DelegationKey delegationKey)
                                          throws Exception

Blocking API Derived classes must implement this method to remove the state of DelegationToken Master Key

Throws:

Exception

storeOrUpdateAMRMTokenSecretManagerState

public abstract void storeOrUpdateAMRMTokenSecretManagerState(AMRMTokenSecretManagerState amrmTokenSecretManagerState,
                                                              boolean isUpdate)

Blocking API Derived classes must implement this method to store or update the state of AMRMToken Master Key

removeApplication

public void removeApplication(RMApp app)

Non-blocking API ResourceManager services call this to remove an application from the state store This does not block the dispatcher threads There is no notification of completion for this operation.

removeApplicationStateInternal

protected abstract void removeApplicationStateInternal(RMStateStore.ApplicationState appState)
                                                throws Exception

Blocking API Derived classes must implement this method to remove the state of an application and its attempts

Throws:

Exception

getCredentialsFromAppAttempt

public org.apache.hadoop.security.Credentials getCredentialsFromAppAttempt(RMAppAttempt appAttempt)

handleStoreEvent

protected void handleStoreEvent(RMStateStoreEvent event)

notifyStoreOperationFailed

protected void notifyStoreOperationFailed(Exception failureCause)

deleteStore

public abstract void deleteStore()
                          throws Exception

Derived classes must implement this method to delete the state store

Throws:

Exception

setResourceManager

public void setResourceManager(ResourceManager rm)