org.apache.hadoop.fs.s3a.auth.delegation

Class S3ADelegationTokens

java.lang.Object

org.apache.hadoop.service.AbstractService

org.apache.hadoop.fs.s3a.auth.delegation.AbstractDTService

org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.service.Service

@InterfaceAudience.Private
public class S3ADelegationTokens
extends AbstractDTService

Support for creating a DT from a filesystem. Isolated from S3A for control and testability. The S3A Delegation Tokens are special in that the tokens are not directly used to authenticate with the AWS services. Instead they can session/role credentials requested off AWS on demand. The design is extensible in that different back-end bindings can be used to switch to different session creation mechanisms, or indeed, to any other authentication mechanism supported by an S3 service, provided it ultimately accepts some form of AWS credentials for authentication through the AWS SDK. That is, if someone wants to wire this up to Kerberos, or OAuth2, this design should support them. URIs processed must be the canonical URIs for the service.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	S3ADelegationTokens.TokenIssuingPolicy How will tokens be issued on request? The S3ADelegationTokens.TokenIssuingPolicy.RequestNewToken policy does not guarantee that a tokens can be created, only that an attempt will be made to request one.

Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

org.apache.hadoop.service.Service.STATE

Field Summary

Fields

Modifier and Type	Field and Description
protected static EnumSet<AWSPolicyProvider.AccessLevel>	ACCESS_POLICY The access policies we want for operations.
static String	E_DELEGATION_TOKENS_DISABLED

Constructor Summary

Constructors

Constructor and Description
S3ADelegationTokens() Instantiate.

Method Summary

Modifier and Type	Method and Description
void	bindToDelegationToken(org.apache.hadoop.security.token.Token<AbstractS3ATokenIdentifier> token) Bind to a delegation token retrieved for this filesystem.
void	bindToFileSystem(URI uri, StoreContext context, DelegationOperations delegationOperations) Bind to the filesystem.
org.apache.hadoop.security.token.Token<AbstractS3ATokenIdentifier>	createDelegationToken(EncryptionSecrets encryptionSecrets, org.apache.hadoop.io.Text renewer) Create a delegation token for the user.
AbstractS3ATokenIdentifier	extractIdentifier(org.apache.hadoop.security.token.Token<? extends AbstractS3ATokenIdentifier> token) From a token, get the session token identifier.
org.apache.hadoop.security.token.DelegationTokenIssuer[]	getAdditionalTokenIssuers() Get a null/possibly empty list of extra delegation token issuers.
Optional<org.apache.hadoop.security.token.Token<AbstractS3ATokenIdentifier>>	getBoundDT() Get any bound DT.
org.apache.hadoop.security.token.Token<AbstractS3ATokenIdentifier>	getBoundOrNewDT(EncryptionSecrets encryptionSecrets, org.apache.hadoop.io.Text renewer) Get any bound DT or create a new one.
String	getCanonicalServiceName() The canonical name of the service.
int	getCreationCount() How many delegation tokens have been issued?
AWSCredentialProviderList	getCredentialProviders() Get the AWS credential provider.
Optional<AbstractS3ATokenIdentifier>	getDecodedIdentifier() Get any decoded identifier from the bound DT; empty if not bound.
Optional<EncryptionSecrets>	getEncryptionSecrets() Get the encryption secrets of the DT.
org.apache.hadoop.io.Text	getService() Get the service identifier of the owning FS.
S3ADelegationTokens.TokenIssuingPolicy	getTokenIssuingPolicy() Predicate: will this binding issue a DT if requested in a call to getBoundOrNewDT(EncryptionSecrets, Text)? That is: should the filesystem declare that it is issuing delegation tokens?
org.apache.hadoop.io.Text	getTokenKind() Get the kind of the issued tokens.
String	getUserAgentField() Return a string for use in building up the User-Agent field, so get into the S3 access logs.
static boolean	hasDelegationTokenBinding(org.apache.hadoop.conf.Configuration conf) Predicate: does this configuration enable delegation tokens? That is: is there any text in the option DelegationConstants.DELEGATION_TOKEN_BINDING ?
boolean	isBoundToDT() Predicate: is there a bound DT?
static org.apache.hadoop.security.token.Token<AbstractS3ATokenIdentifier>	lookupS3ADelegationToken(org.apache.hadoop.security.Credentials credentials, URI uri) Look for any S3A token for the given FS service.
static org.apache.hadoop.security.token.Token<AbstractS3ATokenIdentifier>	lookupToken(org.apache.hadoop.security.Credentials credentials, org.apache.hadoop.io.Text service) Look up any token from the service; cast it to one of ours.
static org.apache.hadoop.security.token.Token<AbstractS3ATokenIdentifier>	lookupToken(org.apache.hadoop.security.Credentials credentials, org.apache.hadoop.io.Text service, org.apache.hadoop.io.Text kind) Look up a token from the credentials, verify it is of the correct kind.
org.apache.hadoop.security.token.Token<AbstractS3ATokenIdentifier>	selectTokenFromFSOwner() Find a token for the FS user and canonical filesystem URI.
protected void	serviceInit(org.apache.hadoop.conf.Configuration conf) Init the service.
protected void	serviceStart() Service startup includes binding to any delegation token, and deploying unbounded if there is none.
protected void	serviceStop() Stop the token binding.
String	toString()

Methods inherited from class org.apache.hadoop.fs.s3a.auth.delegation.AbstractDTService

getCanonicalUri, getOwner, getPolicyProvider, getStoreContext, requireServiceStarted, requireServiceState

Methods inherited from class org.apache.hadoop.service.AbstractService

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, setConfig, start, stop, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

E_DELEGATION_TOKENS_DISABLED

public static final String E_DELEGATION_TOKENS_DISABLED

See Also:

Constant Field Values

ACCESS_POLICY

protected static final EnumSet<AWSPolicyProvider.AccessLevel> ACCESS_POLICY

The access policies we want for operations. There's no attempt to ask for "admin" permissions here.

Constructor Detail

S3ADelegationTokens

public S3ADelegationTokens()
                    throws IOException

Instantiate.

Throws:

IOException - if login fails.

Method Detail

bindToFileSystem

public void bindToFileSystem(URI uri,
                             StoreContext context,
                             DelegationOperations delegationOperations)
                      throws IOException

Description copied from class: AbstractDTService

Bind to the filesystem. Subclasses can use this to perform their own binding operations - but they must always call their superclass implementation. This Must be called before calling init(). Important: This binding will happen during FileSystem.initialize(); the FS is not live for actual use and will not yet have interacted with AWS services.

Overrides:

bindToFileSystem in class AbstractDTService

Parameters:

uri - the canonical URI of the FS.

context - store context

delegationOperations - delegation operations

Throws:

IOException - failure.

serviceInit

protected void serviceInit(org.apache.hadoop.conf.Configuration conf)
                    throws Exception

Init the service. This identifies the token binding class to use and creates, initializes and starts it. Will raise an exception if delegation tokens are not enabled.

Overrides:

serviceInit in class AbstractDTService

Parameters:

conf - configuration

Throws:

Exception - any failure to start up

serviceStart

protected void serviceStart()
                     throws Exception

Service startup includes binding to any delegation token, and deploying unbounded if there is none. It is after this that token operations can be used.

Overrides:

serviceStart in class org.apache.hadoop.service.AbstractService

Throws:

Exception - any failure

serviceStop

protected void serviceStop()
                    throws Exception

Stop the token binding.

Overrides:

serviceStop in class org.apache.hadoop.service.AbstractService

Throws:

Exception - on any failure

bindToDelegationToken

@VisibleForTesting
public void bindToDelegationToken(org.apache.hadoop.security.token.Token<AbstractS3ATokenIdentifier> token)
                                              throws IOException

Bind to a delegation token retrieved for this filesystem. Extract the secrets from the token and set internal fields to the values.

1. boundDT is set to token.
2. decodedIdentifier is set to the extracted identifier.
3. bindingInfo is set to the info returned by the token binding.

Parameters:

token - token to decode and bind to.

Throws:

IOException - selection/extraction/validation failure.

isBoundToDT

public boolean isBoundToDT()

Predicate: is there a bound DT?

Returns:

true if there's a value in boundDT.

getBoundDT

public Optional<org.apache.hadoop.security.token.Token<AbstractS3ATokenIdentifier>> getBoundDT()

Get any bound DT.

Returns:

a delegation token if this instance was bound to it.

getTokenIssuingPolicy

public S3ADelegationTokens.TokenIssuingPolicy getTokenIssuingPolicy()

Predicate: will this binding issue a DT if requested in a call to getBoundOrNewDT(EncryptionSecrets, Text)? That is: should the filesystem declare that it is issuing delegation tokens?

Returns:

a declaration of what will happen when asked for a token.

getBoundOrNewDT

public org.apache.hadoop.security.token.Token<AbstractS3ATokenIdentifier> getBoundOrNewDT(EncryptionSecrets encryptionSecrets,
                                                                                          org.apache.hadoop.io.Text renewer)
                                                                                   throws IOException

Get any bound DT or create a new one.

Parameters:

encryptionSecrets - encryption secrets for any new token.

renewer - the token renewer.

Returns:

a delegation token.

Throws:

IOException - if one cannot be created

getCreationCount

public int getCreationCount()

How many delegation tokens have been issued?

Returns:

the number times createDelegationToken(EncryptionSecrets, Text) returned a token.

createDelegationToken

@VisibleForTesting
public org.apache.hadoop.security.token.Token<AbstractS3ATokenIdentifier> createDelegationToken(EncryptionSecrets encryptionSecrets,
                                                                                                                   org.apache.hadoop.io.Text renewer)
                                                                                                            throws IOException

Create a delegation token for the user. This will only be called if a new DT is needed, that is: the filesystem has been deployed unbonded.

Parameters:

encryptionSecrets - encryption secrets for the token.

renewer - the token renewer

Returns:

the token

Throws:

IOException - if one cannot be created

getAdditionalTokenIssuers

public org.apache.hadoop.security.token.DelegationTokenIssuer[] getAdditionalTokenIssuers()
                                                                                   throws IOException

Get a null/possibly empty list of extra delegation token issuers. These will be asked for tokens when DelegationTokenIssuer.getAdditionalTokenIssuers() recursively collects all DTs a filesystem can offer.

Returns:

a null or empty array. Default implementation: null

Throws:

IOException - failure

getCredentialProviders

public AWSCredentialProviderList getCredentialProviders()
                                                 throws IOException

Get the AWS credential provider.

Returns:

the DT credential provider

Throws:

IOException - failure to parse the DT

IllegalStateException - if this instance is not bound to a DT

getEncryptionSecrets

public Optional<EncryptionSecrets> getEncryptionSecrets()

Get the encryption secrets of the DT. non-empty iff service is started and was bound to a DT.

Returns:

any encryption settings propagated with the DT.

getDecodedIdentifier

public Optional<AbstractS3ATokenIdentifier> getDecodedIdentifier()

Get any decoded identifier from the bound DT; empty if not bound.

Returns:

the decoded identifier.

getService

public org.apache.hadoop.io.Text getService()

Get the service identifier of the owning FS.

Returns:

a service identifier to use when registering tokens

getCanonicalServiceName

public String getCanonicalServiceName()

The canonical name of the service. This can be used as the canonical service name for the FS.

Returns:

the canonicalized FS URI.

selectTokenFromFSOwner

@VisibleForTesting
public org.apache.hadoop.security.token.Token<AbstractS3ATokenIdentifier> selectTokenFromFSOwner()
                                                                                                             throws IOException

Find a token for the FS user and canonical filesystem URI.

Returns:

the token, or null if one cannot be found.

Throws:

IOException - on a failure to unmarshall the token.

toString

public String toString()

Overrides:

toString in class org.apache.hadoop.service.AbstractService

getTokenKind

public org.apache.hadoop.io.Text getTokenKind()

Get the kind of the issued tokens.

Returns:

token kind.

extractIdentifier

public AbstractS3ATokenIdentifier extractIdentifier(org.apache.hadoop.security.token.Token<? extends AbstractS3ATokenIdentifier> token)
                                             throws IOException

From a token, get the session token identifier.

Parameters:

token - token to process

Returns:

the session token identifier

Throws:

IOException - failure to validate/read data encoded in identifier.

IllegalArgumentException - if the token isn't an S3A session token

getUserAgentField

public String getUserAgentField()

Return a string for use in building up the User-Agent field, so get into the S3 access logs. Useful for diagnostics. Delegates to {AbstractDelegationTokenBinding.getUserAgentField()} for the current binding.

Returns:

a string for the S3 logs or "" for "nothing to add"

lookupToken

@VisibleForTesting
public static org.apache.hadoop.security.token.Token<AbstractS3ATokenIdentifier> lookupToken(org.apache.hadoop.security.Credentials credentials,
                                                                                                                org.apache.hadoop.io.Text service,
                                                                                                                org.apache.hadoop.io.Text kind)
                                                                                                         throws DelegationTokenIOException

Look up a token from the credentials, verify it is of the correct kind.

Parameters:

credentials - credentials to look up.

service - service name

kind - token kind to look for

Returns:

the token or null if no suitable token was found

Throws:

DelegationTokenIOException - wrong token kind found

lookupToken

public static org.apache.hadoop.security.token.Token<AbstractS3ATokenIdentifier> lookupToken(org.apache.hadoop.security.Credentials credentials,
                                                                                             org.apache.hadoop.io.Text service)

Look up any token from the service; cast it to one of ours.

Parameters:

credentials - credentials

service - service to look up

Returns:

any token found or null if none was

Throws:

ClassCastException - if the token is of a wrong type.

lookupS3ADelegationToken

public static org.apache.hadoop.security.token.Token<AbstractS3ATokenIdentifier> lookupS3ADelegationToken(org.apache.hadoop.security.Credentials credentials,
                                                                                                          URI uri)

Look for any S3A token for the given FS service.

Parameters:

credentials - credentials to scan.

uri - the URI of the FS to look for

Returns:

the token or null if none was found

hasDelegationTokenBinding

public static boolean hasDelegationTokenBinding(org.apache.hadoop.conf.Configuration conf)

Predicate: does this configuration enable delegation tokens? That is: is there any text in the option DelegationConstants.DELEGATION_TOKEN_BINDING ?

Parameters:

conf - configuration to examine

Returns:

true iff the trimmed configuration option is not empty.