org.apache.hadoop.fs.s3a.audit.impl

Class ActiveAuditManagerS3A

java.lang.Object

org.apache.hadoop.service.AbstractService

org.apache.hadoop.service.CompositeService

org.apache.hadoop.fs.s3a.audit.impl.ActiveAuditManagerS3A

All Implemented Interfaces:

Closeable, AutoCloseable, AuditManagerS3A, AWSAuditEventCallbacks, org.apache.hadoop.fs.store.audit.ActiveThreadSpanSource<AuditSpanS3A>, org.apache.hadoop.fs.store.audit.AuditSpanSource<AuditSpanS3A>, org.apache.hadoop.service.Service

@InterfaceAudience.Private
public final class ActiveAuditManagerS3A
extends org.apache.hadoop.service.CompositeService
implements AuditManagerS3A

Thread management for the active audit. This should be created by whatever wants to have active audit span tracking. It creates and starts the actual OperationAuditor for auditing. It then stores the thread-local span and returns a wrapping span. When the wrapper is closed/deactivated it will deactivate the wrapped span and then switch the active span to the unbounded span. The inner class AWSAuditEventCallbacks is returned as a request handler in createRequestHandlers(); this forwards all requests to the outer ActiveAuditManagerS3A, which then locates the active span and forwards the request. If any such invocation raises an AuditFailureException then the IOStatistics counter for AUDIT_FAILURE is incremented.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.service.CompositeService

org.apache.hadoop.service.CompositeService.CompositeServiceShutdownHook

Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

org.apache.hadoop.service.Service.STATE

Field Summary

Fields

Modifier and Type	Field and Description
static String	AUDIT_MANAGER_OPERATION
static String	NOT_A_WRAPPED_SPAN
static org.apache.hadoop.fs.store.LogExactlyOnce	WARN_OF_SPAN_TYPE One of logger for warnings about span retrieval.

Fields inherited from class org.apache.hadoop.service.CompositeService

STOP_ONLY_STARTED_SERVICES

Constructor Summary

Constructors

Constructor and Description
ActiveAuditManagerS3A(org.apache.hadoop.fs.statistics.impl.IOStatisticsStore iostatistics) Instantiate.

Method Summary

Modifier and Type	Method and Description
void	afterAttempt(com.amazonaws.handlers.HandlerAfterAttemptContext context) After any attempt is made.
void	afterError(com.amazonaws.Request<?> request, com.amazonaws.Response<?> response, Exception exception) Forward to active span.
void	afterResponse(com.amazonaws.Request<?> request, com.amazonaws.Response<?> response) Forward to active span.
void	beforeAttempt(com.amazonaws.handlers.HandlerBeforeAttemptContext context) Before any attempt is made.
<T extends com.amazonaws.AmazonWebServiceRequest> T	beforeExecution(T request) Forward to the active span.
com.amazonaws.AmazonWebServiceRequest	beforeMarshalling(com.amazonaws.AmazonWebServiceRequest request) Request before marshalling.
void	beforeRequest(com.amazonaws.Request<?> request) Request before marshalling.
com.amazonaws.http.HttpResponse	beforeUnmarshalling(com.amazonaws.Request<?> request, com.amazonaws.http.HttpResponse httpResponse) Before unmarshalling the response.
boolean	checkAccess(org.apache.hadoop.fs.Path path, S3AFileStatus status, org.apache.hadoop.fs.permission.FsAction mode) Check for permission to access a path.
List<com.amazonaws.handlers.RequestHandler2>	createRequestHandlers() Return a request handler for the AWS SDK which relays to this class.
AuditSpanS3A	createSpan(String operation, String path1, String path2) Start an operation; as well as invoking the audit service to do this, sets the operation as the active operation for this thread.
com.amazonaws.services.s3.transfer.internal.TransferStateChangeListener	createStateChangeListener() Return a transfer state change callback which fixes the active span context to be that in which the state change listener was created.
AuditSpanS3A	getActiveAuditSpan() Return the active wrapped span.
OperationAuditor	getAuditor() Get the auditor; valid once initialized.
String	getOperationName() Get the name of the operation.
String	getSpanId() The Span ID in the audit manager is the ID of the auditor, which can be used in the filesystem toString() method to assist in correlating client logs with S3 logs.
<T extends com.amazonaws.AmazonWebServiceRequest> T	requestCreated(T request) Attach a reference to the active thread span, then invoke the same callback on that active thread.
protected void	serviceInit(org.apache.hadoop.conf.Configuration conf)
protected void	serviceStart() After starting the auditor, it is queried for its unbonded span, which is then wrapped and stored for use.
String	toString()

Methods inherited from class org.apache.hadoop.service.CompositeService

addIfService, addService, getServices, removeService, serviceStop

Methods inherited from class org.apache.hadoop.service.AbstractService

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, setConfig, start, stop, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.service.Service

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, registerServiceListener, start, stop, unregisterServiceListener, waitForServiceToStop

Field Detail

WARN_OF_SPAN_TYPE

public static final org.apache.hadoop.fs.store.LogExactlyOnce WARN_OF_SPAN_TYPE

One of logger for warnings about span retrieval.

AUDIT_MANAGER_OPERATION

public static final String AUDIT_MANAGER_OPERATION

See Also:

Constant Field Values

NOT_A_WRAPPED_SPAN

public static final String NOT_A_WRAPPED_SPAN

See Also:

Constant Field Values

Constructor Detail

ActiveAuditManagerS3A

public ActiveAuditManagerS3A(org.apache.hadoop.fs.statistics.impl.IOStatisticsStore iostatistics)

Instantiate.

Parameters:

iostatistics - statistics target

Method Detail

serviceInit

protected void serviceInit(org.apache.hadoop.conf.Configuration conf)
                    throws Exception

Overrides:

serviceInit in class org.apache.hadoop.service.CompositeService

Throws:

Exception

serviceStart

protected void serviceStart()
                     throws Exception

After starting the auditor, it is queried for its unbonded span, which is then wrapped and stored for use.

Overrides:

serviceStart in class org.apache.hadoop.service.CompositeService

Throws:

Exception

toString

public String toString()

Overrides:

toString in class org.apache.hadoop.service.AbstractService

getAuditor

public OperationAuditor getAuditor()

Description copied from interface: AuditManagerS3A

Get the auditor; valid once initialized.

Specified by:

getAuditor in interface AuditManagerS3A

Returns:

the auditor.

getActiveAuditSpan

public AuditSpanS3A getActiveAuditSpan()

Return the active wrapped span.

Specified by:

getActiveAuditSpan in interface org.apache.hadoop.fs.store.audit.ActiveThreadSpanSource<AuditSpanS3A>

Returns:

a span.

getSpanId

public String getSpanId()

The Span ID in the audit manager is the ID of the auditor, which can be used in the filesystem toString() method to assist in correlating client logs with S3 logs. It is returned here as part of the implementation of AWSAuditEventCallbacks.

Specified by:

getSpanId in interface AWSAuditEventCallbacks

Returns:

the unique ID of the FS.

getOperationName

public String getOperationName()

Description copied from interface: AWSAuditEventCallbacks

Get the name of the operation.

Specified by:

getOperationName in interface AWSAuditEventCallbacks

Returns:

the operation name.

createSpan

public AuditSpanS3A createSpan(String operation,
                               @Nullable
                               String path1,
                               @Nullable
                               String path2)
                        throws IOException

Start an operation; as well as invoking the audit service to do this, sets the operation as the active operation for this thread.

Specified by:

createSpan in interface org.apache.hadoop.fs.store.audit.AuditSpanSource<AuditSpanS3A>

Parameters:

operation - operation name.

path1 - first path of operation

path2 - second path of operation

Returns:

a wrapped audit span

Throws:

IOException - failure

createRequestHandlers

public List<com.amazonaws.handlers.RequestHandler2> createRequestHandlers()
                                                                   throws IOException

Return a request handler for the AWS SDK which relays to this class.

Specified by:

createRequestHandlers in interface AuditManagerS3A

Returns:

a request handler.

Throws:

IOException - failure.

createStateChangeListener

public com.amazonaws.services.s3.transfer.internal.TransferStateChangeListener createStateChangeListener()

Description copied from interface: AuditManagerS3A

Return a transfer state change callback which fixes the active span context to be that in which the state change listener was created. This can be used to audit the creation of the multipart upload initiation request which the transfer manager makes when a file to be copied is split up. This must be invoked/used within the active span.

Specified by:

createStateChangeListener in interface AuditManagerS3A

Returns:

a state change listener.

checkAccess

public boolean checkAccess(org.apache.hadoop.fs.Path path,
                           S3AFileStatus status,
                           org.apache.hadoop.fs.permission.FsAction mode)
                    throws IOException

Description copied from interface: AuditManagerS3A

Check for permission to access a path. The path is fully qualified and the status is the status of the path. This is called from the FileSystem.access() command and is a soft permission check used by Hive.

Specified by:

checkAccess in interface AuditManagerS3A

Parameters:

path - path to check

status - status of the path.

mode - access mode.

Returns:

true if access is allowed.

Throws:

IOException - failure

requestCreated

public <T extends com.amazonaws.AmazonWebServiceRequest> T requestCreated(T request)

Attach a reference to the active thread span, then invoke the same callback on that active thread.

Specified by:

requestCreated in interface AWSAuditEventCallbacks

Type Parameters:

T - type of request

Parameters:

request - request request.

Returns:

the request, possibly modified.

beforeExecution

public <T extends com.amazonaws.AmazonWebServiceRequest> T beforeExecution(T request)

Forward to the active span. All invocations increment the statistics counter for Statistic.AUDIT_REQUEST_EXECUTION; failures will also increment Statistic.AUDIT_FAILURE; Preflight preparation of AWS request.

Specified by:

beforeExecution in interface AWSAuditEventCallbacks

Type Parameters:

T - type of request

Parameters:

request - request

Returns:

an updated request.

afterResponse

public void afterResponse(com.amazonaws.Request<?> request,
                          com.amazonaws.Response<?> response)
                   throws AuditFailureException,
                          com.amazonaws.SdkBaseException

Forward to active span.

Specified by:

afterResponse in interface AWSAuditEventCallbacks

Parameters:

request - request

response - response.

Throws:

AuditFailureException - for generic audit failures

com.amazonaws.SdkBaseException - for other reasons.

afterError

public void afterError(com.amazonaws.Request<?> request,
                       com.amazonaws.Response<?> response,
                       Exception exception)
                throws AuditFailureException,
                       com.amazonaws.SdkBaseException

Forward to active span.

Specified by:

afterError in interface AWSAuditEventCallbacks

Parameters:

request - request

response - response.

exception - exception raised.

Throws:

AuditFailureException - for generic audit failures

com.amazonaws.SdkBaseException - for other reasons.

beforeMarshalling

public com.amazonaws.AmazonWebServiceRequest beforeMarshalling(com.amazonaws.AmazonWebServiceRequest request)

Description copied from interface: AWSAuditEventCallbacks

Request before marshalling.

Specified by:

beforeMarshalling in interface AWSAuditEventCallbacks

Parameters:

request - request

Returns:

possibly modified request.

beforeRequest

public void beforeRequest(com.amazonaws.Request<?> request)

Description copied from interface: AWSAuditEventCallbacks

Request before marshalling.

Specified by:

beforeRequest in interface AWSAuditEventCallbacks

Parameters:

request - request

beforeAttempt

public void beforeAttempt(com.amazonaws.handlers.HandlerBeforeAttemptContext context)

Description copied from interface: AWSAuditEventCallbacks

Before any attempt is made.

Specified by:

beforeAttempt in interface AWSAuditEventCallbacks

Parameters:

context - full context, including the request.

afterAttempt

public void afterAttempt(com.amazonaws.handlers.HandlerAfterAttemptContext context)

Description copied from interface: AWSAuditEventCallbacks

After any attempt is made.

Specified by:

afterAttempt in interface AWSAuditEventCallbacks

Parameters:

context - full context, including the request.

beforeUnmarshalling

public com.amazonaws.http.HttpResponse beforeUnmarshalling(com.amazonaws.Request<?> request,
                                                           com.amazonaws.http.HttpResponse httpResponse)

Description copied from interface: AWSAuditEventCallbacks

Before unmarshalling the response.

Specified by:

beforeUnmarshalling in interface AWSAuditEventCallbacks

Parameters:

request - request made.

httpResponse - response received

Returns:

updated response.