org.apache.hadoop.fs.s3a.audit.impl

Class ActiveAuditManagerS3A

java.lang.Object

org.apache.hadoop.service.AbstractService

org.apache.hadoop.service.CompositeService

org.apache.hadoop.fs.s3a.audit.impl.ActiveAuditManagerS3A

All Implemented Interfaces:

Closeable, AutoCloseable, AuditManagerS3A, AWSAuditEventCallbacks, org.apache.hadoop.fs.store.audit.ActiveThreadSpanSource<AuditSpanS3A>, org.apache.hadoop.fs.store.audit.AuditSpanSource<AuditSpanS3A>, org.apache.hadoop.service.Service, software.amazon.awssdk.core.interceptor.ExecutionInterceptor

@InterfaceAudience.Private
public final class ActiveAuditManagerS3A
extends org.apache.hadoop.service.CompositeService
implements AuditManagerS3A

Thread management for the active audit. This should be created by whatever wants to have active audit span tracking. It creates and starts the actual OperationAuditor for auditing. It then stores the thread-local span and returns a wrapping span. When the wrapper is closed/deactivated it will deactivate the wrapped span and then switch the active span to the unbounded span. This class also implements ExecutionInterceptor and returns itself in createExecutionInterceptors(); once registered with the S3 client, the implemented methods will be called during different parts of an SDK request lifecycle, which then locate the active span and forward the request. If any such invocation raises an AuditFailureException then the IOStatistics counter for AUDIT_FAILURE is incremented. Uses the WeakReferenceThreadMap to store spans for threads. Provided a calling class retains a reference to the span, the active span will be retained.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.service.CompositeService

org.apache.hadoop.service.CompositeService.CompositeServiceShutdownHook

Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

org.apache.hadoop.service.Service.STATE

Field Summary

Fields

Modifier and Type	Field and Description
static String	AUDIT_MANAGER_OPERATION
static String	NOT_A_WRAPPED_SPAN
static org.apache.hadoop.fs.store.LogExactlyOnce	WARN_OF_SPAN_TYPE One of logger for warnings about span retrieval.

Fields inherited from class org.apache.hadoop.service.CompositeService

STOP_ONLY_STARTED_SERVICES

Constructor Summary

Constructors

Constructor and Description
ActiveAuditManagerS3A(org.apache.hadoop.fs.statistics.impl.IOStatisticsStore iostatistics) Instantiate.

Method Summary

Modifier and Type	Method and Description
void	afterExecution(software.amazon.awssdk.core.interceptor.Context.AfterExecution context, software.amazon.awssdk.core.interceptor.ExecutionAttributes executionAttributes) Forward to active span.
void	afterMarshalling(software.amazon.awssdk.core.interceptor.Context.AfterMarshalling context, software.amazon.awssdk.core.interceptor.ExecutionAttributes executionAttributes)
void	afterTransmission(software.amazon.awssdk.core.interceptor.Context.AfterTransmission context, software.amazon.awssdk.core.interceptor.ExecutionAttributes executionAttributes)
void	afterUnmarshalling(software.amazon.awssdk.core.interceptor.Context.AfterUnmarshalling context, software.amazon.awssdk.core.interceptor.ExecutionAttributes executionAttributes)
void	beforeExecution(software.amazon.awssdk.core.interceptor.Context.BeforeExecution context, software.amazon.awssdk.core.interceptor.ExecutionAttributes executionAttributes) Forward to the active span.
void	beforeMarshalling(software.amazon.awssdk.core.interceptor.Context.BeforeMarshalling context, software.amazon.awssdk.core.interceptor.ExecutionAttributes executionAttributes)
void	beforeTransmission(software.amazon.awssdk.core.interceptor.Context.BeforeTransmission context, software.amazon.awssdk.core.interceptor.ExecutionAttributes executionAttributes)
void	beforeUnmarshalling(software.amazon.awssdk.core.interceptor.Context.BeforeUnmarshalling context, software.amazon.awssdk.core.interceptor.ExecutionAttributes executionAttributes)
boolean	checkAccess(org.apache.hadoop.fs.Path path, S3AFileStatus status, org.apache.hadoop.fs.permission.FsAction mode) Check for permission to access a path.
List<software.amazon.awssdk.core.interceptor.ExecutionInterceptor>	createExecutionInterceptors() Return a list of execution interceptors for the AWS SDK which relays to this class.
AuditSpanS3A	createSpan(String operation, String path1, String path2) Start an operation; as well as invoking the audit service to do this, sets the operation as the active operation for this thread.
software.amazon.awssdk.transfer.s3.progress.TransferListener	createTransferListener() Return a transfer callback which fixes the active span context to be that in which the transfer listener was created.
AuditSpanS3A	getActiveAuditSpan() Return the active wrapped span.
OperationAuditor	getAuditor() Get the auditor; valid once initialized.
String	getOperationName() Get the name of the operation.
String	getSpanId() The Span ID in the audit manager is the ID of the auditor, which can be used in the filesystem toString() method to assist in correlating client logs with S3 logs.
software.amazon.awssdk.http.SdkHttpRequest	modifyHttpRequest(software.amazon.awssdk.core.interceptor.Context.ModifyHttpRequest context, software.amazon.awssdk.core.interceptor.ExecutionAttributes executionAttributes)
software.amazon.awssdk.http.SdkHttpResponse	modifyHttpResponse(software.amazon.awssdk.core.interceptor.Context.ModifyHttpResponse context, software.amazon.awssdk.core.interceptor.ExecutionAttributes executionAttributes)
software.amazon.awssdk.core.SdkRequest	modifyRequest(software.amazon.awssdk.core.interceptor.Context.ModifyRequest context, software.amazon.awssdk.core.interceptor.ExecutionAttributes executionAttributes)
software.amazon.awssdk.core.SdkResponse	modifyResponse(software.amazon.awssdk.core.interceptor.Context.ModifyResponse context, software.amazon.awssdk.core.interceptor.ExecutionAttributes executionAttributes)
void	onExecutionFailure(software.amazon.awssdk.core.interceptor.Context.FailedExecution context, software.amazon.awssdk.core.interceptor.ExecutionAttributes executionAttributes) Forward to active span.
void	requestCreated(software.amazon.awssdk.core.SdkRequest.Builder builder) Audit the creation of a request and retrieve a reference to the active thread span.
protected void	serviceInit(org.apache.hadoop.conf.Configuration conf)
protected void	serviceStart() After starting the auditor, it is queried for its unbonded span, which is then wrapped and stored for use.
protected void	serviceStop()
String	toString()

Methods inherited from class org.apache.hadoop.service.CompositeService

addIfService, addService, getServices, removeService

Methods inherited from class org.apache.hadoop.service.AbstractService

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, setConfig, start, stop, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.service.Service

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, registerServiceListener, start, stop, unregisterServiceListener, waitForServiceToStop

Methods inherited from interface software.amazon.awssdk.core.interceptor.ExecutionInterceptor

modifyAsyncHttpContent, modifyAsyncHttpResponseContent, modifyException, modifyHttpContent, modifyHttpResponseContent

Field Detail

WARN_OF_SPAN_TYPE

public static final org.apache.hadoop.fs.store.LogExactlyOnce WARN_OF_SPAN_TYPE

One of logger for warnings about span retrieval.

AUDIT_MANAGER_OPERATION

public static final String AUDIT_MANAGER_OPERATION

See Also:

Constant Field Values

NOT_A_WRAPPED_SPAN

public static final String NOT_A_WRAPPED_SPAN

See Also:

Constant Field Values

Constructor Detail

ActiveAuditManagerS3A

public ActiveAuditManagerS3A(org.apache.hadoop.fs.statistics.impl.IOStatisticsStore iostatistics)

Instantiate.

Parameters:

iostatistics - statistics target

Method Detail

serviceInit

protected void serviceInit(org.apache.hadoop.conf.Configuration conf)
                    throws Exception

Overrides:

serviceInit in class org.apache.hadoop.service.CompositeService

Throws:

Exception

serviceStart

protected void serviceStart()
                     throws Exception

After starting the auditor, it is queried for its unbonded span, which is then wrapped and stored for use.

Overrides:

serviceStart in class org.apache.hadoop.service.CompositeService

Throws:

Exception

serviceStop

protected void serviceStop()
                    throws Exception

Overrides:

serviceStop in class org.apache.hadoop.service.CompositeService

Throws:

Exception

toString

public String toString()

Overrides:

toString in class org.apache.hadoop.service.AbstractService

getAuditor

public OperationAuditor getAuditor()

Description copied from interface: AuditManagerS3A

Get the auditor; valid once initialized.

Specified by:

getAuditor in interface AuditManagerS3A

Returns:

the auditor.

getActiveAuditSpan

public AuditSpanS3A getActiveAuditSpan()

Return the active wrapped span.

Specified by:

getActiveAuditSpan in interface org.apache.hadoop.fs.store.audit.ActiveThreadSpanSource<AuditSpanS3A>

Returns:

a span.

getSpanId

public String getSpanId()

The Span ID in the audit manager is the ID of the auditor, which can be used in the filesystem toString() method to assist in correlating client logs with S3 logs. It is returned here as part of the implementation of AWSAuditEventCallbacks.

Specified by:

getSpanId in interface AWSAuditEventCallbacks

Returns:

the unique ID of the FS.

getOperationName

public String getOperationName()

Description copied from interface: AWSAuditEventCallbacks

Get the name of the operation.

Specified by:

getOperationName in interface AWSAuditEventCallbacks

Returns:

the operation name.

createSpan

public AuditSpanS3A createSpan(String operation,
                               @Nullable
                               String path1,
                               @Nullable
                               String path2)
                        throws IOException

Start an operation; as well as invoking the audit service to do this, sets the operation as the active operation for this thread.

Specified by:

createSpan in interface org.apache.hadoop.fs.store.audit.AuditSpanSource<AuditSpanS3A>

Parameters:

operation - operation name.

path1 - first path of operation

path2 - second path of operation

Returns:

a wrapped audit span

Throws:

IOException - failure

createExecutionInterceptors

public List<software.amazon.awssdk.core.interceptor.ExecutionInterceptor> createExecutionInterceptors()
                                                                                               throws IOException

Return a list of execution interceptors for the AWS SDK which relays to this class.

Specified by:

createExecutionInterceptors in interface AuditManagerS3A

Returns:

a list of execution interceptors.

Throws:

IOException - failure.

createTransferListener

public software.amazon.awssdk.transfer.s3.progress.TransferListener createTransferListener()

Description copied from interface: AuditManagerS3A

Return a transfer callback which fixes the active span context to be that in which the transfer listener was created. This can be used to audit the creation of the multipart upload initiation request which the transfer manager makes when a file to be copied is split up. This must be invoked/used within the active span.

Specified by:

createTransferListener in interface AuditManagerS3A

Returns:

a transfer listener.

checkAccess

public boolean checkAccess(org.apache.hadoop.fs.Path path,
                           S3AFileStatus status,
                           org.apache.hadoop.fs.permission.FsAction mode)
                    throws IOException

Description copied from interface: AuditManagerS3A

Check for permission to access a path. The path is fully qualified and the status is the status of the path. This is called from the FileSystem.access() command and is a soft permission check used by Hive.

Specified by:

checkAccess in interface AuditManagerS3A

Parameters:

path - path to check

status - status of the path.

mode - access mode.

Returns:

true if access is allowed.

Throws:

IOException - failure

requestCreated

public void requestCreated(software.amazon.awssdk.core.SdkRequest.Builder builder)

Audit the creation of a request and retrieve a reference to the active thread span.

Specified by:

requestCreated in interface AWSAuditEventCallbacks

Parameters:

builder - the request builder.

beforeExecution

public void beforeExecution(software.amazon.awssdk.core.interceptor.Context.BeforeExecution context,
                            software.amazon.awssdk.core.interceptor.ExecutionAttributes executionAttributes)

Forward to the active span. All invocations increment the statistics counter for Statistic.AUDIT_REQUEST_EXECUTION; failures will also increment Statistic.AUDIT_FAILURE;

Specified by:

beforeExecution in interface software.amazon.awssdk.core.interceptor.ExecutionInterceptor

afterExecution

public void afterExecution(software.amazon.awssdk.core.interceptor.Context.AfterExecution context,
                           software.amazon.awssdk.core.interceptor.ExecutionAttributes executionAttributes)

Forward to active span.

Specified by:

afterExecution in interface software.amazon.awssdk.core.interceptor.ExecutionInterceptor

onExecutionFailure

public void onExecutionFailure(software.amazon.awssdk.core.interceptor.Context.FailedExecution context,
                               software.amazon.awssdk.core.interceptor.ExecutionAttributes executionAttributes)

Forward to active span.

Specified by:

onExecutionFailure in interface software.amazon.awssdk.core.interceptor.ExecutionInterceptor

Parameters:

context - execution context

executionAttributes - the execution attributes

modifyRequest

public software.amazon.awssdk.core.SdkRequest modifyRequest(software.amazon.awssdk.core.interceptor.Context.ModifyRequest context,
                                                            software.amazon.awssdk.core.interceptor.ExecutionAttributes executionAttributes)

Specified by:

modifyRequest in interface software.amazon.awssdk.core.interceptor.ExecutionInterceptor

beforeMarshalling

public void beforeMarshalling(software.amazon.awssdk.core.interceptor.Context.BeforeMarshalling context,
                              software.amazon.awssdk.core.interceptor.ExecutionAttributes executionAttributes)

Specified by:

beforeMarshalling in interface software.amazon.awssdk.core.interceptor.ExecutionInterceptor

afterMarshalling

public void afterMarshalling(software.amazon.awssdk.core.interceptor.Context.AfterMarshalling context,
                             software.amazon.awssdk.core.interceptor.ExecutionAttributes executionAttributes)

Specified by:

afterMarshalling in interface software.amazon.awssdk.core.interceptor.ExecutionInterceptor

modifyHttpRequest

public software.amazon.awssdk.http.SdkHttpRequest modifyHttpRequest(software.amazon.awssdk.core.interceptor.Context.ModifyHttpRequest context,
                                                                    software.amazon.awssdk.core.interceptor.ExecutionAttributes executionAttributes)

Specified by:

modifyHttpRequest in interface software.amazon.awssdk.core.interceptor.ExecutionInterceptor

beforeTransmission

public void beforeTransmission(software.amazon.awssdk.core.interceptor.Context.BeforeTransmission context,
                               software.amazon.awssdk.core.interceptor.ExecutionAttributes executionAttributes)

Specified by:

beforeTransmission in interface software.amazon.awssdk.core.interceptor.ExecutionInterceptor

afterTransmission

public void afterTransmission(software.amazon.awssdk.core.interceptor.Context.AfterTransmission context,
                              software.amazon.awssdk.core.interceptor.ExecutionAttributes executionAttributes)

Specified by:

afterTransmission in interface software.amazon.awssdk.core.interceptor.ExecutionInterceptor

modifyHttpResponse

public software.amazon.awssdk.http.SdkHttpResponse modifyHttpResponse(software.amazon.awssdk.core.interceptor.Context.ModifyHttpResponse context,
                                                                      software.amazon.awssdk.core.interceptor.ExecutionAttributes executionAttributes)

Specified by:

modifyHttpResponse in interface software.amazon.awssdk.core.interceptor.ExecutionInterceptor

beforeUnmarshalling

public void beforeUnmarshalling(software.amazon.awssdk.core.interceptor.Context.BeforeUnmarshalling context,
                                software.amazon.awssdk.core.interceptor.ExecutionAttributes executionAttributes)

Specified by:

beforeUnmarshalling in interface software.amazon.awssdk.core.interceptor.ExecutionInterceptor

afterUnmarshalling

public void afterUnmarshalling(software.amazon.awssdk.core.interceptor.Context.AfterUnmarshalling context,
                               software.amazon.awssdk.core.interceptor.ExecutionAttributes executionAttributes)

Specified by:

afterUnmarshalling in interface software.amazon.awssdk.core.interceptor.ExecutionInterceptor

modifyResponse

public software.amazon.awssdk.core.SdkResponse modifyResponse(software.amazon.awssdk.core.interceptor.Context.ModifyResponse context,
                                                              software.amazon.awssdk.core.interceptor.ExecutionAttributes executionAttributes)

Specified by:

modifyResponse in interface software.amazon.awssdk.core.interceptor.ExecutionInterceptor