S3AFileSystem (Apache Hadoop Amazon Web Services support 3.3.4 API)

java.lang.Object
- org.apache.hadoop.conf.Configured
- - org.apache.hadoop.fs.FileSystem
  - - org.apache.hadoop.fs.s3a.S3AFileSystem

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable, org.apache.hadoop.fs.PathCapabilities, AWSPolicyProvider, DelegationTokenProvider, org.apache.hadoop.fs.statistics.IOStatisticsSource, org.apache.hadoop.fs.store.audit.ActiveThreadSpanSource<AuditSpanS3A>, org.apache.hadoop.fs.store.audit.AuditSpanSource<AuditSpanS3A>, org.apache.hadoop.fs.StreamCapabilities, org.apache.hadoop.security.token.DelegationTokenIssuer
```
@InterfaceAudience.Private
 @InterfaceStability.Evolving
public class S3AFileSystem
extends org.apache.hadoop.fs.FileSystem
implements org.apache.hadoop.fs.StreamCapabilities, AWSPolicyProvider, DelegationTokenProvider, org.apache.hadoop.fs.statistics.IOStatisticsSource, org.apache.hadoop.fs.store.audit.AuditSpanSource<AuditSpanS3A>, org.apache.hadoop.fs.store.audit.ActiveThreadSpanSource<AuditSpanS3A>
```
The core S3A Filesystem implementation. This subclass is marked as private as code should not be creating it directly; use FileSystem.get(Configuration) and variants to create one. If cast to S3AFileSystem, extra methods and features may be accessed. Consider those private and unstable. Because it prints some of the state of the instrumentation, the output of toString() must also be considered unstable.

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`protected class`	`S3AFileSystem.CopyFromLocalCallbacksImpl`
`protected class`	`S3AFileSystem.GetContentSummaryCallbacksImpl` Callbacks from the `GetContentSummaryOperation`.
`protected class`	`S3AFileSystem.HeaderProcessingCallbacksImpl` Operations needed for Header Processing.
`protected class`	`S3AFileSystem.ListingOperationCallbacksImpl` Callbacks from `Listing`.
`protected class`	`S3AFileSystem.MkdirOperationCallbacksImpl` Callbacks from the `MkdirOperation`.

Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem
org.apache.hadoop.fs.FileSystem.DirectoryEntries, org.apache.hadoop.fs.FileSystem.DirListingIterator<T extends org.apache.hadoop.fs.FileStatus>, org.apache.hadoop.fs.FileSystem.Statistics

Nested classes/interfaces inherited from interface org.apache.hadoop.fs.StreamCapabilities
org.apache.hadoop.fs.StreamCapabilities.StreamCapability

Nested classes/interfaces inherited from interface org.apache.hadoop.fs.s3a.auth.delegation.AWSPolicyProvider
AWSPolicyProvider.AccessLevel

Field Summary

Fields
Modifier and Type Field and Description

static int DEFAULT_BLOCKSIZE
Default blocksize as used in blocksize and FS status queries.

static org.slf4j.Logger LOG
- Fields inherited from class org.apache.hadoop.fs.FileSystem
  DEFAULT_FS, FS_DEFAULT_NAME_KEY, SHUTDOWN_HOOK_PRIORITY, statistics, TRASH_PREFIX, USER_HOME_PREFIX
- Fields inherited from interface org.apache.hadoop.fs.StreamCapabilities
  ABORTABLE_STREAM, DROPBEHIND, HFLUSH, HSYNC, IOSTATISTICS, PREADBYTEBUFFER, READAHEAD, READBYTEBUFFER, UNBUFFER

Fields
Modifier and Type	Field and Description
`static int`	`DEFAULT_BLOCKSIZE` Default blocksize as used in blocksize and FS status queries.
`static org.slf4j.Logger`	`LOG`

Constructor Summary

Constructors
Constructor and Description

S3AFileSystem()

Constructors
Constructor and Description
`S3AFileSystem()`

Method Summary

All Methods Static Methods Instance Methods Concrete Methods Deprecated Methods
Modifier and Type	Method and Description
`void`	`abortOutstandingMultipartUploads(long seconds)` Abort all outstanding MPUs older than a given age.
`void`	`access(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsAction mode)` Soft check of access by forwarding to the audit manager and so on to the auditor.
`boolean`	`allowAuthoritative(org.apache.hadoop.fs.Path path)` Is a path to be considered as authoritative? True iff there is an authoritative metastore or if there is a non-auth store with the supplied path under one of the paths declared as authoritative.
`org.apache.hadoop.fs.FSDataOutputStream`	`append(org.apache.hadoop.fs.Path f, int bufferSize, org.apache.hadoop.util.Progressable progress)` Append to an existing file (optional operation).
`protected URI`	`canonicalizeUri(URI rawUri)` Override the base canonicalization logic and relay to `S3xLoginHelper.canonicalizeUri(URI, int)`.
`void`	`checkPath(org.apache.hadoop.fs.Path path)` Check that a Path belongs to this FileSystem.
`void`	`close()` Close the filesystem.
`protected S3ListResult`	`continueListObjects(S3ListRequest request, S3ListResult prevResult, org.apache.hadoop.fs.statistics.DurationTrackerFactory trackerFactory)` List the next set of objects.
`void`	`copyFromLocalFile(boolean delSrc, boolean overwrite, org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst)` The src file is on the local disk.
`org.apache.hadoop.fs.FSDataOutputStream`	`create(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress)` Create an FSDataOutputStream at the indicated Path with write-progress reporting.
`protected CopyFromLocalOperation.CopyFromLocalOperationCallbacks`	`createCopyFromLocalCallbacks()`
`DelegationOperations`	`createDelegationOperations()` Create an instance of the delegation operations.
`protected GetContentSummaryOperation.GetContentSummaryCallbacks`	`createGetContentSummaryCallbacks()` Override point: create the callbacks for getContentSummary.
`protected HeaderProcessing.HeaderProcessingCallbacks`	`createHeaderProcessingCallbacks()` Create the callbacks.
`S3ListRequest`	`createListObjectsRequest(String key, String delimiter)` Create a `ListObjectsRequest` request against this bucket, with the maximum keys returned in a query set by `maxKeys`.
`MarkerToolOperations`	`createMarkerToolOperations(String target)` Create a marker tools operations binding for this store.
`MkdirOperation.MkdirCallbacks`	`createMkdirOperationCallbacks()` Override point: create the callbacks for Mkdir.
`S3AMultipartUploaderBuilder`	`createMultipartUploader(org.apache.hadoop.fs.Path basePath)`
`org.apache.hadoop.fs.FSDataOutputStream`	`createNonRecursive(org.apache.hadoop.fs.Path p, org.apache.hadoop.fs.permission.FsPermission permission, EnumSet<org.apache.hadoop.fs.CreateFlag> flags, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress)`
`protected S3AReadOpContext`	`createReadContext(org.apache.hadoop.fs.FileStatus fileStatus, S3AInputPolicy seekPolicy, ChangeDetectionPolicy changePolicy, long readAheadRange, org.apache.hadoop.fs.store.audit.AuditSpan auditSpan)` Create the read context for reading from the referenced file, using FS state as well as the status.
`protected RequestFactory`	`createRequestFactory()` Build the request factory.
`AuditSpanS3A`	`createSpan(String operation, String path1, String path2)` Start an operation; this informs the audit service of the event and then sets it as the active span.
`protected static S3AStorageStatistics`	`createStorageStatistics(org.apache.hadoop.fs.statistics.IOStatistics ioStatistics)` Create the storage statistics or bind to an existing one.
`StoreContext`	`createStoreContext()` Build an immutable store context.
`WriteOperationHelper`	`createWriteOperationHelper(org.apache.hadoop.fs.store.audit.AuditSpan auditSpan)` Create a Write Operation Helper with the given span.
`protected void`	`decrementGauge(Statistic statistic, long count)` Decrement a gauge by a specific value.
`boolean`	`delete(org.apache.hadoop.fs.Path f, boolean recursive)` Delete a Path.
`protected void`	`deleteObject(String key)` Delete an object.
`protected org.apache.hadoop.fs.store.audit.AuditSpan`	`entryPoint(Statistic operation, org.apache.hadoop.fs.Path path)` Entry point to an operation.
`protected org.apache.hadoop.fs.store.audit.AuditSpan`	`entryPoint(Statistic operation, String path1, String path2)` Entry point to an operation.
`boolean`	`exists(org.apache.hadoop.fs.Path f)` Override superclass so as to add statistic collection.
`AuditSpanS3A`	`getActiveAuditSpan()` Get the active audit span.
`org.apache.hadoop.security.token.DelegationTokenIssuer[]`	`getAdditionalTokenIssuers()` Ask any DT plugin for any extra token issuers.
`com.amazonaws.services.s3.AmazonS3`	`getAmazonS3ClientForTesting(String reason)` Returns the S3 client used by this filesystem.
`AuditManagerS3A`	`getAuditManager()` The audit manager.
`OperationAuditor`	`getAuditor()` Get the auditor; valid once initialized.
`org.apache.hadoop.fs.store.audit.AuditSpanSource`	`getAuditSpanSource()` Get the audit span source; allows for components like the committers to have a source of spans without being hard coded to the FS only.
`String`	`getBucket()` Get the bucket of this filesystem.
`String`	`getBucketLocation()` Get the region of a bucket.
`String`	`getBucketLocation(String bucketName)` Get the region of a bucket; fixing up the region so it can be used in the builders of other AWS clients.
`String`	`getCanonicalServiceName()` Return a service name iff delegation tokens are enabled and the token binding is issuing delegation tokens.
`URI`	`getCanonicalUri()` Get the canonical URI.
`ChangeDetectionPolicy`	`getChangeDetectionPolicy()` Get the change detection policy for this FS instance.
`org.apache.hadoop.fs.ContentSummary`	`getContentSummary(org.apache.hadoop.fs.Path f)` This is a very slow operation against object storage.
`long`	`getDefaultBlockSize()` Deprecated. use `FileSystem.getDefaultBlockSize(Path)` instead
`int`	`getDefaultPort()`
`org.apache.hadoop.security.token.Token<AbstractS3ATokenIdentifier>`	`getDelegationToken(String renewer)` Get a delegation token if the FS is set up for them.
`Optional<S3ADelegationTokens>`	`getDelegationTokens()` Get the delegation token support for this filesystem; not null iff delegation support is enabled.
`DirectoryPolicy`	`getDirectoryMarkerPolicy()` Get the directory marker policy of this filesystem.
`protected org.apache.hadoop.fs.statistics.DurationTrackerFactory`	`getDurationTrackerFactory()` Get the factory for duration tracking.
`EncryptionSecrets`	`getEncryptionSecrets()` Get the encryption secrets.
`org.apache.hadoop.fs.store.EtagChecksum`	`getFileChecksum(org.apache.hadoop.fs.Path f, long length)` When enabled, get the etag of a object at the path via HEAD request and return it as a checksum object.
`org.apache.hadoop.fs.FileStatus`	`getFileStatus(org.apache.hadoop.fs.Path f)` Return a file status object that represents the path.
`org.apache.hadoop.security.token.Token<? extends org.apache.hadoop.security.token.TokenIdentifier>`	`getFsDelegationToken()`
`org.apache.hadoop.fs.FileSystem.Statistics`	`getFsStatistics()` Get FS Statistic for this S3AFS instance.
`S3AInputPolicy`	`getInputPolicy()` Get the input policy for this FS instance.
`S3AInstrumentation`	`getInstrumentation()` Get S3A Instrumentation.
`org.apache.hadoop.fs.statistics.IOStatistics`	`getIOStatistics()` Get the instrumentation's IOStatistics.
`Listing`	`getListing()` Get current listing instance.
`MetadataStore`	`getMetadataStore()` Get the metadata store.
`long`	`getMultiPartThreshold()` Get the threshold for multipart files.
`com.amazonaws.services.s3.model.ObjectMetadata`	`getObjectMetadata(org.apache.hadoop.fs.Path path)` Low-level call to get at the object metadata.
`protected com.amazonaws.services.s3.model.ObjectMetadata`	`getObjectMetadata(String key, ChangeTracker changeTracker, Invoker changeInvoker, String operation)` Request object metadata; increments counters in the process.
`org.apache.hadoop.security.UserGroupInformation`	`getOwner()` Get the owner of this FS: who created it?
`long`	`getPartitionSize()` Get the partition size for multipart operations.
`RequestFactory`	`getRequestFactory()` Get the request factory which uses this store's audit span.
`S3AEncryptionMethods`	`getS3EncryptionAlgorithm()` Get the encryption algorithm of this endpoint.
`String`	`getScheme()` Return the protocol scheme for the FileSystem.
`S3AStorageStatistics`	`getStorageStatistics()` Get the storage statistics of this filesystem.
`ITtlTimeProvider`	`getTtlTimeProvider()`
`URI`	`getUri()` Returns a URI whose scheme and authority identify this FileSystem.
`String`	`getUsername()` Get the username of the FS.
`org.apache.hadoop.fs.Path`	`getWorkingDirectory()` Get the current working directory for the given file system.
`WriteOperationHelper`	`getWriteOperationHelper()` Create a Write Operation Helper with the current active span.
`byte[]`	`getXAttr(org.apache.hadoop.fs.Path path, String name)`
`Map<String,byte[]>`	`getXAttrs(org.apache.hadoop.fs.Path path)`
`Map<String,byte[]>`	`getXAttrs(org.apache.hadoop.fs.Path path, List<String> names)`
`org.apache.hadoop.fs.FileStatus[]`	`globStatus(org.apache.hadoop.fs.Path pathPattern)` Increments the statistic `Statistic.INVOCATION_GLOB_STATUS`.
`org.apache.hadoop.fs.FileStatus[]`	`globStatus(org.apache.hadoop.fs.Path pathPattern, org.apache.hadoop.fs.PathFilter filter)` Increments the statistic `Statistic.INVOCATION_GLOB_STATUS`.
`boolean`	`hasAuthoritativeMetadataStore()` Does the filesystem have an authoritative metadata store?
`boolean`	`hasCapability(String capability)` Deprecated.
`boolean`	`hasMetadataStore()` Does this Filesystem have a metadata store?
`boolean`	`hasPathCapability(org.apache.hadoop.fs.Path path, String capability)`
`protected void`	`incrementGauge(Statistic statistic, long count)` Increment a gauge by a specific value.
`void`	`incrementPutCompletedStatistics(boolean success, long bytes)` At the end of a put/multipart upload operation, update the relevant counters and gauges.
`void`	`incrementPutProgressStatistics(String key, long bytes)` Callback for use in progress callbacks from put/multipart upload events.
`void`	`incrementPutStartStatistics(long bytes)` At the start of a put/multipart upload operation, update the relevant counters.
`void`	`incrementReadOperations()` Increment read operations.
`protected void`	`incrementStatistic(Statistic statistic)` Increment a statistic by 1.
`protected void`	`incrementStatistic(Statistic statistic, long count)` Increment a statistic by a specific value.
`void`	`incrementWriteOperations()` Increment the write operation counter.
`void`	`initialize(URI name, org.apache.hadoop.conf.Configuration originalConf)` Called after a new FileSystem instance is constructed.
`protected void`	`initializeAuditService()` Initialize and launch the audit manager and service.
`static void`	`initializeClass()` This is purely for testing, as it force initializes all static initializers.
`protected void`	`initializeStatisticsBinding()` Initialize the statistics binding.
`boolean`	`isCSEEnabled()` a method to know if Client side encryption is enabled or not.
`boolean`	`isDirectory(org.apache.hadoop.fs.Path f)` Optimized probe for a path referencing a dir.
`boolean`	`isFile(org.apache.hadoop.fs.Path f)` Optimized probe for a path referencing a file.
`boolean`	`isMagicCommitEnabled()` Is magic commit enabled?
`boolean`	`isMagicCommitPath(org.apache.hadoop.fs.Path path)` Predicate: is a path a magic commit path? True if magic commit is enabled and the path qualifies as special.
`org.apache.hadoop.fs.Path`	`keyToQualifiedPath(String key)` Convert a key to a fully qualified path.
`List<RoleModel.Statement>`	`listAWSPolicyRules(Set<AWSPolicyProvider.AccessLevel> access)` Build the AWS policy for restricted access to the resources needed by this bucket.
`org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus>`	`listFiles(org.apache.hadoop.fs.Path f, boolean recursive)` .
`org.apache.hadoop.fs.RemoteIterator<S3ALocatedFileStatus>`	`listFilesAndEmptyDirectories(org.apache.hadoop.fs.Path f, boolean recursive)` Recursive List of files and empty directories.
`org.apache.hadoop.fs.RemoteIterator<S3ALocatedFileStatus>`	`listFilesAndEmptyDirectoriesForceNonAuth(org.apache.hadoop.fs.Path f, boolean recursive)` Recursive List of files and empty directories, force metadatastore to act like it is non-authoritative.
`org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus>`	`listLocatedStatus(org.apache.hadoop.fs.Path f)` Override superclass so as to add statistic collection.
`org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus>`	`listLocatedStatus(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.PathFilter filter)` .
`List<com.amazonaws.services.s3.model.MultipartUpload>`	`listMultipartUploads(String prefix)` Listing all multipart uploads; limited to the first few hundred.
`protected S3ListResult`	`listObjects(S3ListRequest request, org.apache.hadoop.fs.statistics.DurationTrackerFactory trackerFactory)` Initiate a `listObjects` operation, incrementing metrics in the process.
`org.apache.hadoop.fs.FileStatus[]`	`listStatus(org.apache.hadoop.fs.Path f)` List the statuses of the files/directories in the given path if the path is a directory.
`org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.FileStatus>`	`listStatusIterator(org.apache.hadoop.fs.Path p)` Override subclass such that we benefit for async listing done in `S3AFileSystem`.
`MultipartUtils.UploadIterator`	`listUploads(String prefix)` List any pending multipart uploads whose keys begin with prefix, using an iterator that can handle an unlimited number of entries.
`List<String>`	`listXAttrs(org.apache.hadoop.fs.Path path)`
`org.apache.hadoop.fs.Path`	`makeQualified(org.apache.hadoop.fs.Path path)`
`String`	`maybeAddTrailingSlash(String key)` Turns a path (relative or otherwise) into an S3 key, adding a trailing "/" if the path is not the root and does not already have a "/" at the end.
`protected void`	`maybeCreateFakeParentDirectory(org.apache.hadoop.fs.Path path)` Create a fake parent directory if required.
`void`	`metastoreOperationRetried(Exception ex, int retries, boolean idempotent)` Callback from `Invoker` when an operation against a metastore is retried.
`boolean`	`mkdirs(org.apache.hadoop.fs.Path p, org.apache.hadoop.fs.permission.FsPermission permission)` Make the given path and all non-existent parents into directories.
`CommitterStatistics`	`newCommitterStatistics()` Create a new instance of the committer statistics.
`com.amazonaws.services.s3.model.ObjectMetadata`	`newObjectMetadata(long length)` Create a new object metadata instance.
`com.amazonaws.services.s3.model.PutObjectRequest`	`newPutObjectRequest(String key, com.amazonaws.services.s3.model.ObjectMetadata metadata, File srcfile)` Create a putObject request.
`org.apache.hadoop.fs.FSDataInputStream`	`open(org.apache.hadoop.fs.Path f, int bufferSize)` Opens an FSDataInputStream at the indicated Path.
`CompletableFuture<org.apache.hadoop.fs.FSDataInputStream>`	`openFileWithOptions(org.apache.hadoop.fs.Path rawPath, org.apache.hadoop.fs.impl.OpenFileParameters parameters)` Initiate the open or select operation.
`void`	`operationRetried(Exception ex)` Callback when an operation was retried.
`void`	`operationRetried(String text, Exception ex, int retries, boolean idempotent)` Callback from `Invoker` when an operation is retried.
`String`	`pathToKey(org.apache.hadoop.fs.Path path)` Turns a path (relative or otherwise) into an S3 key.
`UploadInfo`	`putObject(com.amazonaws.services.s3.model.PutObjectRequest putObjectRequest)` Start a transfer-manager managed async PUT of an object, incrementing the put requests and put bytes counters.
`org.apache.hadoop.fs.Path`	`qualify(org.apache.hadoop.fs.Path path)` Qualify a path.
`void`	`removeKeys(List<com.amazonaws.services.s3.model.DeleteObjectsRequest.KeyVersion> keysToDelete, boolean deleteFakeDir, BulkOperationState operationState)` Invoke `removeKeysS3(List, boolean, boolean)` with handling of `MultiObjectDeleteException`.
`boolean`	`rename(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst)` Renames Path src to Path dst.
`protected void`	`setAmazonS3Client(com.amazonaws.services.s3.AmazonS3 client)` Set the client -used in mocking tests to force in a different client.
`protected void`	`setBucket(String bucket)` Set the bucket.
`protected void`	`setEncryptionSecrets(EncryptionSecrets secrets)` Set the encryption secrets for requests.
`void`	`setInputPolicy(S3AInputPolicy inputPolicy)` Change the input policy for this FS.
`protected void`	`setTtlTimeProvider(ITtlTimeProvider ttlTimeProvider)`
`protected void`	`setUri(URI fsUri, boolean canonicalize)` Set the URI field through `S3xLoginHelper` and optionally `canonicalizeUri(URI)` Exported for testing.
`void`	`setWorkingDirectory(org.apache.hadoop.fs.Path newDir)` Set the current working directory for the given file system.
`AWSCredentialProviderList`	`shareCredentials(String purpose)` Get a shared copy of the AWS credentials, with its reference counter updated.
`protected void`	`stopAllServices()` Stop all services.
`String`	`toString()`
`protected void`	`verifyBucketExists()` Verify that the bucket exists.
`protected void`	`verifyBucketExistsV2()` Verify that the bucket exists.

Methods inherited from class org.apache.hadoop.fs.FileSystem
append, append, appendFile, areSymlinksEnabled, cancelDeleteOnExit, clearStatistics, closeAll, closeAllForUGI, completeLocalOutput, concat, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, create, createDataInputStreamBuilder, createDataInputStreamBuilder, createDataOutputStreamBuilder, createFile, createNewFile, createNonRecursive, createNonRecursive, createPathHandle, createSnapshot, createSnapshot, createSymlink, delete, deleteOnExit, deleteSnapshot, enableSymlinks, fixRelativePart, get, get, get, getAclStatus, getAllStatistics, getAllStoragePolicies, getBlockSize, getChildFileSystems, getDefaultBlockSize, getDefaultReplication, getDefaultReplication, getDefaultUri, getFileBlockLocations, getFileBlockLocations, getFileChecksum, getFileLinkStatus, getFileSystemClass, getFSofPath, getGlobalStorageStatistics, getHomeDirectory, getInitialWorkingDirectory, getLength, getLinkTarget, getLocal, getName, getNamed, getPathHandle, getQuotaUsage, getReplication, getServerDefaults, getServerDefaults, getStatistics, getStatistics, getStatus, getStatus, getStoragePolicy, getTrashRoot, getTrashRoots, getUsed, getUsed, listCorruptFileBlocks, listStatus, listStatus, listStatus, listStatusBatch, mkdirs, mkdirs, modifyAclEntries, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, msync, newInstance, newInstance, newInstance, newInstanceLocal, open, open, open, openFile, openFile, openFileWithOptions, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, processDeleteOnExit, removeAcl, removeAclEntries, removeDefaultAcl, removeXAttr, rename, renameSnapshot, resolveLink, resolvePath, satisfyStoragePolicy, setAcl, setDefaultUri, setDefaultUri, setOwner, setPermission, setQuota, setQuotaByStorageType, setReplication, setStoragePolicy, setTimes, setVerifyChecksum, setWriteChecksum, setXAttr, setXAttr, startLocalOutput, supportsSymlinks, truncate, unsetStoragePolicy

Methods inherited from class org.apache.hadoop.conf.Configured
getConf, setConf

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.security.token.DelegationTokenIssuer
addDelegationTokens, collectDelegationTokens

- Field Detail
  - DEFAULT_BLOCKSIZE
```
public static final int DEFAULT_BLOCKSIZE
```
    Default blocksize as used in blocksize and FS status queries.
    
    See Also:
    
    Constant Field Values
  - LOG
```
public static final org.slf4j.Logger LOG
```
- Constructor Detail
  - S3AFileSystem
```
public S3AFileSystem()
```
- Method Detail
  - initialize
```
public void initialize(URI name,
                       org.apache.hadoop.conf.Configuration originalConf)
                throws IOException
```
    Called after a new FileSystem instance is constructed.
    
    Overrides:
    
    initialize in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    name - a uri whose authority section names the host, port, etc. for this FileSystem
    
    originalConf - the configuration to use for the FS. The bucket-specific options are patched over the base ones before any use is made of the config.
    
    Throws:
    
    IOException
  - initializeStatisticsBinding
```
protected void initializeStatisticsBinding()
```
    Initialize the statistics binding. This is done by creating an IntegratedS3AStatisticsContext with callbacks to get the FS's instrumentation and FileSystem.statistics field; the latter may change after initialize(URI, Configuration), so needs to be dynamically adapted. Protected so that (mock) subclasses can replace it with a different statistics binding, if desired.
  - createStorageStatistics
```
protected static S3AStorageStatistics createStorageStatistics(org.apache.hadoop.fs.statistics.IOStatistics ioStatistics)
```
    Create the storage statistics or bind to an existing one.
    
    Parameters:
    
    ioStatistics - IOStatistics to build the storage statistics from.
    
    Returns:
    
    a storage statistics instance; expected to be that of the FS.
  - verifyBucketExists
```
@Retries.RetryTranslated
protected void verifyBucketExists()
                                                    throws UnknownStoreException,
                                                           IOException
```
    Verify that the bucket exists. This does not check permissions, not even read access. Retry policy: retrying, translated.
    
    Throws:
    
    UnknownStoreException - the bucket is absent
    
    IOException - any other problem talking to S3
  - verifyBucketExistsV2
```
@Retries.RetryTranslated
protected void verifyBucketExistsV2()
                                                      throws UnknownStoreException,
                                                             IOException
```
    Verify that the bucket exists. This will correctly throw an exception when credentials are invalid. Retry policy: retrying, translated.
    
    Throws:
    
    UnknownStoreException - the bucket is absent
    
    IOException - any other problem talking to S3
  - getInstrumentation
```
public S3AInstrumentation getInstrumentation()
```
    Get S3A Instrumentation. For test purposes.
    
    Returns:
    
    this instance's instrumentation.
  - getFsStatistics
```
public org.apache.hadoop.fs.FileSystem.Statistics getFsStatistics()
```
    Get FS Statistic for this S3AFS instance.
    
    Returns:
    
    FS statistic instance.
  - getListing
```
public Listing getListing()
```
    Get current listing instance.
    
    Returns:
    
    this instance's listing.
  - initializeAuditService
```
protected void initializeAuditService()
                               throws IOException
```
    Initialize and launch the audit manager and service. As this takes the FS IOStatistics store, it must be invoked after instrumentation is initialized.
    
    Throws:
    
    IOException - failure to instantiate/initialize.
  - getAuditManager
```
@InterfaceAudience.Private
public AuditManagerS3A getAuditManager()
```
    The audit manager.
    
    Returns:
    
    the audit manager
  - getAuditor
```
@InterfaceAudience.Private
public OperationAuditor getAuditor()
```
    Get the auditor; valid once initialized.
    
    Returns:
    
    the auditor.
  - getActiveAuditSpan
```
@InterfaceAudience.Private
public AuditSpanS3A getActiveAuditSpan()
```
    Get the active audit span.
    
    Specified by:
    
    getActiveAuditSpan in interface org.apache.hadoop.fs.store.audit.ActiveThreadSpanSource<AuditSpanS3A>
    
    Returns:
    
    the span.
  - getAuditSpanSource
```
@InterfaceAudience.Private
public org.apache.hadoop.fs.store.audit.AuditSpanSource getAuditSpanSource()
```
    Get the audit span source; allows for components like the committers to have a source of spans without being hard coded to the FS only.
    
    Returns:
    
    the source of spans -base implementation is this instance.
  - createSpan
```
public AuditSpanS3A createSpan(String operation,
                               @Nullable
                               String path1,
                               @Nullable
                               String path2)
                        throws IOException
```
    Start an operation; this informs the audit service of the event and then sets it as the active span.
    
    Specified by:
    
    createSpan in interface org.apache.hadoop.fs.store.audit.AuditSpanSource<AuditSpanS3A>
    
    Parameters:
    
    operation - operation name.
    
    path1 - first path of operation
    
    path2 - second path of operation
    
    Returns:
    
    a span for the audit
    
    Throws:
    
    IOException - failure
  - createRequestFactory
```
protected RequestFactory createRequestFactory()
```
    Build the request factory. MUST be called after reading encryption secrets from settings/ delegation token. Protected, in case test/mock classes want to implement their own variants.
    
    Returns:
    
    request factory.
  - getRequestFactory
```
public RequestFactory getRequestFactory()
```
    Get the request factory which uses this store's audit span.
    
    Returns:
    
    the request factory.
  - createDelegationOperations
```
public DelegationOperations createDelegationOperations()
```
    Create an instance of the delegation operations.
    
    Returns:
    
    callbacks for DT support.
  - setEncryptionSecrets
```
protected void setEncryptionSecrets(EncryptionSecrets secrets)
```
    Set the encryption secrets for requests.
    
    Parameters:
    
    secrets - secrets
  - getEncryptionSecrets
```
public EncryptionSecrets getEncryptionSecrets()
```
    Get the encryption secrets. This potentially sensitive information and must be treated with care.
    
    Returns:
    
    the current encryption secrets.
  - abortOutstandingMultipartUploads
```
@Retries.RetryTranslated
public void abortOutstandingMultipartUploads(long seconds)
                                                               throws IOException
```
    Abort all outstanding MPUs older than a given age.
    
    Parameters:
    
    seconds - time in seconds
    
    Throws:
    
    IOException - on any failure, other than 403 "permission denied"
  - getScheme
```
public String getScheme()
```
    Return the protocol scheme for the FileSystem.
    
    Overrides:
    
    getScheme in class org.apache.hadoop.fs.FileSystem
    
    Returns:
    
    "s3a"
  - getUri
```
public URI getUri()
```
    Returns a URI whose scheme and authority identify this FileSystem.
    
    Specified by:
    
    getUri in class org.apache.hadoop.fs.FileSystem
  - setUri
```
protected void setUri(URI fsUri,
                      boolean canonicalize)
```
    Set the URI field through S3xLoginHelper and optionally canonicalizeUri(URI) Exported for testing.
    
    Parameters:
    
    fsUri - filesystem URI.
    
    canonicalize - true if the URI should be canonicalized.
  - getCanonicalUri
```
public URI getCanonicalUri()
```
    Get the canonical URI.
    
    Overrides:
    
    getCanonicalUri in class org.apache.hadoop.fs.FileSystem
    
    Returns:
    
    the canonical URI of this FS.
  - getDefaultPort
```
public int getDefaultPort()
```
    Overrides:
    
    getDefaultPort in class org.apache.hadoop.fs.FileSystem
  - getAmazonS3ClientForTesting
```
public com.amazonaws.services.s3.AmazonS3 getAmazonS3ClientForTesting(String reason)
```
    Returns the S3 client used by this filesystem. Warning: this must only be used for testing, as it bypasses core S3A operations.
    
    Parameters:
    
    reason - a justification for requesting access.
    
    Returns:
    
    AmazonS3Client
  - setAmazonS3Client
```
protected void setAmazonS3Client(com.amazonaws.services.s3.AmazonS3 client)
```
    Set the client -used in mocking tests to force in a different client.
    
    Parameters:
    
    client - client.
  - getBucketLocation
```
@Retries.RetryTranslated
 @InterfaceAudience.LimitedPrivate(value="diagnostics")
public String getBucketLocation()
                                                                                                          throws IOException
```
    Get the region of a bucket. Invoked from StoreContext; consider an entry point.
    
    Returns:
    
    the region in which a bucket is located
    
    Throws:
    
    AccessDeniedException - if the caller lacks permission.
    
    IOException - on any failure.
  - getBucketLocation
```
@AuditEntryPoint
 @Retries.RetryTranslated
public String getBucketLocation(String bucketName)
                                                                    throws IOException
```
    Get the region of a bucket; fixing up the region so it can be used in the builders of other AWS clients. Requires the caller to have the AWS role permission s3:GetBucketLocation. Retry policy: retrying, translated.
    
    Parameters:
    
    bucketName - the name of the bucket
    
    Returns:
    
    the region in which a bucket is located
    
    Throws:
    
    AccessDeniedException - if the caller lacks permission.
    
    IOException - on any failure.
  - getInputPolicy
```
@InterfaceStability.Unstable
public S3AInputPolicy getInputPolicy()
```
    Get the input policy for this FS instance.
    
    Returns:
    
    the input policy
  - getChangeDetectionPolicy
```
public ChangeDetectionPolicy getChangeDetectionPolicy()
```
    Get the change detection policy for this FS instance. Only public to allow access in tests in other packages.
    
    Returns:
    
    the change detection policy
  - getS3EncryptionAlgorithm
```
public S3AEncryptionMethods getS3EncryptionAlgorithm()
```
    Get the encryption algorithm of this endpoint.
    
    Returns:
    
    the encryption algorithm.
  - getBucket
```
public String getBucket()
```
    Get the bucket of this filesystem.
    
    Returns:
    
    the bucket
  - setBucket
```
protected void setBucket(String bucket)
```
    Set the bucket.
    
    Parameters:
    
    bucket - the bucket
  - setInputPolicy
```
@InterfaceStability.Unstable
public void setInputPolicy(S3AInputPolicy inputPolicy)
```
    Change the input policy for this FS.
    
    Parameters:
    
    inputPolicy - new policy
  - pathToKey
```
public String pathToKey(org.apache.hadoop.fs.Path path)
```
    Turns a path (relative or otherwise) into an S3 key.
    
    Parameters:
    
    path - input path, may be relative to the working dir
    
    Returns:
    
    a key excluding the leading "/", or, if it is the root path, ""
  - maybeAddTrailingSlash
```
@InterfaceAudience.Private
public String maybeAddTrailingSlash(String key)
```
    Turns a path (relative or otherwise) into an S3 key, adding a trailing "/" if the path is not the root and does not already have a "/" at the end.
    
    Parameters:
    
    key - s3 key or ""
    
    Returns:
    
    the with a trailing "/", or, if it is the root key, "",
  - keyToQualifiedPath
```
public org.apache.hadoop.fs.Path keyToQualifiedPath(String key)
```
    Convert a key to a fully qualified path. This includes fixing up the URI so that if it ends with a trailing slash, that is corrected, similar to Path.normalizePath().
    
    Parameters:
    
    key - input key
    
    Returns:
    
    the fully qualified path including URI scheme and bucket name.
  - makeQualified
```
public org.apache.hadoop.fs.Path makeQualified(org.apache.hadoop.fs.Path path)
```
    Overrides:
    
    makeQualified in class org.apache.hadoop.fs.FileSystem
  - qualify
```
public org.apache.hadoop.fs.Path qualify(org.apache.hadoop.fs.Path path)
```
    Qualify a path. This includes fixing up the URI so that if it ends with a trailing slash, that is corrected, similar to Path.normalizePath().
    
    Parameters:
    
    path - path to qualify
    
    Returns:
    
    a qualified path.
  - checkPath
```
public void checkPath(org.apache.hadoop.fs.Path path)
```
    Check that a Path belongs to this FileSystem. Unlike the superclass, this version does not look at authority, only hostnames.
    
    Overrides:
    
    checkPath in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    path - to check
    
    Throws:
    
    IllegalArgumentException - if there is an FS mismatch
  - canonicalizeUri
```
protected URI canonicalizeUri(URI rawUri)
```
    Override the base canonicalization logic and relay to S3xLoginHelper.canonicalizeUri(URI, int). This allows for the option of changing this logic for better DT handling.
    
    Overrides:
    
    canonicalizeUri in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    rawUri - raw URI.
    
    Returns:
    
    the canonical URI to use in delegation tokens and file context.
  - open
```
@Retries.RetryTranslated
public org.apache.hadoop.fs.FSDataInputStream open(org.apache.hadoop.fs.Path f,
                                                                            int bufferSize)
                                                                     throws IOException
```
    Opens an FSDataInputStream at the indicated Path.
    
    Specified by:
    
    open in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    f - the file name to open
    
    bufferSize - the size of the buffer to be used.
    
    Throws:
    
    IOException
  - createReadContext
```
protected S3AReadOpContext createReadContext(org.apache.hadoop.fs.FileStatus fileStatus,
                                             S3AInputPolicy seekPolicy,
                                             ChangeDetectionPolicy changePolicy,
                                             long readAheadRange,
                                             org.apache.hadoop.fs.store.audit.AuditSpan auditSpan)
```
    Create the read context for reading from the referenced file, using FS state as well as the status.
    
    Parameters:
    
    fileStatus - file status.
    
    seekPolicy - input policy for this operation
    
    changePolicy - change policy for this operation.
    
    readAheadRange - readahead value.
    
    auditSpan - audit span.
    
    Returns:
    
    a context for read and select operations.
  - create
```
@AuditEntryPoint
public org.apache.hadoop.fs.FSDataOutputStream create(org.apache.hadoop.fs.Path f,
                                                                       org.apache.hadoop.fs.permission.FsPermission permission,
                                                                       boolean overwrite,
                                                                       int bufferSize,
                                                                       short replication,
                                                                       long blockSize,
                                                                       org.apache.hadoop.util.Progressable progress)
                                                                throws IOException
```
    Create an FSDataOutputStream at the indicated Path with write-progress reporting. Retry policy: retrying, translated on the getFileStatus() probe. No data is uploaded to S3 in this call, so retry issues related to that.
    
    Specified by:
    
    create in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    f - the file name to open
    
    permission - the permission to set.
    
    overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.
    
    bufferSize - the size of the buffer to be used.
    
    replication - required block replication for the file.
    
    blockSize - the requested block size.
    
    progress - the progress reporter.
    
    Throws:
    
    IOException - in the event of IO related errors.
    
    See Also:
    
    FileSystem.setPermission(Path, FsPermission)
  - getWriteOperationHelper
```
@InterfaceAudience.Private
public WriteOperationHelper getWriteOperationHelper()
```
    Create a Write Operation Helper with the current active span. All operations made through this helper will activate the span before execution. This class permits other low-level operations against the store. It is unstable and only intended for code with intimate knowledge of the object store. If using this, be prepared for changes even on minor point releases.
    
    Returns:
    
    a new helper.
  - createWriteOperationHelper
```
@InterfaceAudience.Private
public WriteOperationHelper createWriteOperationHelper(org.apache.hadoop.fs.store.audit.AuditSpan auditSpan)
```
    Create a Write Operation Helper with the given span. All operations made through this helper will activate the span before execution.
    
    Parameters:
    
    auditSpan - audit span
    
    Returns:
    
    a new helper.
  - createNonRecursive
```
@AuditEntryPoint
public org.apache.hadoop.fs.FSDataOutputStream createNonRecursive(org.apache.hadoop.fs.Path p,
                                                                                   org.apache.hadoop.fs.permission.FsPermission permission,
                                                                                   EnumSet<org.apache.hadoop.fs.CreateFlag> flags,
                                                                                   int bufferSize,
                                                                                   short replication,
                                                                                   long blockSize,
                                                                                   org.apache.hadoop.util.Progressable progress)
                                                                            throws IOException
```
    Overrides:
    
    createNonRecursive in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    FileNotFoundException - if the parent directory is not present -or is not a directory.
    
    IOException
  - append
```
public org.apache.hadoop.fs.FSDataOutputStream append(org.apache.hadoop.fs.Path f,
                                                      int bufferSize,
                                                      org.apache.hadoop.util.Progressable progress)
                                               throws IOException
```
    Append to an existing file (optional operation).
    
    Specified by:
    
    append in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    f - the existing file to be appended.
    
    bufferSize - the size of the buffer to be used.
    
    progress - for reporting progress if it is not null.
    
    Throws:
    
    IOException - indicating that append is not supported.
  - rename
```
@AuditEntryPoint
 @Retries.RetryTranslated
public boolean rename(org.apache.hadoop.fs.Path src,
                                                                 org.apache.hadoop.fs.Path dst)
                                                          throws IOException
```
    Renames Path src to Path dst. Can take place on local fs or remote DFS. Warning: S3 does not support renames. This method does a copy which can take S3 some time to execute with large files and directories. Since there is no Progressable passed in, this can time out jobs. Note: This implementation differs with other S3 drivers. Specifically:
```
       Fails if src is a file and dst is a directory.
       Fails if src is a directory and dst is a file.
       Fails if the parent of dst does not exist or is a file.
       Fails if dst is a directory that is not empty.
 
```
    Specified by:
    
    rename in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    src - path to be renamed
    
    dst - new path after rename
    
    Returns:
    
    true if rename is successful
    
    Throws:
    
    IOException - on IO failure
  - getFsDelegationToken
```
public org.apache.hadoop.security.token.Token<? extends org.apache.hadoop.security.token.TokenIdentifier> getFsDelegationToken()
                                                                                                                        throws IOException
```
    Specified by:
    
    getFsDelegationToken in interface DelegationTokenProvider
    
    Throws:
    
    IOException
  - getObjectMetadata
```
@AuditEntryPoint
 @InterfaceAudience.LimitedPrivate(value="utilities")
 @Retries.RetryTranslated
 @InterfaceStability.Evolving
public com.amazonaws.services.s3.model.ObjectMetadata getObjectMetadata(org.apache.hadoop.fs.Path path)
                                                                                                                                                                                                throws IOException
```
    Low-level call to get at the object metadata. This method is used in some external applications and so must be viewed as a public entry point. Auditing: An audit entry point.
    
    Parameters:
    
    path - path to the object. This will be qualified.
    
    Returns:
    
    metadata
    
    Throws:
    
    IOException - IO and object access problems.
  - hasMetadataStore
```
public boolean hasMetadataStore()
```
    Does this Filesystem have a metadata store?
    
    Returns:
    
    true iff the FS has been instantiated with a metadata store
  - hasAuthoritativeMetadataStore
```
public boolean hasAuthoritativeMetadataStore()
```
    Does the filesystem have an authoritative metadata store?
    
    Returns:
    
    true if there is a metadata store and the authoritative flag is set for this filesystem.
  - getMetadataStore
```
public MetadataStore getMetadataStore()
```
    Get the metadata store. This will always be non-null, but may be bound to the NullMetadataStore.
    
    Returns:
    
    the metadata store of this FS instance
  - entryPoint
```
protected org.apache.hadoop.fs.store.audit.AuditSpan entryPoint(Statistic operation,
                                                                org.apache.hadoop.fs.Path path)
                                                         throws IOException
```
    Entry point to an operation. Increments the statistic; verifies the FS is active.
    
    Parameters:
    
    operation - The operation being invoked
    
    path - first path of operation
    
    Returns:
    
    a span for the audit
    
    Throws:
    
    IOException - failure of audit service
  - entryPoint
```
protected org.apache.hadoop.fs.store.audit.AuditSpan entryPoint(Statistic operation,
                                                                @Nullable
                                                                String path1,
                                                                @Nullable
                                                                String path2)
                                                         throws IOException
```
    Entry point to an operation. Increments the statistic; verifies the FS is active.
    
    Parameters:
    
    operation - The operation being invoked
    
    path1 - first path of operation
    
    path2 - second path of operation
    
    Returns:
    
    a span for the audit
    
    Throws:
    
    IOException - failure of audit service
  - incrementStatistic
```
protected void incrementStatistic(Statistic statistic)
```
    Increment a statistic by 1. This increments both the instrumentation and storage statistics.
    
    Parameters:
    
    statistic - The operation to increment
  - incrementStatistic
```
protected void incrementStatistic(Statistic statistic,
                                  long count)
```
    Increment a statistic by a specific value. This increments both the instrumentation and storage statistics.
    
    Parameters:
    
    statistic - The operation to increment
    
    count - the count to increment
  - decrementGauge
```
protected void decrementGauge(Statistic statistic,
                              long count)
```
    Decrement a gauge by a specific value.
    
    Parameters:
    
    statistic - The operation to decrement
    
    count - the count to decrement
  - incrementGauge
```
protected void incrementGauge(Statistic statistic,
                              long count)
```
    Increment a gauge by a specific value.
    
    Parameters:
    
    statistic - The operation to increment
    
    count - the count to increment
  - operationRetried
```
public void operationRetried(Exception ex)
```
    Callback when an operation was retried. Increments the statistics of ignored errors or throttled requests, depending up on the exception class.
    
    Parameters:
    
    ex - exception.
  - operationRetried
```
public void operationRetried(String text,
                             Exception ex,
                             int retries,
                             boolean idempotent)
```
    Callback from Invoker when an operation is retried.
    
    Parameters:
    
    text - text of the operation
    
    ex - exception
    
    retries - number of retries
    
    idempotent - is the method idempotent
  - metastoreOperationRetried
```
public void metastoreOperationRetried(Exception ex,
                                      int retries,
                                      boolean idempotent)
```
    Callback from Invoker when an operation against a metastore is retried. Always increments the Statistic.S3GUARD_METADATASTORE_RETRY statistic/counter; if it is a throttling exception will update the associated throttled metrics/statistics.
    
    Parameters:
    
    ex - exception
    
    retries - number of retries
    
    idempotent - is the method idempotent
  - getStorageStatistics
```
public S3AStorageStatistics getStorageStatistics()
```
    Get the storage statistics of this filesystem.
    
    Overrides:
    
    getStorageStatistics in class org.apache.hadoop.fs.FileSystem
    
    Returns:
    
    the storage statistics
  - getIOStatistics
```
public org.apache.hadoop.fs.statistics.IOStatistics getIOStatistics()
```
    Get the instrumentation's IOStatistics.
    
    Specified by:
    
    getIOStatistics in interface org.apache.hadoop.fs.statistics.IOStatisticsSource
    
    Returns:
    
    statistics
  - getDurationTrackerFactory
```
protected org.apache.hadoop.fs.statistics.DurationTrackerFactory getDurationTrackerFactory()
```
    Get the factory for duration tracking.
    
    Returns:
    
    a factory from the instrumentation.
  - getObjectMetadata
```
@Retries.RetryRaw
protected com.amazonaws.services.s3.model.ObjectMetadata getObjectMetadata(String key,
                                                                                             ChangeTracker changeTracker,
                                                                                             Invoker changeInvoker,
                                                                                             String operation)
                                                                                      throws IOException
```
    Request object metadata; increments counters in the process. Retry policy: retry untranslated. Uses changeTracker to detect an unexpected file version (eTag or versionId)
    
    Parameters:
    
    key - key
    
    changeTracker - the change tracker to detect unexpected object version
    
    changeInvoker - the invoker providing the retry policy
    
    operation - the operation (e.g. "read" or "copy") triggering this call
    
    Returns:
    
    the metadata
    
    Throws:
    
    IOException - if the retry invocation raises one (it shouldn't).
    
    RemoteFileChangedException - if an unexpected version is detected
  - listObjects
```
@Retries.RetryRaw
protected S3ListResult listObjects(S3ListRequest request,
                                                     @Nullable
                                                     org.apache.hadoop.fs.statistics.DurationTrackerFactory trackerFactory)
                                              throws IOException
```
    Initiate a listObjects operation, incrementing metrics in the process. Retry policy: retry untranslated.
    
    Parameters:
    
    request - request to initiate
    
    trackerFactory - duration tracking
    
    Returns:
    
    the results
    
    Throws:
    
    IOException - if the retry invocation raises one (it shouldn't).
  - continueListObjects
```
@Retries.RetryRaw
protected S3ListResult continueListObjects(S3ListRequest request,
                                                             S3ListResult prevResult,
                                                             org.apache.hadoop.fs.statistics.DurationTrackerFactory trackerFactory)
                                                      throws IOException
```
    List the next set of objects. Retry policy: retry untranslated.
    
    Parameters:
    
    request - last list objects request to continue
    
    prevResult - last paged result to continue from
    
    trackerFactory - duration tracking
    
    Returns:
    
    the next result object
    
    Throws:
    
    IOException - none, just there for retryUntranslated.
  - incrementReadOperations
```
public void incrementReadOperations()
```
    Increment read operations.
  - incrementWriteOperations
```
public void incrementWriteOperations()
```
    Increment the write operation counter. This is somewhat inaccurate, as it appears to be invoked more often than needed in progress callbacks.
  - deleteObject
```
@Retries.RetryRaw
protected void deleteObject(String key)
                                       throws com.amazonaws.AmazonClientException,
                                              IOException
```
    Delete an object. This is the low-level internal call which does not update the metastore. Increments the OBJECT_DELETE_REQUESTS and write operation statistics. This call does not create any mock parent entries. Retry policy: retry untranslated; delete considered idempotent.
    
    Parameters:
    
    key - key to blob to delete.
    
    Throws:
    
    com.amazonaws.AmazonClientException - problems working with S3
    
    org.apache.hadoop.fs.InvalidRequestException - if the request was rejected due to a mistaken attempt to delete the root directory.
    
    IOException
  - newPutObjectRequest
```
public com.amazonaws.services.s3.model.PutObjectRequest newPutObjectRequest(String key,
                                                                            com.amazonaws.services.s3.model.ObjectMetadata metadata,
                                                                            File srcfile)
```
    Create a putObject request. Adds the ACL and metadata
    
    Parameters:
    
    key - key of object
    
    metadata - metadata header
    
    srcfile - source file
    
    Returns:
    
    the request
  - newObjectMetadata
```
public com.amazonaws.services.s3.model.ObjectMetadata newObjectMetadata(long length)
```
    Create a new object metadata instance. Any standard metadata headers are added here, for example: encryption.
    
    Parameters:
    
    length - length of data to set in header.
    
    Returns:
    
    a new metadata instance
  - putObject
```
@Retries.OnceRaw
public UploadInfo putObject(com.amazonaws.services.s3.model.PutObjectRequest putObjectRequest)
```
    Start a transfer-manager managed async PUT of an object, incrementing the put requests and put bytes counters. It does not update the other counters, as existing code does that as progress callbacks come in. Byte length is calculated from the file length, or, if there is no file, from the content length of the header. Because the operation is async, any stream supplied in the request must reference data (files, buffers) which stay valid until the upload completes. Retry policy: N/A: the transfer manager is performing the upload. Auditing: must be inside an audit span.
    
    Parameters:
    
    putObjectRequest - the request
    
    Returns:
    
    the upload initiated
  - incrementPutStartStatistics
```
public void incrementPutStartStatistics(long bytes)
```
    At the start of a put/multipart upload operation, update the relevant counters.
    
    Parameters:
    
    bytes - bytes in the request.
  - incrementPutCompletedStatistics
```
public void incrementPutCompletedStatistics(boolean success,
                                            long bytes)
```
    At the end of a put/multipart upload operation, update the relevant counters and gauges.
    
    Parameters:
    
    success - did the operation succeed?
    
    bytes - bytes in the request.
  - incrementPutProgressStatistics
```
public void incrementPutProgressStatistics(String key,
                                           long bytes)
```
    Callback for use in progress callbacks from put/multipart upload events. Increments those statistics which are expected to be updated during the ongoing upload operation.
    
    Parameters:
    
    key - key to file that is being written (for logging)
    
    bytes - bytes successfully uploaded.
  - removeKeys
```
@Retries.RetryMixed
public void removeKeys(List<com.amazonaws.services.s3.model.DeleteObjectsRequest.KeyVersion> keysToDelete,
                                           boolean deleteFakeDir,
                                           BulkOperationState operationState)
                                    throws com.amazonaws.services.s3.model.MultiObjectDeleteException,
                                           com.amazonaws.AmazonClientException,
                                           IOException
```
    Invoke removeKeysS3(List, boolean, boolean) with handling of MultiObjectDeleteException.
    
    Parameters:
    
    keysToDelete - collection of keys to delete on the s3-backend. if empty, no request is made of the object store.
    
    deleteFakeDir - indicates whether this is for deleting fake dirs
    
    operationState - (nullable) operational state for a bulk update
    
    Throws:
    
    org.apache.hadoop.fs.InvalidRequestException - if the request was rejected due to a mistaken attempt to delete the root directory.
    
    com.amazonaws.services.s3.model.MultiObjectDeleteException - one or more of the keys could not be deleted in a multiple object delete operation.
    
    com.amazonaws.AmazonClientException - amazon-layer failure.
    
    IOException - other IO Exception.
  - delete
```
@Retries.RetryTranslated
 @AuditEntryPoint
public boolean delete(org.apache.hadoop.fs.Path f,
                                                                 boolean recursive)
                                                          throws IOException
```
    Delete a Path. This operation is at least O(files), with added overheads to enumerate the path. It is also not atomic.
    
    Specified by:
    
    delete in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    f - the path to delete.
    
    recursive - if path is a directory and set to true, the directory is deleted else throws an exception. In case of a file the recursive can be set to either true or false.
    
    Returns:
    
    true if the path existed and then was deleted; false if there was no path in the first place, or the corner cases of root path deletion have surfaced.
    
    Throws:
    
    IOException - due to inability to delete a directory or file.
  - maybeCreateFakeParentDirectory
```
@Retries.RetryTranslated
protected void maybeCreateFakeParentDirectory(org.apache.hadoop.fs.Path path)
                                                                throws IOException,
                                                                       com.amazonaws.AmazonClientException
```
    Create a fake parent directory if required. That is: it parent is not the root path and does not yet exist.
    
    Parameters:
    
    path - whose parent is created if needed.
    
    Throws:
    
    IOException - IO problem
    
    com.amazonaws.AmazonClientException
  - listStatusIterator
```
@AuditEntryPoint
public org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.FileStatus> listStatusIterator(org.apache.hadoop.fs.Path p)
                                                                                                         throws FileNotFoundException,
                                                                                                                IOException
```
    Override subclass such that we benefit for async listing done in S3AFileSystem. See Listing#ObjectListingIterator.
    
    Overrides:
    
    listStatusIterator in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    FileNotFoundException
    
    IOException
  - listStatus
```
@AuditEntryPoint
public org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.fs.Path f)
                                                              throws FileNotFoundException,
                                                                     IOException
```
    List the statuses of the files/directories in the given path if the path is a directory.
    
    Specified by:
    
    listStatus in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    f - given path
    
    Returns:
    
    the statuses of the files/directories in the given patch
    
    Throws:
    
    FileNotFoundException - when the path does not exist; IOException see specific implementation
    
    IOException
  - allowAuthoritative
```
public boolean allowAuthoritative(org.apache.hadoop.fs.Path path)
```
    Is a path to be considered as authoritative? True iff there is an authoritative metastore or if there is a non-auth store with the supplied path under one of the paths declared as authoritative.
    
    Parameters:
    
    path - path
    
    Returns:
    
    true if the path is auth
  - createListObjectsRequest
```
public S3ListRequest createListObjectsRequest(String key,
                                              String delimiter)
```
    Create a ListObjectsRequest request against this bucket, with the maximum keys returned in a query set by maxKeys.
    
    Parameters:
    
    key - key for request
    
    delimiter - any delimiter
    
    Returns:
    
    the request
  - setWorkingDirectory
```
public void setWorkingDirectory(org.apache.hadoop.fs.Path newDir)
```
    Set the current working directory for the given file system. All relative paths will be resolved relative to it.
    
    Specified by:
    
    setWorkingDirectory in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    newDir - the current working directory.
  - getWorkingDirectory
```
public org.apache.hadoop.fs.Path getWorkingDirectory()
```
    Get the current working directory for the given file system.
    
    Specified by:
    
    getWorkingDirectory in class org.apache.hadoop.fs.FileSystem
    
    Returns:
    
    the directory pathname
  - getUsername
```
public String getUsername()
```
    Get the username of the FS.
    
    Returns:
    
    the short name of the user who instantiated the FS
  - getOwner
```
public org.apache.hadoop.security.UserGroupInformation getOwner()
```
    Get the owner of this FS: who created it?
    
    Returns:
    
    the owner of the FS.
  - mkdirs
```
@AuditEntryPoint
public boolean mkdirs(org.apache.hadoop.fs.Path p,
                                       org.apache.hadoop.fs.permission.FsPermission permission)
                                throws IOException,
                                       org.apache.hadoop.fs.FileAlreadyExistsException
```
    Make the given path and all non-existent parents into directories. Has the semantics of Unix 'mkdir -p'. Existence of the directory hierarchy is not an error.
    
    Specified by:
    
    mkdirs in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    p - path to create
    
    permission - to apply to path
    
    Returns:
    
    true if a directory was created or already existed
    
    Throws:
    
    org.apache.hadoop.fs.FileAlreadyExistsException - there is a file at the path specified or is discovered on one of its ancestors.
    
    IOException - other IO problems
  - createMkdirOperationCallbacks
```
public MkdirOperation.MkdirCallbacks createMkdirOperationCallbacks()
```
    Override point: create the callbacks for Mkdir. This does not create a new span; caller must be in one.
    
    Returns:
    
    an implementation of the MkdirCallbacks,
  - getContentSummary
```
@Retries.RetryTranslated
 @AuditEntryPoint
public org.apache.hadoop.fs.ContentSummary getContentSummary(org.apache.hadoop.fs.Path f)
                                                                                                 throws IOException
```
    This is a very slow operation against object storage. Execute it as a single span with whatever optimizations have been implemented.
    
    Overrides:
    
    getContentSummary in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    IOException
  - createGetContentSummaryCallbacks
```
protected GetContentSummaryOperation.GetContentSummaryCallbacks createGetContentSummaryCallbacks()
```
    Override point: create the callbacks for getContentSummary. This does not create a new span; caller must be in one.
    
    Returns:
    
    an implementation of the GetContentSummaryCallbacksImpl
  - access
```
@AuditEntryPoint
public void access(org.apache.hadoop.fs.Path f,
                                    org.apache.hadoop.fs.permission.FsAction mode)
                             throws org.apache.hadoop.security.AccessControlException,
                                    FileNotFoundException,
                                    IOException
```
    Soft check of access by forwarding to the audit manager and so on to the auditor.
    
    Overrides:
    
    access in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    org.apache.hadoop.security.AccessControlException
    
    FileNotFoundException
    
    IOException
  - getFileStatus
```
@AuditEntryPoint
 @Retries.RetryTranslated
public org.apache.hadoop.fs.FileStatus getFileStatus(org.apache.hadoop.fs.Path f)
                                                                                         throws IOException
```
    Return a file status object that represents the path.
    
    Specified by:
    
    getFileStatus in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    f - The path we want information from
    
    Returns:
    
    a FileStatus object
    
    Throws:
    
    FileNotFoundException - when the path does not exist
    
    IOException - on other problems.
  - copyFromLocalFile
```
@AuditEntryPoint
public void copyFromLocalFile(boolean delSrc,
                                               boolean overwrite,
                                               org.apache.hadoop.fs.Path src,
                                               org.apache.hadoop.fs.Path dst)
                                        throws IOException
```
    The src file is on the local disk. Add it to FS at the given dst name. This version doesn't need to create a temporary file to calculate the md5. Sadly this doesn't seem to be used by the shell cp :( delSrc indicates if the source should be removed
    
    Overrides:
    
    copyFromLocalFile in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    delSrc - whether to delete the src
    
    overwrite - whether to overwrite an existing file
    
    src - path
    
    dst - path
    
    Throws:
    
    IOException - IO problem
    
    org.apache.hadoop.fs.FileAlreadyExistsException - the destination file exists and overwrite==false
    
    com.amazonaws.AmazonClientException - failure in the AWS SDK
  - createCopyFromLocalCallbacks
```
protected CopyFromLocalOperation.CopyFromLocalOperationCallbacks createCopyFromLocalCallbacks()
                                                                                       throws IOException
```
    Throws:
    
    IOException
  - close
```
public void close()
           throws IOException
```
    Close the filesystem. This shuts down all transfers.
    
    Specified by:
    
    close in interface Closeable
    
    Specified by:
    
    close in interface AutoCloseable
    
    Overrides:
    
    close in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    IOException - IO problem
  - stopAllServices
```
protected void stopAllServices()
```
    Stop all services. This is invoked in close() and during failures of initialize() -make sure that all operations here are robust to failures in both the expected state of this FS and of failures while being stopped.
  - getDelegationTokens
```
public Optional<S3ADelegationTokens> getDelegationTokens()
```
    Get the delegation token support for this filesystem; not null iff delegation support is enabled.
    
    Returns:
    
    the token support, or an empty option.
  - getCanonicalServiceName
```
public String getCanonicalServiceName()
```
    Return a service name iff delegation tokens are enabled and the token binding is issuing delegation tokens.
    
    Specified by:
    
    getCanonicalServiceName in interface org.apache.hadoop.security.token.DelegationTokenIssuer
    
    Overrides:
    
    getCanonicalServiceName in class org.apache.hadoop.fs.FileSystem
    
    Returns:
    
    the canonical service name or null
  - getDelegationToken
```
@AuditEntryPoint
public org.apache.hadoop.security.token.Token<AbstractS3ATokenIdentifier> getDelegationToken(String renewer)
                                                                                                       throws IOException
```
    Get a delegation token if the FS is set up for them. If the user already has a token, it is returned, even if it has expired.
    
    Specified by:
    
    getDelegationToken in interface org.apache.hadoop.security.token.DelegationTokenIssuer
    
    Overrides:
    
    getDelegationToken in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    renewer - the account name that is allowed to renew the token.
    
    Returns:
    
    the delegation token or null
    
    Throws:
    
    IOException - IO failure
  - getAdditionalTokenIssuers
```
public org.apache.hadoop.security.token.DelegationTokenIssuer[] getAdditionalTokenIssuers()
                                                                                   throws IOException
```
    Ask any DT plugin for any extra token issuers. These do not get told of the encryption secrets and can return any type of token. This allows DT plugins to issue extra tokens for ancillary services.
    
    Specified by:
    
    getAdditionalTokenIssuers in interface org.apache.hadoop.security.token.DelegationTokenIssuer
    
    Overrides:
    
    getAdditionalTokenIssuers in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    IOException
  - listAWSPolicyRules
```
@InterfaceAudience.Private
public List<RoleModel.Statement> listAWSPolicyRules(Set<AWSPolicyProvider.AccessLevel> access)
```
    Build the AWS policy for restricted access to the resources needed by this bucket. The policy generated includes S3 access, S3Guard access if needed, and KMS operations.
    
    Specified by:
    
    listAWSPolicyRules in interface AWSPolicyProvider
    
    Parameters:
    
    access - access level desired.
    
    Returns:
    
    a policy for use in roles
  - getDefaultBlockSize
```
@Deprecated
public long getDefaultBlockSize()
```
    Deprecated. use FileSystem.getDefaultBlockSize(Path) instead
    
    Return the number of bytes that large input files should be optimally be split into to minimize I/O time.
    
    Overrides:
    
    getDefaultBlockSize in class org.apache.hadoop.fs.FileSystem
  - getDirectoryMarkerPolicy
```
public DirectoryPolicy getDirectoryMarkerPolicy()
```
    Get the directory marker policy of this filesystem.
    
    Returns:
    
    the marker policy.
  - toString
```
public String toString()
```
    Overrides:
    
    toString in class Object
  - getPartitionSize
```
public long getPartitionSize()
```
    Get the partition size for multipart operations.
    
    Returns:
    
    the value as set during initialization
  - getMultiPartThreshold
```
public long getMultiPartThreshold()
```
    Get the threshold for multipart files.
    
    Returns:
    
    the value as set during initialization
  - isMagicCommitEnabled
```
public boolean isMagicCommitEnabled()
```
    Is magic commit enabled?
    
    Returns:
    
    true if magic commit support is turned on.
  - isMagicCommitPath
```
public boolean isMagicCommitPath(org.apache.hadoop.fs.Path path)
```
    Predicate: is a path a magic commit path? True if magic commit is enabled and the path qualifies as special.
    
    Parameters:
    
    path - path to examine
    
    Returns:
    
    true if the path is or is under a magic directory
  - globStatus
```
public org.apache.hadoop.fs.FileStatus[] globStatus(org.apache.hadoop.fs.Path pathPattern)
                                             throws IOException
```
    Increments the statistic Statistic.INVOCATION_GLOB_STATUS. Override superclass so as to disable symlink resolution as symlinks are not supported by S3A.
    
    Overrides:
    
    globStatus in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    IOException
  - globStatus
```
@AuditEntryPoint
public org.apache.hadoop.fs.FileStatus[] globStatus(org.apache.hadoop.fs.Path pathPattern,
                                                                     org.apache.hadoop.fs.PathFilter filter)
                                                              throws IOException
```
    Increments the statistic Statistic.INVOCATION_GLOB_STATUS. Override superclass so as to disable symlink resolution as symlinks are not supported by S3A. Although an AuditEntryPoint, the globber itself will talk do the filesystem through the filesystem API, so its operations will all appear part of separate operations.
    
    Overrides:
    
    globStatus in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    IOException
  - exists
```
@AuditEntryPoint
public boolean exists(org.apache.hadoop.fs.Path f)
                                throws IOException
```
    Override superclass so as to add statistic collection.
    
    Overrides:
    
    exists in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    IOException
  - isDirectory
```
@AuditEntryPoint
public boolean isDirectory(org.apache.hadoop.fs.Path f)
                                     throws IOException
```
    Optimized probe for a path referencing a dir. Even though it is optimized to a single HEAD, applications should not over-use this method...it is all too common.
    
    Overrides:
    
    isDirectory in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    IOException
  - isFile
```
@AuditEntryPoint
public boolean isFile(org.apache.hadoop.fs.Path f)
                                throws IOException
```
    Optimized probe for a path referencing a file. Even though it is optimized to a single HEAD, applications should not over-use this method...it is all too common.
    
    Overrides:
    
    isFile in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    IOException
  - getFileChecksum
```
@Retries.RetryTranslated
 @AuditEntryPoint
public org.apache.hadoop.fs.store.EtagChecksum getFileChecksum(org.apache.hadoop.fs.Path f,
                                                                                                          long length)
                                                                                                   throws IOException
```
    When enabled, get the etag of a object at the path via HEAD request and return it as a checksum object.
    1. If a tag has not changed, consider the object unchanged.
    2. Two tags being different does not imply the data is different.
    Different S3 implementations may offer different guarantees. This check is (currently) only made if Constants.ETAG_CHECKSUM_ENABLED is set; turning it on has caused problems with Distcp (HADOOP-15273).
    Overrides:
    
    getFileChecksum in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    f - The file path
    
    length - The length of the file range for checksum calculation
    
    Returns:
    
    The EtagChecksum or null if checksums are not enabled or supported.
    
    Throws:
    
    IOException - IO failure
    
    See Also:
    
    Common Response Headers
  - getXAttr
```
@AuditEntryPoint
public byte[] getXAttr(org.apache.hadoop.fs.Path path,
                                        String name)
                                 throws IOException
```
    Overrides:
    
    getXAttr in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    IOException
  - getXAttrs
```
@AuditEntryPoint
public Map<String,byte[]> getXAttrs(org.apache.hadoop.fs.Path path)
                                              throws IOException
```
    Overrides:
    
    getXAttrs in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    IOException
  - getXAttrs
```
@AuditEntryPoint
public Map<String,byte[]> getXAttrs(org.apache.hadoop.fs.Path path,
                                                     List<String> names)
                                              throws IOException
```
    Overrides:
    
    getXAttrs in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    IOException
  - listXAttrs
```
@AuditEntryPoint
public List<String> listXAttrs(org.apache.hadoop.fs.Path path)
                                         throws IOException
```
    Overrides:
    
    listXAttrs in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    IOException
  - createHeaderProcessingCallbacks
```
protected HeaderProcessing.HeaderProcessingCallbacks createHeaderProcessingCallbacks()
```
    Create the callbacks.
    
    Returns:
    
    An implementation of the header processing callbacks.
  - listFiles
```
@Retries.RetryTranslated
 @AuditEntryPoint
public org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> listFiles(org.apache.hadoop.fs.Path f,
                                                                                                                                        boolean recursive)
                                                                                                                                 throws FileNotFoundException,
                                                                                                                                        IOException
```
    . This implementation is optimized for S3, which can do a bulk listing off all entries under a path in one single operation. Thus there is no need to recursively walk the directory tree. Instead a ListObjectsRequest is created requesting a (windowed) listing of all entries under the given path. This is used to construct an ObjectListingIterator instance, iteratively returning the sequence of lists of elements under the path. This is then iterated over in a FileStatusListingIterator, which generates S3AFileStatus instances, one per listing entry. These are then translated into LocatedFileStatus instances. This is essentially a nested and wrapped set of iterators, with some generator classes.
    
    Overrides:
    
    listFiles in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    f - a path
    
    recursive - if the subdirectories need to be traversed recursively
    
    Returns:
    
    an iterator that traverses statuses of the files/directories in the given path
    
    Throws:
    
    FileNotFoundException - if path does not exist
    
    IOException - if any I/O error occurred
  - listFilesAndEmptyDirectories
```
@InterfaceAudience.Private
 @Retries.RetryTranslated
 @AuditEntryPoint
public org.apache.hadoop.fs.RemoteIterator<S3ALocatedFileStatus> listFilesAndEmptyDirectories(org.apache.hadoop.fs.Path f,
                                                                                                                                                                     boolean recursive)
                                                                                                                                                              throws IOException
```
    Recursive List of files and empty directories.
    
    Parameters:
    
    f - path to list from
    
    recursive - recursive?
    
    Returns:
    
    an iterator.
    
    Throws:
    
    IOException - failure
  - listFilesAndEmptyDirectoriesForceNonAuth
```
@InterfaceAudience.Private
 @Retries.RetryTranslated
 @AuditEntryPoint
public org.apache.hadoop.fs.RemoteIterator<S3ALocatedFileStatus> listFilesAndEmptyDirectoriesForceNonAuth(org.apache.hadoop.fs.Path f,
                                                                                                                                                                                 boolean recursive)
                                                                                                                                                                          throws IOException
```
    Recursive List of files and empty directories, force metadatastore to act like it is non-authoritative.
    
    Parameters:
    
    f - path to list from
    
    recursive - recursive listing?
    
    Returns:
    
    an iterator.
    
    Throws:
    
    IOException - failure
  - listLocatedStatus
```
public org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> listLocatedStatus(org.apache.hadoop.fs.Path f)
                                                                                              throws FileNotFoundException,
                                                                                                     IOException
```
    Override superclass so as to add statistic collection.
    
    Overrides:
    
    listLocatedStatus in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    FileNotFoundException
    
    IOException
  - listLocatedStatus
```
@Retries.OnceTranslated(value="s3guard not retrying")
 @AuditEntryPoint
public org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> listLocatedStatus(org.apache.hadoop.fs.Path f,
                                                                                                                                                                             org.apache.hadoop.fs.PathFilter filter)
                                                                                                                                                                      throws FileNotFoundException,
                                                                                                                                                                             IOException
```
    . S3 Optimized directory listing. The initial operation performs the first bulk listing; extra listings will take place when all the current set of results are used up.
    
    Overrides:
    
    listLocatedStatus in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    f - a path
    
    filter - a path filter
    
    Returns:
    
    an iterator that traverses statuses of the files/directories in the given path
    
    Throws:
    
    FileNotFoundException - if path does not exist
    
    IOException - if any I/O error occurred
  - listUploads
```
@InterfaceAudience.Private
 @Retries.RetryTranslated
 @AuditEntryPoint
public MultipartUtils.UploadIterator listUploads(@Nullable
                                                                                                                        String prefix)
                                                                                                                 throws IOException
```
    List any pending multipart uploads whose keys begin with prefix, using an iterator that can handle an unlimited number of entries. See listMultipartUploads(String) for a non-iterator version of this.
    
    Parameters:
    
    prefix - optional key prefix to search
    
    Returns:
    
    Iterator over multipart uploads.
    
    Throws:
    
    IOException - on failure
  - listMultipartUploads
```
@InterfaceAudience.Private
 @Retries.RetryTranslated
public List<com.amazonaws.services.s3.model.MultipartUpload> listMultipartUploads(String prefix)
                                                                                                                                throws IOException
```
    Listing all multipart uploads; limited to the first few hundred. See listUploads(String) for an iterator-based version that does not limit the number of entries returned. Retry policy: retry, translated.
    
    Parameters:
    
    prefix - prefix to scan for, "" for none
    
    Returns:
    
    a listing of multipart uploads.
    
    Throws:
    
    IOException - IO failure, including any uprated AmazonClientException
  - newCommitterStatistics
```
public CommitterStatistics newCommitterStatistics()
```
    Create a new instance of the committer statistics.
    
    Returns:
    
    a new committer statistics instance
  - hasPathCapability
```
public boolean hasPathCapability(org.apache.hadoop.fs.Path path,
                                 String capability)
                          throws IOException
```
    Specified by:
    
    hasPathCapability in interface org.apache.hadoop.fs.PathCapabilities
    
    Overrides:
    
    hasPathCapability in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    IOException
  - hasCapability
```
@Deprecated
public boolean hasCapability(String capability)
```
    Deprecated.
    
    Return the capabilities of this filesystem instance. This has been supplanted by hasPathCapability(Path, String).
    
    Specified by:
    
    hasCapability in interface org.apache.hadoop.fs.StreamCapabilities
    
    Parameters:
    
    capability - string to query the stream support for.
    
    Returns:
    
    whether the FS instance has the capability.
  - shareCredentials
```
public AWSCredentialProviderList shareCredentials(String purpose)
```
    Get a shared copy of the AWS credentials, with its reference counter updated. Caller is required to call close() on this after they have finished using it.
    
    Parameters:
    
    purpose - what is this for? This is initially for logging
    
    Returns:
    
    a reference to shared credentials.
  - getTtlTimeProvider
```
public ITtlTimeProvider getTtlTimeProvider()
```
  - setTtlTimeProvider
```
protected void setTtlTimeProvider(ITtlTimeProvider ttlTimeProvider)
```
  - openFileWithOptions
```
@Retries.RetryTranslated
 @AuditEntryPoint
public CompletableFuture<org.apache.hadoop.fs.FSDataInputStream> openFileWithOptions(org.apache.hadoop.fs.Path rawPath,
                                                                                                                                org.apache.hadoop.fs.impl.OpenFileParameters parameters)
                                                                                                                         throws IOException
```
    Initiate the open or select operation. This is invoked from both the FileSystem and FileContext APIs. It's declared as an audit entry point but the span creation is pushed down into the open/select methods it ultimately calls.
    
    Overrides:
    
    openFileWithOptions in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    rawPath - path to the file
    
    parameters - open file parameters from the builder.
    
    Returns:
    
    a future which will evaluate to the opened/selected file.
    
    Throws:
    
    IOException - failure to resolve the link.
    
    org.apache.hadoop.fs.PathIOException - operation is a select request but S3 select is disabled
    
    IllegalArgumentException - unknown mandatory key
  - createMultipartUploader
```
@AuditEntryPoint
public S3AMultipartUploaderBuilder createMultipartUploader(org.apache.hadoop.fs.Path basePath)
                                                                     throws IOException
```
    Overrides:
    
    createMultipartUploader in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    IOException
  - createStoreContext
```
@InterfaceAudience.Private
public StoreContext createStoreContext()
```
    Build an immutable store context. If called while the FS is being initialized, some of the context will be incomplete. new store context instances should be created as appropriate.
    
    Returns:
    
    the store context of this FS.
  - createMarkerToolOperations
```
@AuditEntryPoint
 @InterfaceAudience.Private
public MarkerToolOperations createMarkerToolOperations(String target)
                                                                                             throws IOException
```
    Create a marker tools operations binding for this store. Auditing:
    
    Parameters:
    
    target - target path
    
    Returns:
    
    callbacks for operations.
    
    Throws:
    
    IOException - if raised during span creation
  - initializeClass
```
@InterfaceAudience.Private
public static void initializeClass()
```
    This is purely for testing, as it force initializes all static initializers. See HADOOP-17385 for details.
  - isCSEEnabled
```
public boolean isCSEEnabled()
```
    a method to know if Client side encryption is enabled or not.
    
    Returns:
    
    a boolean stating if CSE is enabled.

Class S3AFileSystem

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem

Nested classes/interfaces inherited from interface org.apache.hadoop.fs.StreamCapabilities

Nested classes/interfaces inherited from interface org.apache.hadoop.fs.s3a.auth.delegation.AWSPolicyProvider

Field Summary

Fields inherited from class org.apache.hadoop.fs.FileSystem

Fields inherited from interface org.apache.hadoop.fs.StreamCapabilities

Constructor Summary

Method Summary

Methods inherited from class org.apache.hadoop.fs.FileSystem

Methods inherited from class org.apache.hadoop.conf.Configured

Methods inherited from class java.lang.Object

Methods inherited from interface org.apache.hadoop.security.token.DelegationTokenIssuer

Field Detail

DEFAULT_BLOCKSIZE

LOG

Constructor Detail

S3AFileSystem

Method Detail

initialize

initializeStatisticsBinding

createStorageStatistics

verifyBucketExists

verifyBucketExistsV2

getInstrumentation

getFsStatistics

getListing

initializeAuditService

getAuditManager

getAuditor

getActiveAuditSpan

getAuditSpanSource

createSpan

createRequestFactory

getRequestFactory

createDelegationOperations

setEncryptionSecrets

getEncryptionSecrets

abortOutstandingMultipartUploads

getScheme

getUri

setUri

getCanonicalUri

getDefaultPort

getAmazonS3ClientForTesting

setAmazonS3Client

getBucketLocation

getBucketLocation

getInputPolicy

getChangeDetectionPolicy

getS3EncryptionAlgorithm

getBucket

setBucket

setInputPolicy

pathToKey

maybeAddTrailingSlash

keyToQualifiedPath

makeQualified

qualify

checkPath

canonicalizeUri

open

createReadContext

create

getWriteOperationHelper

createWriteOperationHelper

createNonRecursive

append

rename

getFsDelegationToken

getObjectMetadata

hasMetadataStore

hasAuthoritativeMetadataStore

getMetadataStore

entryPoint

entryPoint

incrementStatistic

incrementStatistic

decrementGauge