| Package | Description |
|---|---|
| org.apache.hadoop.fs.s3a |
S3A Filesystem.
|
| org.apache.hadoop.fs.s3a.auth |
Authentication and permissions support.
|
| org.apache.hadoop.fs.s3a.auth.delegation |
Extensible delegation token support for the S3A connector.
|
| org.apache.hadoop.fs.s3a.impl |
Implementation classes private to the S3A store.
|
| org.apache.hadoop.fs.s3a.s3guard |
This package contains classes related to S3Guard: a feature of S3A to mask
the eventual consistency behavior of S3 and optimize access patterns by
coordinating with a strongly consistent external store for file system
metadata.
|
| org.apache.hadoop.fs.s3a.select |
Support for S3 Select.
|
| org.apache.hadoop.fs.s3a.tools |
S3A Command line tools independent of S3Guard.
|
| Modifier and Type | Method and Description |
|---|---|
void |
WriteOperationHelper.abortMultipartCommit(String destKey,
String uploadId)
Abort a multipart commit operation.
|
void |
WriteOperations.abortMultipartCommit(String destKey,
String uploadId)
Abort a multipart commit operation.
|
void |
WriteOperationHelper.abortMultipartUpload(com.amazonaws.services.s3.model.MultipartUpload upload)
Abort a multipart commit operation.
|
void |
WriteOperations.abortMultipartUpload(com.amazonaws.services.s3.model.MultipartUpload upload)
Abort a multipart commit operation.
|
void |
WriteOperationHelper.abortMultipartUpload(String destKey,
String uploadId,
boolean shouldRetry,
Invoker.Retried retrying)
Abort a multipart upload operation.
|
void |
WriteOperations.abortMultipartUpload(String destKey,
String uploadId,
boolean shouldRetry,
Invoker.Retried retrying)
Abort a multipart upload operation.
|
int |
WriteOperationHelper.abortMultipartUploadsUnderPath(String prefix)
Abort multipart uploads under a path: limited to the first
few hundred.
|
int |
WriteOperations.abortMultipartUploadsUnderPath(String prefix)
Abort multipart uploads under a path: limited to the first
few hundred.
|
void |
S3AFileSystem.abortOutstandingMultipartUploads(long seconds)
Abort all outstanding MPUs older than a given age.
|
com.amazonaws.services.s3.model.CompleteMultipartUploadResult |
WriteOperationHelper.commitUpload(String destKey,
String uploadId,
List<com.amazonaws.services.s3.model.PartETag> partETags,
long length,
BulkOperationState operationState)
This completes a multipart upload to the destination key via
finalizeMultipartUpload(). |
com.amazonaws.services.s3.model.CompleteMultipartUploadResult |
WriteOperations.commitUpload(String destKey,
String uploadId,
List<com.amazonaws.services.s3.model.PartETag> partETags,
long length,
BulkOperationState operationState)
This completes a multipart upload to the destination key via
finalizeMultipartUpload(). |
com.amazonaws.services.s3.model.CompleteMultipartUploadResult |
WriteOperationHelper.completeMPUwithRetries(String destKey,
String uploadId,
List<com.amazonaws.services.s3.model.PartETag> partETags,
long length,
AtomicInteger errorCount)
This completes a multipart upload to the destination key via
finalizeMultipartUpload(). |
com.amazonaws.services.s3.model.CompleteMultipartUploadResult |
WriteOperations.completeMPUwithRetries(String destKey,
String uploadId,
List<com.amazonaws.services.s3.model.PartETag> partETags,
long length,
AtomicInteger errorCount)
This completes a multipart upload to the destination key via
finalizeMultipartUpload(). |
boolean |
S3AFileSystem.delete(org.apache.hadoop.fs.Path f,
boolean recursive)
Delete a Path.
|
String |
S3AFileSystem.getBucketLocation()
Get the region of a bucket.
|
String |
S3AFileSystem.getBucketLocation(String bucketName)
Get the region of a bucket; fixing up the region so it can be used
in the builders of other AWS clients.
|
org.apache.hadoop.fs.ContentSummary |
S3AFileSystem.getContentSummary(org.apache.hadoop.fs.Path f)
This is a very slow operation against object storage.
|
org.apache.hadoop.fs.store.EtagChecksum |
S3AFileSystem.getFileChecksum(org.apache.hadoop.fs.Path f,
long length)
When enabled, get the etag of a object at the path via HEAD request and
return it as a checksum object.
|
org.apache.hadoop.fs.FileStatus |
S3AFileSystem.getFileStatus(org.apache.hadoop.fs.Path f)
Return a file status object that represents the path.
|
com.amazonaws.services.s3.model.ObjectMetadata |
S3AFileSystem.getObjectMetadata(org.apache.hadoop.fs.Path path)
Low-level call to get at the object metadata.
|
String |
WriteOperationHelper.initiateMultiPartUpload(String destKey)
Start the multipart upload process.
|
String |
WriteOperations.initiateMultiPartUpload(String destKey)
Start the multipart upload process.
|
org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> |
S3AFileSystem.listFiles(org.apache.hadoop.fs.Path f,
boolean recursive)
.
|
org.apache.hadoop.fs.RemoteIterator<S3ALocatedFileStatus> |
S3AFileSystem.listFilesAndEmptyDirectories(org.apache.hadoop.fs.Path f,
boolean recursive)
Recursive List of files and empty directories.
|
org.apache.hadoop.fs.RemoteIterator<S3ALocatedFileStatus> |
S3AFileSystem.listFilesAndEmptyDirectoriesForceNonAuth(org.apache.hadoop.fs.Path f,
boolean recursive)
Recursive List of files and empty directories, force metadatastore
to act like it is non-authoritative.
|
List<com.amazonaws.services.s3.model.MultipartUpload> |
WriteOperationHelper.listMultipartUploads(String prefix) |
List<com.amazonaws.services.s3.model.MultipartUpload> |
S3AFileSystem.listMultipartUploads(String prefix)
Listing all multipart uploads; limited to the first few hundred.
|
MultipartUtils.UploadIterator |
S3AFileSystem.listUploads(String prefix)
List any pending multipart uploads whose keys begin with prefix, using
an iterator that can handle an unlimited number of entries.
|
protected void |
S3AFileSystem.maybeCreateFakeParentDirectory(org.apache.hadoop.fs.Path path)
Create a fake parent directory if required.
|
void |
Invoker.maybeRetry(boolean doRetry,
String action,
String path,
boolean idempotent,
org.apache.hadoop.util.functional.InvocationRaisingIOE operation)
Execute a void operation with the default retry callback invoked when
doRetry=true, else just once.
|
<T> T |
Invoker.maybeRetry(boolean doRetry,
String action,
String path,
boolean idempotent,
Invoker.Retried retrying,
org.apache.hadoop.util.functional.CallableRaisingIOE<T> operation)
Execute a function with retry processing when doRetry=true, else just once.
|
void |
Invoker.maybeRetry(boolean doRetry,
String action,
String path,
boolean idempotent,
Invoker.Retried retrying,
org.apache.hadoop.util.functional.InvocationRaisingIOE operation)
Execute a void operation with retry processing when doRetry=true, else
just once.
|
org.apache.hadoop.fs.FSDataInputStream |
S3AFileSystem.open(org.apache.hadoop.fs.Path f,
int bufferSize)
Opens an FSDataInputStream at the indicated Path.
|
CompletableFuture<org.apache.hadoop.fs.FSDataInputStream> |
S3AFileSystem.openFileWithOptions(org.apache.hadoop.fs.Path rawPath,
org.apache.hadoop.fs.impl.OpenFileParameters parameters)
Initiate the open or select operation.
|
com.amazonaws.services.s3.model.PutObjectResult |
WriteOperationHelper.putObject(com.amazonaws.services.s3.model.PutObjectRequest putObjectRequest)
PUT an object directly (i.e.
|
com.amazonaws.services.s3.model.PutObjectResult |
WriteOperations.putObject(com.amazonaws.services.s3.model.PutObjectRequest putObjectRequest)
PUT an object directly (i.e.
|
int |
S3AInputStream.read() |
int |
S3AInputStream.read(byte[] buf,
int off,
int len)
This updates the statistics on read operations started and whether
or not the read operation "completed", that is: returned the exact
number of bytes requested.
|
void |
S3AInputStream.readFully(long position,
byte[] buffer,
int offset,
int length)
Subclass
readFully() operation which only seeks at the start
of the series of operations; seeking back at the end. |
boolean |
S3AFileSystem.rename(org.apache.hadoop.fs.Path src,
org.apache.hadoop.fs.Path dst)
Renames Path src to Path dst.
|
<T> T |
Invoker.retry(String action,
String path,
boolean idempotent,
org.apache.hadoop.util.functional.CallableRaisingIOE<T> operation)
Execute a function with the default retry callback invoked.
|
void |
Invoker.retry(String action,
String path,
boolean idempotent,
org.apache.hadoop.util.functional.InvocationRaisingIOE operation)
Execute a void operation with the default retry callback invoked.
|
<T> T |
Invoker.retry(String action,
String path,
boolean idempotent,
Invoker.Retried retrying,
org.apache.hadoop.util.functional.CallableRaisingIOE<T> operation)
Execute a function with retry processing.
|
void |
Invoker.retry(String action,
String path,
boolean idempotent,
Invoker.Retried retrying,
org.apache.hadoop.util.functional.InvocationRaisingIOE operation)
Execute a void operation with retry processing.
|
com.amazonaws.services.s3.model.SelectObjectContentResult |
WriteOperationHelper.select(org.apache.hadoop.fs.Path source,
com.amazonaws.services.s3.model.SelectObjectContentRequest request,
String action)
Execute an S3 Select operation.
|
com.amazonaws.services.s3.model.SelectObjectContentResult |
WriteOperations.select(org.apache.hadoop.fs.Path source,
com.amazonaws.services.s3.model.SelectObjectContentRequest request,
String action)
Execute an S3 Select operation.
|
com.amazonaws.services.s3.transfer.model.UploadResult |
WriteOperationHelper.uploadObject(com.amazonaws.services.s3.model.PutObjectRequest putObjectRequest)
PUT an object via the transfer manager.
|
com.amazonaws.services.s3.transfer.model.UploadResult |
WriteOperations.uploadObject(com.amazonaws.services.s3.model.PutObjectRequest putObjectRequest)
PUT an object via the transfer manager.
|
com.amazonaws.services.s3.model.UploadPartResult |
WriteOperationHelper.uploadPart(com.amazonaws.services.s3.model.UploadPartRequest request)
Upload part of a multi-partition file.
|
com.amazonaws.services.s3.model.UploadPartResult |
WriteOperations.uploadPart(com.amazonaws.services.s3.model.UploadPartRequest request)
Upload part of a multi-partition file.
|
protected void |
S3AFileSystem.verifyBucketExists()
Verify that the bucket exists.
|
protected void |
S3AFileSystem.verifyBucketExistsV2()
Verify that the bucket exists.
|
| Constructor and Description |
|---|
UploadIterator(StoreContext storeContext,
com.amazonaws.services.s3.AmazonS3 s3,
int maxKeys,
String prefix) |
| Modifier and Type | Method and Description |
|---|---|
com.amazonaws.services.securitytoken.model.Credentials |
STSClientFactory.STSClient.requestRole(String roleARN,
String sessionName,
String policy,
long duration,
TimeUnit timeUnit)
Request a set of role credentials.
|
static MarshalledCredentials |
MarshalledCredentialBinding.requestSessionCredentials(com.amazonaws.auth.AWSCredentialsProvider parentCredentials,
com.amazonaws.ClientConfiguration awsConf,
String stsEndpoint,
String stsRegion,
int duration,
Invoker invoker)
Request a set of credentials from an STS endpoint.
|
com.amazonaws.services.securitytoken.model.Credentials |
STSClientFactory.STSClient.requestSessionCredentials(long duration,
TimeUnit timeUnit)
Request a set of session credentials.
|
| Modifier and Type | Method and Description |
|---|---|
SessionTokenIdentifier |
SessionTokenBinding.createTokenIdentifier(Optional<RoleModel.Policy> policy,
EncryptionSecrets encryptionSecrets,
org.apache.hadoop.io.Text renewer) |
RoleTokenIdentifier |
RoleTokenBinding.createTokenIdentifier(Optional<RoleModel.Policy> policy,
EncryptionSecrets encryptionSecrets,
org.apache.hadoop.io.Text renewer)
Create the Token Identifier.
|
| Modifier and Type | Method and Description |
|---|---|
com.amazonaws.services.s3.transfer.model.CopyResult |
OperationCallbacks.copyFile(String srcKey,
String destKey,
S3ObjectAttributes srcAttributes,
S3AReadOpContext readContext)
Copy a single object in the bucket via a COPY operation.
|
void |
MkdirOperation.MkdirCallbacks.createFakeDirectory(String key)
Create a fake directory, always ending in "/".
|
protected void |
DeleteOperation.deleteDirectoryTree(org.apache.hadoop.fs.Path path,
String dirKey)
Delete a directory tree.
|
void |
OperationCallbacks.deleteObjectAtPath(org.apache.hadoop.fs.Path path,
String key,
boolean isFile,
BulkOperationState operationState)
Delete an object, also updating the metastore.
|
Boolean |
DeleteOperation.execute()
Delete a file or directory tree.
|
Boolean |
MkdirOperation.execute()
Make the given path and all non-existent parents into
directories.
|
Void |
CopyFromLocalOperation.execute()
Executes the
CopyFromLocalOperation. |
org.apache.hadoop.fs.ContentSummary |
GetContentSummaryOperation.execute()
Return the
ContentSummary of a given path. |
String |
ContextAccessors.getBucketLocation()
Get the region of a bucket.
|
com.amazonaws.services.s3.model.ObjectMetadata |
HeaderProcessing.HeaderProcessingCallbacks.getObjectMetadata(String key)
Retrieve the object metadata.
|
org.apache.hadoop.fs.RemoteIterator<S3ALocatedFileStatus> |
OperationCallbacks.listFilesAndDirectoryMarkers(org.apache.hadoop.fs.Path path,
S3AFileStatus status,
boolean collectTombstones,
boolean includeSelf)
Recursive list of files and directory markers.
|
org.apache.hadoop.fs.RemoteIterator<S3AFileStatus> |
OperationCallbacks.listObjects(org.apache.hadoop.fs.Path path,
String key)
Create an iterator over objects in S3 only; S3Guard
is not involved.
|
S3AFileStatus |
MkdirOperation.MkdirCallbacks.probePathStatus(org.apache.hadoop.fs.Path path,
Set<StatusProbeEnum> probes)
Get the status of a path.
|
S3AFileStatus |
GetContentSummaryOperation.GetContentSummaryCallbacks.probePathStatus(org.apache.hadoop.fs.Path path,
Set<StatusProbeEnum> probes)
Get the status of a path.
|
| Modifier and Type | Method and Description |
|---|---|
static void |
S3Guard.addAncestors(MetadataStore metadataStore,
org.apache.hadoop.fs.Path qualifiedPath,
ITtlTimeProvider timeProvider,
BulkOperationState operationState)
This adds all new ancestors of a path as directories.
|
void |
DynamoDBMetadataStore.addAncestors(org.apache.hadoop.fs.Path qualifiedPath,
BulkOperationState operationState)
This adds all new ancestors of a path as directories.
|
void |
MetadataStore.addAncestors(org.apache.hadoop.fs.Path qualifiedPath,
BulkOperationState operationState)
This adds all new ancestors of a path as directories.
|
void |
DynamoDBMetadataStore.delete(org.apache.hadoop.fs.Path path,
BulkOperationState operationState) |
void |
DynamoDBMetadataStore.deletePaths(Collection<org.apache.hadoop.fs.Path> paths,
BulkOperationState operationState) |
void |
MetadataStore.deletePaths(Collection<org.apache.hadoop.fs.Path> paths,
BulkOperationState operationState)
Delete the paths.
|
void |
DynamoDBMetadataStore.deleteSubtree(org.apache.hadoop.fs.Path path,
BulkOperationState operationState) |
void |
MetadataStore.deleteSubtree(org.apache.hadoop.fs.Path path,
BulkOperationState operationState)
Deletes the entire sub-tree rooted at the given path, leaving tombstones
to prevent lingering, inconsistent copies of it from being listed.
|
void |
DynamoDBMetadataStore.destroy() |
void |
DynamoDBMetadataStoreTableManager.destroy() |
void |
DynamoDBMetadataStore.forgetMetadata(org.apache.hadoop.fs.Path path) |
DDBPathMetadata |
DynamoDBMetadataStore.get(org.apache.hadoop.fs.Path path) |
DDBPathMetadata |
DynamoDBMetadataStore.get(org.apache.hadoop.fs.Path path,
boolean wantEmptyDirectoryFlag) |
protected com.amazonaws.services.dynamodbv2.document.Item |
DynamoDBMetadataStoreTableManager.getVersionMarkerItem()
Get the version mark item in the existing DynamoDB table.
|
DirListingMetadata |
DynamoDBMetadataStore.listChildren(org.apache.hadoop.fs.Path path) |
DirListingMetadata |
MetadataStore.listChildren(org.apache.hadoop.fs.Path path)
Lists metadata for all direct children of a path.
|
static DirListingMetadata |
S3Guard.listChildrenWithTtl(MetadataStore ms,
org.apache.hadoop.fs.Path path,
ITtlTimeProvider timeProvider,
boolean allowAuthoritative)
List children; mark the result as non-auth if the TTL has expired.
|
void |
DynamoDBMetadataStore.move(Collection<org.apache.hadoop.fs.Path> pathsToDelete,
Collection<PathMetadata> pathsToCreate,
BulkOperationState operationState)
Record the effects of a
FileSystem.rename(Path, Path) in the
MetadataStore. |
void |
DynamoDBMetadataStore.prune(MetadataStore.PruneMode pruneMode,
long cutoff) |
long |
DynamoDBMetadataStore.prune(MetadataStore.PruneMode pruneMode,
long cutoff,
String keyPrefix)
Prune files, in batches.
|
void |
DynamoDBMetadataStore.put(Collection<? extends PathMetadata> metas,
BulkOperationState operationState) |
void |
DynamoDBMetadataStore.put(DirListingMetadata meta,
List<org.apache.hadoop.fs.Path> unchangedEntries,
BulkOperationState operationState)
Save directory listing metadata.
|
void |
DynamoDBMetadataStore.put(PathMetadata meta) |
void |
MetadataStore.put(PathMetadata meta)
Saves metadata for exactly one path.
|
void |
DynamoDBMetadataStore.put(PathMetadata meta,
BulkOperationState operationState) |
void |
MetadataStore.put(PathMetadata meta,
BulkOperationState operationState)
Saves metadata for exactly one path, potentially
using any bulk operation state to eliminate duplicate work.
|
static S3AFileStatus |
S3Guard.putAndReturn(MetadataStore ms,
S3AFileStatus status,
ITtlTimeProvider timeProvider)
Helper function which puts a given S3AFileStatus into the MetadataStore and
returns the same S3AFileStatus.
|
static S3AFileStatus |
S3Guard.putAndReturn(MetadataStore ms,
S3AFileStatus status,
ITtlTimeProvider timeProvider,
BulkOperationState operationState)
Helper function which puts a given S3AFileStatus into the MetadataStore and
returns the same S3AFileStatus.
|
static void |
S3Guard.putAuthDirectoryMarker(MetadataStore ms,
S3AFileStatus status,
ITtlTimeProvider timeProvider,
BulkOperationState operationState)
Creates an authoritative directory marker for the store.
|
static boolean |
S3Guard.refreshEntry(MetadataStore metadataStore,
PathMetadata pm,
S3AFileStatus s3AFileStatus,
ITtlTimeProvider timeProvider)
We update the metastore for the specific case of S3 value == S3Guard value
so as to place a more recent modtime in the store.
|
| Modifier and Type | Method and Description |
|---|---|
org.apache.hadoop.fs.FSDataInputStream |
SelectBinding.select(S3AReadOpContext readContext,
String expression,
org.apache.hadoop.conf.Configuration builderOptions,
S3ObjectAttributes objectAttributes)
Build and execute a select request.
|
| Modifier and Type | Method and Description |
|---|---|
org.apache.hadoop.fs.RemoteIterator<S3AFileStatus> |
MarkerToolOperations.listObjects(org.apache.hadoop.fs.Path path,
String key)
Create an iterator over objects in S3 only; S3Guard
is not involved.
|
Copyright © 2008–2022 Apache Software Foundation. All rights reserved.