| Package | Description |
|---|---|
| org.apache.hadoop.fs.s3a |
S3A Filesystem.
|
| org.apache.hadoop.fs.s3a.commit |
Support for committing the output of analytics jobs directly to S3.
|
| org.apache.hadoop.fs.s3a.commit.impl |
Internal classes which make use of mapreduce code.
|
| org.apache.hadoop.fs.s3a.commit.magic |
This is the "Magic" committer and support.
|
| org.apache.hadoop.fs.s3a.commit.staging |
The staging committers.
|
| org.apache.hadoop.fs.s3a.impl |
Implementation classes private to the S3A store.
|
| org.apache.hadoop.fs.s3a.s3guard |
This package contained S3Guard support; now the feature has been removed,
its contents are limited to the public command line and some
methods still used by directory marker code.
|
| Constructor and Description |
|---|
ProgressableProgressListener(S3AFileSystem fs,
String key,
org.apache.hadoop.util.Progressable progress)
Instantiate.
|
WriteOperationHelper(S3AFileSystem owner,
org.apache.hadoop.conf.Configuration conf,
S3AStatisticsContext statisticsContext,
org.apache.hadoop.fs.store.audit.AuditSpanSource auditSpanSource,
org.apache.hadoop.fs.store.audit.AuditSpan auditSpan,
WriteOperationHelper.WriteOperationHelperCallbacks writeOperationHelperCallbacks)
Constructor.
|
| Modifier and Type | Method and Description |
|---|---|
S3AFileSystem |
AbstractS3ACommitter.getDestS3AFS()
Get the destination as an S3A Filesystem; casting it.
|
static S3AFileSystem |
CommitUtils.getS3AFileSystem(org.apache.hadoop.fs.Path path,
org.apache.hadoop.conf.Configuration conf,
boolean magicCommitRequired)
Get the S3A FS of a path.
|
static S3AFileSystem |
CommitUtils.verifyIsS3AFS(org.apache.hadoop.fs.FileSystem fs,
org.apache.hadoop.fs.Path path)
Verify that an FS is an S3A FS.
|
| Modifier and Type | Method and Description |
|---|---|
abstract org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter |
AbstractS3ACommitterFactory.createTaskCommitter(S3AFileSystem fileSystem,
org.apache.hadoop.fs.Path outputPath,
org.apache.hadoop.mapreduce.TaskAttemptContext context)
Implementation point: create a task committer for a specific filesystem.
|
org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter |
S3ACommitterFactory.createTaskCommitter(S3AFileSystem fileSystem,
org.apache.hadoop.fs.Path outputPath,
org.apache.hadoop.mapreduce.TaskAttemptContext context)
Create a task committer.
|
static void |
CommitUtils.verifyIsMagicCommitFS(S3AFileSystem fs)
Verify that an S3A FS instance is a magic commit FS.
|
static void |
CommitUtils.verifyIsMagicCommitPath(S3AFileSystem fs,
org.apache.hadoop.fs.Path path)
Verify that the path is a magic one.
|
| Constructor and Description |
|---|
MagicCommitIntegration(S3AFileSystem owner,
boolean magicCommitEnabled)
Instantiate.
|
| Constructor and Description |
|---|
CommitOperations(S3AFileSystem fs)
Instantiate.
|
CommitOperations(S3AFileSystem fs,
CommitterStatistics committerStatistics,
String outputPath)
Instantiate.
|
| Modifier and Type | Method and Description |
|---|---|
org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter |
MagicS3GuardCommitterFactory.createTaskCommitter(S3AFileSystem fileSystem,
org.apache.hadoop.fs.Path outputPath,
org.apache.hadoop.mapreduce.TaskAttemptContext context) |
| Modifier and Type | Method and Description |
|---|---|
org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter |
PartitionedStagingCommitterFactory.createTaskCommitter(S3AFileSystem fileSystem,
org.apache.hadoop.fs.Path outputPath,
org.apache.hadoop.mapreduce.TaskAttemptContext context) |
org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter |
StagingCommitterFactory.createTaskCommitter(S3AFileSystem fileSystem,
org.apache.hadoop.fs.Path outputPath,
org.apache.hadoop.mapreduce.TaskAttemptContext context) |
org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter |
DirectoryStagingCommitterFactory.createTaskCommitter(S3AFileSystem fileSystem,
org.apache.hadoop.fs.Path outputPath,
org.apache.hadoop.mapreduce.TaskAttemptContext context) |
| Constructor and Description |
|---|
S3AMultipartUploaderBuilder(S3AFileSystem fileSystem,
WriteOperations writeOperations,
StoreContext context,
org.apache.hadoop.fs.Path p,
S3AMultipartUploaderStatistics statistics) |
| Modifier and Type | Method and Description |
|---|---|
protected S3AFileSystem |
S3GuardTool.bindFilesystem(org.apache.hadoop.fs.FileSystem bindingFS)
Sets the filesystem; it must be an S3A FS instance, or a FilterFS
around an S3A Filesystem.
|
protected S3AFileSystem |
S3GuardTool.getFilesystem() |
| Modifier and Type | Method and Description |
|---|---|
static boolean |
S3Guard.allowAuthoritative(org.apache.hadoop.fs.Path p,
S3AFileSystem fs,
Collection<String> authPaths)
Is the path for the given FS instance authoritative?
|
static Collection<String> |
S3Guard.getAuthoritativePaths(S3AFileSystem fs) |
Copyright © 2008–2024 Apache Software Foundation. All rights reserved.