org.apache.hadoop.fs.s3a.commit.staging

Class Paths

java.lang.Object

org.apache.hadoop.fs.s3a.commit.staging.Paths

public final class Paths
extends Object

Path operations for the staging committers.

Method Summary

Modifier and Type	Method and Description
static String	addUUID(String pathStr, String uuid) Insert the UUID to a path if it is not there already.
static void	clearTempFolderInfo(org.apache.hadoop.mapreduce.TaskAttemptID attemptID) Remove all information held about task attempts.
static org.apache.hadoop.fs.Path	getLocalTaskAttemptTempDir(org.apache.hadoop.conf.Configuration conf, String uuid, org.apache.hadoop.mapreduce.TaskAttemptID attemptID) Get the task attempt temporary directory in the local filesystem.
static org.apache.hadoop.fs.Path	getMultipartUploadCommitsDirectory(org.apache.hadoop.conf.Configuration conf, String uuid) Build a qualified temporary path for the multipart upload commit information in the cluster filesystem.
static String	getParent(String pathStr) Get the parent path of a string path: everything up to but excluding the last "/" in the path.
protected static String	getPartition(String relative) Returns the partition of a relative file path, or null if the path is a file name with no relative directory.
static Set<String>	getPartitions(org.apache.hadoop.fs.Path attemptPath, List<? extends org.apache.hadoop.fs.FileStatus> taskOutput) Get the set of partitions from the list of files being staged.
static String	getRelativePath(org.apache.hadoop.fs.Path basePath, org.apache.hadoop.fs.Path fullPath) Using URI#relativize(), build the relative path from the base path to the full path.
static org.apache.hadoop.fs.Path	getStagingUploadsParentDirectory(org.apache.hadoop.conf.Configuration conf, String uuid) Build a qualified parent path for the temporary multipart upload commit directory built by getMultipartUploadCommitsDirectory(Configuration, String).
static org.apache.hadoop.fs.Path	path(org.apache.hadoop.fs.Path parent, String... child) Varags constructor of paths.
static void	resetTempFolderCache() Reset the temp folder cache; useful in tests.
static org.apache.hadoop.fs.Path	tempDirForStaging(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf) Try to come up with a good temp directory for different filesystems.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

addUUID

public static String addUUID(String pathStr,
                             String uuid)

Insert the UUID to a path if it is not there already. If there is a trailing "." in the prefix after the last slash, the UUID is inserted before it with a "-" prefix; otherwise appended. Examples:

   /example/part-0000  ==> /example/part-0000-0ab34
   /example/part-0001.gz.csv  ==> /example/part-0001-0ab34.gz.csv
   /example/part-0002-0abc3.gz.csv  ==> /example/part-0002-0abc3.gz.csv
   /example0abc3/part-0002.gz.csv  ==> /example0abc3/part-0002.gz.csv
 

Parameters:

pathStr - path as a string; must not have a trailing "/".

uuid - UUID to append; must not be empty

Returns:

new path.

getParent

public static String getParent(String pathStr)

Get the parent path of a string path: everything up to but excluding the last "/" in the path.

Parameters:

pathStr - path as a string

Returns:

the parent or null if there is no parent.

getRelativePath

public static String getRelativePath(org.apache.hadoop.fs.Path basePath,
                                     org.apache.hadoop.fs.Path fullPath)

Using URI#relativize(), build the relative path from the base path to the full path. If childPath is not a child of basePath the outcome os undefined.

Parameters:

basePath - base path

fullPath - full path under the base path.

Returns:

the relative path

path

public static org.apache.hadoop.fs.Path path(org.apache.hadoop.fs.Path parent,
                                             String... child)

Varags constructor of paths. Not very efficient.

Parameters:

parent - parent path

child - child entries. "" elements are skipped.

Returns:

the full child path.

getLocalTaskAttemptTempDir

public static org.apache.hadoop.fs.Path getLocalTaskAttemptTempDir(org.apache.hadoop.conf.Configuration conf,
                                                                   String uuid,
                                                                   org.apache.hadoop.mapreduce.TaskAttemptID attemptID)
                                                            throws IOException

Get the task attempt temporary directory in the local filesystem. This must be unique to all tasks on all jobs running on all processes on this host. It's constructed as uuid+task-attempt-ID, relying on UUID to be unique for each job.

Parameters:

conf - configuration

uuid - some UUID, such as a job UUID

attemptID - attempt ID

Returns:

a local task attempt directory.

Throws:

IOException - IO problem.

clearTempFolderInfo

public static void clearTempFolderInfo(org.apache.hadoop.mapreduce.TaskAttemptID attemptID)

Remove all information held about task attempts.

Parameters:

attemptID - attempt ID.

resetTempFolderCache

@VisibleForTesting
public static void resetTempFolderCache()

Reset the temp folder cache; useful in tests.

tempDirForStaging

public static org.apache.hadoop.fs.Path tempDirForStaging(org.apache.hadoop.fs.FileSystem fs,
                                                          org.apache.hadoop.conf.Configuration conf)

Try to come up with a good temp directory for different filesystems.

Parameters:

fs - filesystem

conf - configuration

Returns:

a qualified path under which temporary work can go.

getStagingUploadsParentDirectory

public static org.apache.hadoop.fs.Path getStagingUploadsParentDirectory(org.apache.hadoop.conf.Configuration conf,
                                                                         String uuid)
                                                                  throws IOException

Build a qualified parent path for the temporary multipart upload commit directory built by getMultipartUploadCommitsDirectory(Configuration, String).

Parameters:

conf - configuration defining default FS.

uuid - uuid of job

Returns:

a path which can be used for temporary work

Throws:

IOException - on an IO failure.

getMultipartUploadCommitsDirectory

public static org.apache.hadoop.fs.Path getMultipartUploadCommitsDirectory(org.apache.hadoop.conf.Configuration conf,
                                                                           String uuid)
                                                                    throws IOException

Build a qualified temporary path for the multipart upload commit information in the cluster filesystem. Path is built by getMultipartUploadCommitsDirectory(FileSystem, Configuration, String)

Parameters:

conf - configuration defining default FS.

uuid - uuid of job

Returns:

a path which can be used for temporary work

Throws:

IOException - on an IO failure.

getPartition

protected static String getPartition(String relative)

Returns the partition of a relative file path, or null if the path is a file name with no relative directory.

Parameters:

relative - a relative file path

Returns:

the partition of the relative file path

getPartitions

public static Set<String> getPartitions(org.apache.hadoop.fs.Path attemptPath,
                                        List<? extends org.apache.hadoop.fs.FileStatus> taskOutput)
                                 throws IOException

Get the set of partitions from the list of files being staged. This is all immediate parents of those files. If a file is in the root dir, the partition is declared to be StagingCommitterConstants.TABLE_ROOT.

Parameters:

attemptPath - path for the attempt

taskOutput - list of output files.

Returns:

list of partitions.

Throws:

IOException - IO failure