org.apache.hadoop.hive.metastore.utils

Class FileUtils

java.lang.Object

org.apache.hadoop.hive.metastore.utils.FileUtils

public class FileUtils
extends Object

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	FileUtils.RemoteIteratorWithFilter

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.hadoop.fs.PathFilter	HIDDEN_FILES_PATH_FILTER

Constructor Summary

Constructors

Constructor and Description
FileUtils()

Method Summary

Modifier and Type	Method and Description
static boolean	copy(org.apache.hadoop.fs.FileSystem srcFS, org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.FileSystem dstFS, org.apache.hadoop.fs.Path dst, boolean deleteSource, boolean overwrite, org.apache.hadoop.conf.Configuration conf) Copies files between filesystems.
static boolean	equalsFileSystem(org.apache.hadoop.fs.FileSystem fs1, org.apache.hadoop.fs.FileSystem fs2) Determine if two objects reference the same file system.
static String	escapePathName(String path)
static String	escapePathName(String path, String defaultPath) Escapes a path name.
static List<org.apache.hadoop.fs.FileStatus>	getFileStatusRecurse(org.apache.hadoop.fs.Path base, org.apache.hadoop.fs.FileSystem fs) Get all file status from a root path and recursively go deep into certain levels.
static org.apache.hadoop.fs.Path	getTransformedPath(String name, String subDir, String root)
static boolean	isDirEmpty(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path) Utility method that determines if a specified directory already has contents (non-hidden files) or not - useful to determine if an immutable table already has contents, for example.
static boolean	isS3a(org.apache.hadoop.fs.FileSystem fs)
static boolean	isSubdirectory(String parent, String other) Returns a BEST GUESS as to whether or not other is a subdirectory of parent.
static void	makeDir(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf)
static String	makePartName(List<String> partCols, List<String> vals)
static String	makePartName(List<String> partCols, List<String> vals, String defaultStr) Makes a valid partition name.
static org.apache.hadoop.fs.Path	makeQualified(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf) Variant of Path.makeQualified that qualifies the input path against the default file system indicated by the configuration This does not require a FileSystem handle in most cases - only requires the Filesystem URI.
static org.apache.hadoop.fs.Path	makeRelative(org.apache.hadoop.fs.Path parentPath, org.apache.hadoop.fs.Path childPath) Returns a relative path wrt the parent path.
static boolean	mkdir(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path f) Creates the directory and all necessary parent directories.
static boolean	moveToTrash(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path f, org.apache.hadoop.conf.Configuration conf, boolean purge) Move a particular file or directory to the trash.
static boolean	pathHasSnapshotSubDir(org.apache.hadoop.fs.Path p, org.apache.hadoop.fs.FileSystem fs) Check if the path contains a subdirectory named '.snapshot'
static boolean	rename(org.apache.hadoop.fs.FileSystem srcFs, org.apache.hadoop.fs.FileSystem destFs, org.apache.hadoop.fs.Path srcPath, org.apache.hadoop.fs.Path destPath) Rename a file.
static String	unescapePathName(String path)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

HIDDEN_FILES_PATH_FILTER

public static final org.apache.hadoop.fs.PathFilter HIDDEN_FILES_PATH_FILTER

Constructor Detail

FileUtils

public FileUtils()

Method Detail

moveToTrash

public static boolean moveToTrash(org.apache.hadoop.fs.FileSystem fs,
                                  org.apache.hadoop.fs.Path f,
                                  org.apache.hadoop.conf.Configuration conf,
                                  boolean purge)
                           throws IOException

Move a particular file or directory to the trash.

Parameters:

fs - FileSystem to use

f - path of file or directory to move to trash.

conf - configuration object

Returns:

true if move successful

Throws:

IOException

copy

public static boolean copy(org.apache.hadoop.fs.FileSystem srcFS,
                           org.apache.hadoop.fs.Path src,
                           org.apache.hadoop.fs.FileSystem dstFS,
                           org.apache.hadoop.fs.Path dst,
                           boolean deleteSource,
                           boolean overwrite,
                           org.apache.hadoop.conf.Configuration conf)
                    throws IOException

Copies files between filesystems.

Throws:

IOException

mkdir

public static boolean mkdir(org.apache.hadoop.fs.FileSystem fs,
                            org.apache.hadoop.fs.Path f)
                     throws IOException

Creates the directory and all necessary parent directories.

Parameters:

fs - FileSystem to use

f - path to create.

Returns:

true if directory created successfully. False otherwise, including if it exists.

Throws:

IOException - exception in creating the directory

rename

public static boolean rename(org.apache.hadoop.fs.FileSystem srcFs,
                             org.apache.hadoop.fs.FileSystem destFs,
                             org.apache.hadoop.fs.Path srcPath,
                             org.apache.hadoop.fs.Path destPath)
                      throws IOException

Rename a file. Unlike FileSystem.rename(Path, Path), if the destPath already exists and is a directory, this will NOT move the sourcePath into it. It will throw an IOException instead.

Parameters:

srcFs - file system src paths are on

destFs - file system dest paths are on

srcPath - source file or directory to move

destPath - destination file name. This must be a file and not an existing directory.

Returns:

result of fs.rename.

Throws:

IOException - if fs.rename throws it, or if destPath already exists.

escapePathName

public static String escapePathName(String path)

escapePathName

public static String escapePathName(String path,
                                    String defaultPath)

Escapes a path name.

Parameters:

path - The path to escape.

defaultPath - The default name for the path, if the given path is empty or null.

Returns:

An escaped path name.

unescapePathName

public static String unescapePathName(String path)

getFileStatusRecurse

public static List<org.apache.hadoop.fs.FileStatus> getFileStatusRecurse(org.apache.hadoop.fs.Path base,
                                                                         org.apache.hadoop.fs.FileSystem fs)

Get all file status from a root path and recursively go deep into certain levels.

Parameters:

base - the root path

fs - the file system

Returns:

array of FileStatus

makeRelative

public static org.apache.hadoop.fs.Path makeRelative(org.apache.hadoop.fs.Path parentPath,
                                                     org.apache.hadoop.fs.Path childPath)

Returns a relative path wrt the parent path.

Parameters:

parentPath - the parent path.

childPath - the child path.

Returns:

childPath relative to parent path.

isS3a

public static boolean isS3a(org.apache.hadoop.fs.FileSystem fs)

makePartName

public static String makePartName(List<String> partCols,
                                  List<String> vals)

makePartName

public static String makePartName(List<String> partCols,
                                  List<String> vals,
                                  String defaultStr)

Makes a valid partition name.

Parameters:

partCols - The partition keys' names

vals - The partition values

defaultStr - The default name given to a partition value if the respective value is empty or null.

Returns:

An escaped, valid partition name.

equalsFileSystem

public static boolean equalsFileSystem(org.apache.hadoop.fs.FileSystem fs1,
                                       org.apache.hadoop.fs.FileSystem fs2)

Determine if two objects reference the same file system.

Parameters:

fs1 - first file system

fs2 - second file system

Returns:

return true if both file system arguments point to same file system

pathHasSnapshotSubDir

public static boolean pathHasSnapshotSubDir(org.apache.hadoop.fs.Path p,
                                            org.apache.hadoop.fs.FileSystem fs)
                                     throws IOException

Check if the path contains a subdirectory named '.snapshot'

Parameters:

p - path to check

fs - filesystem of the path

Returns:

true if p contains a subdirectory named '.snapshot'

Throws:

IOException

makeDir

public static void makeDir(org.apache.hadoop.fs.Path path,
                           org.apache.hadoop.conf.Configuration conf)
                    throws MetaException

Throws:

MetaException

isDirEmpty

public static boolean isDirEmpty(org.apache.hadoop.fs.FileSystem fs,
                                 org.apache.hadoop.fs.Path path)
                          throws IOException

Utility method that determines if a specified directory already has contents (non-hidden files) or not - useful to determine if an immutable table already has contents, for example.

Parameters:

fs -

path -

Throws:

IOException

makeQualified

public static org.apache.hadoop.fs.Path makeQualified(org.apache.hadoop.fs.Path path,
                                                      org.apache.hadoop.conf.Configuration conf)
                                               throws IOException

Variant of Path.makeQualified that qualifies the input path against the default file system indicated by the configuration This does not require a FileSystem handle in most cases - only requires the Filesystem URI. This saves the cost of opening the Filesystem - which can involve RPCs - as well as cause errors

Parameters:

path - path to be fully qualified

conf - Configuration file

Returns:

path qualified relative to default file system

Throws:

IOException

isSubdirectory

public static boolean isSubdirectory(String parent,
                                     String other)

Returns a BEST GUESS as to whether or not other is a subdirectory of parent. It does not take into account any intricacies of the underlying file system, which is assumed to be HDFS. This should not return any false positives, but may return false negatives.

Parameters:

parent -

other - Directory to check if it is a subdirectory of parent

Returns:

True, if other is subdirectory of parent

getTransformedPath

public static org.apache.hadoop.fs.Path getTransformedPath(String name,
                                                           String subDir,
                                                           String root)