org.apache.hudi.hadoop.fs

Class HadoopFSUtils

java.lang.Object

org.apache.hudi.hadoop.fs.HadoopFSUtils

public class HadoopFSUtils
extends Object

Utility functions related to accessing the file storage on Hadoop.

Constructor Summary

Constructors

Constructor and Description
HadoopFSUtils()

Method Summary

Modifier and Type	Method and Description
static org.apache.hadoop.fs.Path	addSchemeIfLocalPath(String path)
static org.apache.hadoop.fs.Path	constructAbsolutePathInHadoopPath(String basePath, String relativePartitionPath)
static org.apache.hadoop.fs.FileStatus	convertToHadoopFileStatus(StoragePathInfo pathInfo)
static org.apache.hadoop.fs.Path	convertToHadoopPath(StoragePath path)
static StoragePath	convertToStoragePath(org.apache.hadoop.fs.Path path)
static StoragePathInfo	convertToStoragePathInfo(org.apache.hadoop.fs.FileStatus fileStatus)
static Map<String,Boolean>	deleteFilesParallelize(HoodieTableMetaClient metaClient, List<String> paths, HoodieEngineContext context, int parallelism, boolean ignoreFailed)
static HoodieFileStatus	fromFileStatus(org.apache.hadoop.fs.FileStatus fileStatus)
static HoodieFSPermission	fromFSPermission(org.apache.hadoop.fs.permission.FsPermission fsPermission)
static HoodiePath	fromPath(org.apache.hadoop.fs.Path path)
static org.apache.hadoop.fs.FileStatus[]	getAllDataFilesInPartition(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path partitionPath) Get the names of all the base and log files in the given partition path.
static String	getDFSFullPartitionPath(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path fullPartitionPath) Get DFS full partition path (e.g.
static String	getFileIdFromFilePath(org.apache.hadoop.fs.Path filePath) Check if the file is a base file of a log file.
static String	getFileIdFromLogPath(org.apache.hadoop.fs.Path path) Get the first part of the file name in the log file.
static long	getFileSize(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path)
static List<org.apache.hadoop.fs.FileStatus>	getFileStatusAtLevel(HoodieEngineContext hoodieEngineContext, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path rootPath, int expectLevel, int parallelism) Lists file status at a certain level in the directory hierarchy.
static org.apache.hadoop.fs.FileSystem	getFs(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf)
static <T> org.apache.hadoop.fs.FileSystem	getFs(org.apache.hadoop.fs.Path path, StorageConfiguration<T> storageConf)
static <T> org.apache.hadoop.fs.FileSystem	getFs(org.apache.hadoop.fs.Path path, StorageConfiguration<T> storageConf, boolean newCopy)
static org.apache.hadoop.fs.FileSystem	getFs(StoragePath path, org.apache.hadoop.conf.Configuration conf)
static org.apache.hadoop.fs.FileSystem	getFs(String pathStr, org.apache.hadoop.conf.Configuration conf)
static org.apache.hadoop.fs.FileSystem	getFs(String pathStr, org.apache.hadoop.conf.Configuration conf, boolean localByDefault)
static <T> org.apache.hadoop.fs.FileSystem	getFs(String pathStr, StorageConfiguration<T> storageConf)
static org.apache.hadoop.fs.FSDataInputStream	getFSDataInputStream(org.apache.hadoop.fs.FileSystem fs, StoragePath filePath, int bufferSize, boolean wrapStream) Fetch the right FSDataInputStream to be used by wrapping with required input streams.
static String	getRelativePartitionPath(org.apache.hadoop.fs.Path basePath, org.apache.hadoop.fs.Path fullPartitionPath) Given a base partition and a partition path, return relative path of partition path to the base path.
static StorageConfiguration<org.apache.hadoop.conf.Configuration>	getStorageConf(org.apache.hadoop.conf.Configuration conf)
static StorageConfiguration<org.apache.hadoop.conf.Configuration>	getStorageConfWithCopy(org.apache.hadoop.conf.Configuration conf)
static boolean	isBaseFile(org.apache.hadoop.fs.Path path)
static boolean	isCHDFileSystem(org.apache.hadoop.fs.FileSystem fs) Chdfs will throw IOException instead of EOFException.
static boolean	isDataFile(org.apache.hadoop.fs.Path path) Returns true if the given path is a Base file or a Log file.
static boolean	isGCSFileSystem(org.apache.hadoop.fs.FileSystem fs) This is due to HUDI-140 GCS has a different behavior for detecting EOF during seek().
static boolean	isLogFile(org.apache.hadoop.fs.Path logPath)
static <T> Map<String,T>	parallelizeFilesProcess(HoodieEngineContext hoodieEngineContext, org.apache.hadoop.fs.FileSystem fs, int parallelism, FSUtils.SerializableFunction<Pair<String,StorageConfiguration<org.apache.hadoop.conf.Configuration>>,T> pairFunction, List<String> subPaths)
static org.apache.hadoop.conf.Configuration	prepareHadoopConf(org.apache.hadoop.conf.Configuration conf)
static boolean	recoverDFSFileLease(org.apache.hadoop.hdfs.DistributedFileSystem dfs, org.apache.hadoop.fs.Path p) When a file was opened and the task died without closing the stream, another task executor cannot open because the existing lease will be active.
static org.apache.hadoop.conf.Configuration	registerFileSystem(StoragePath file, org.apache.hadoop.conf.Configuration conf)
static org.apache.hadoop.fs.permission.FsPermission	toFSPermission(HoodieFSPermission fsPermission)
static org.apache.hadoop.fs.Path	toPath(HoodiePath path)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HadoopFSUtils

public HadoopFSUtils()

Method Detail

prepareHadoopConf

public static org.apache.hadoop.conf.Configuration prepareHadoopConf(org.apache.hadoop.conf.Configuration conf)

getStorageConf

public static StorageConfiguration<org.apache.hadoop.conf.Configuration> getStorageConf(org.apache.hadoop.conf.Configuration conf)

getStorageConfWithCopy

public static StorageConfiguration<org.apache.hadoop.conf.Configuration> getStorageConfWithCopy(org.apache.hadoop.conf.Configuration conf)

getFs

public static <T> org.apache.hadoop.fs.FileSystem getFs(String pathStr,
                                                        StorageConfiguration<T> storageConf)

getFs

public static <T> org.apache.hadoop.fs.FileSystem getFs(org.apache.hadoop.fs.Path path,
                                                        StorageConfiguration<T> storageConf)

getFs

public static <T> org.apache.hadoop.fs.FileSystem getFs(org.apache.hadoop.fs.Path path,
                                                        StorageConfiguration<T> storageConf,
                                                        boolean newCopy)

getFs

public static org.apache.hadoop.fs.FileSystem getFs(String pathStr,
                                                    org.apache.hadoop.conf.Configuration conf)

getFs

public static org.apache.hadoop.fs.FileSystem getFs(StoragePath path,
                                                    org.apache.hadoop.conf.Configuration conf)

getFs

public static org.apache.hadoop.fs.FileSystem getFs(org.apache.hadoop.fs.Path path,
                                                    org.apache.hadoop.conf.Configuration conf)

getFs

public static org.apache.hadoop.fs.FileSystem getFs(String pathStr,
                                                    org.apache.hadoop.conf.Configuration conf,
                                                    boolean localByDefault)

addSchemeIfLocalPath

public static org.apache.hadoop.fs.Path addSchemeIfLocalPath(String path)

convertToHadoopPath

public static org.apache.hadoop.fs.Path convertToHadoopPath(StoragePath path)

Parameters:

path - StoragePath instance.

Returns:

the Hadoop Path instance after conversion.

convertToStoragePath

public static StoragePath convertToStoragePath(org.apache.hadoop.fs.Path path)

Parameters:

path - Hadoop Path instance.

Returns:

the StoragePath instance after conversion.

convertToStoragePathInfo

public static StoragePathInfo convertToStoragePathInfo(org.apache.hadoop.fs.FileStatus fileStatus)

Parameters:

fileStatus - Hadoop FileStatus instance.

Returns:

the StoragePathInfo instance after conversion.

convertToHadoopFileStatus

public static org.apache.hadoop.fs.FileStatus convertToHadoopFileStatus(StoragePathInfo pathInfo)

Parameters:

pathInfo - StoragePathInfo instance.

Returns:

the FileStatus instance after conversion.

getFSDataInputStream

public static org.apache.hadoop.fs.FSDataInputStream getFSDataInputStream(org.apache.hadoop.fs.FileSystem fs,
                                                                          StoragePath filePath,
                                                                          int bufferSize,
                                                                          boolean wrapStream)

Fetch the right FSDataInputStream to be used by wrapping with required input streams.

Parameters:

fs - instance of FileSystem in use.

filePath - path of the file.

bufferSize - buffer size to be used.

wrapStream - if false, don't attempt to wrap the stream

Returns:

the right FSDataInputStream as required.

isGCSFileSystem

public static boolean isGCSFileSystem(org.apache.hadoop.fs.FileSystem fs)

This is due to HUDI-140 GCS has a different behavior for detecting EOF during seek().

Parameters:

fs - fileSystem instance.

Returns:

true if the inputstream or the wrapped one is of type GoogleHadoopFSInputStream

isCHDFileSystem

public static boolean isCHDFileSystem(org.apache.hadoop.fs.FileSystem fs)

Chdfs will throw IOException instead of EOFException. It will cause error in isBlockCorrupted(). Wrapped by BoundedFsDataInputStream, to check whether the desired offset is out of the file size in advance.

registerFileSystem

public static org.apache.hadoop.conf.Configuration registerFileSystem(StoragePath file,
                                                                      org.apache.hadoop.conf.Configuration conf)

toPath

public static org.apache.hadoop.fs.Path toPath(HoodiePath path)

fromPath

public static HoodiePath fromPath(org.apache.hadoop.fs.Path path)

toFSPermission

public static org.apache.hadoop.fs.permission.FsPermission toFSPermission(HoodieFSPermission fsPermission)

fromFSPermission

public static HoodieFSPermission fromFSPermission(org.apache.hadoop.fs.permission.FsPermission fsPermission)

fromFileStatus

public static HoodieFileStatus fromFileStatus(org.apache.hadoop.fs.FileStatus fileStatus)

getFileSize

public static long getFileSize(org.apache.hadoop.fs.FileSystem fs,
                               org.apache.hadoop.fs.Path path)
                        throws IOException

Throws:

IOException

getRelativePartitionPath

public static String getRelativePartitionPath(org.apache.hadoop.fs.Path basePath,
                                              org.apache.hadoop.fs.Path fullPartitionPath)

Given a base partition and a partition path, return relative path of partition path to the base path.

getFileIdFromLogPath

public static String getFileIdFromLogPath(org.apache.hadoop.fs.Path path)

Get the first part of the file name in the log file. That will be the fileId. Log file do not have instantTime in the file name.

getFileIdFromFilePath

public static String getFileIdFromFilePath(org.apache.hadoop.fs.Path filePath)

Check if the file is a base file of a log file. Then get the fileId appropriately.

isBaseFile

public static boolean isBaseFile(org.apache.hadoop.fs.Path path)

isLogFile

public static boolean isLogFile(org.apache.hadoop.fs.Path logPath)

isDataFile

public static boolean isDataFile(org.apache.hadoop.fs.Path path)

Returns true if the given path is a Base file or a Log file.

getAllDataFilesInPartition

public static org.apache.hadoop.fs.FileStatus[] getAllDataFilesInPartition(org.apache.hadoop.fs.FileSystem fs,
                                                                           org.apache.hadoop.fs.Path partitionPath)
                                                                    throws IOException

Get the names of all the base and log files in the given partition path.

Throws:

IOException

constructAbsolutePathInHadoopPath

public static org.apache.hadoop.fs.Path constructAbsolutePathInHadoopPath(String basePath,
                                                                          String relativePartitionPath)

getDFSFullPartitionPath

public static String getDFSFullPartitionPath(org.apache.hadoop.fs.FileSystem fs,
                                             org.apache.hadoop.fs.Path fullPartitionPath)

Get DFS full partition path (e.g. hdfs://ip-address:8020:/)

parallelizeFilesProcess

public static <T> Map<String,T> parallelizeFilesProcess(HoodieEngineContext hoodieEngineContext,
                                                        org.apache.hadoop.fs.FileSystem fs,
                                                        int parallelism,
                                                        FSUtils.SerializableFunction<Pair<String,StorageConfiguration<org.apache.hadoop.conf.Configuration>>,T> pairFunction,
                                                        List<String> subPaths)

getFileStatusAtLevel

public static List<org.apache.hadoop.fs.FileStatus> getFileStatusAtLevel(HoodieEngineContext hoodieEngineContext,
                                                                         org.apache.hadoop.fs.FileSystem fs,
                                                                         org.apache.hadoop.fs.Path rootPath,
                                                                         int expectLevel,
                                                                         int parallelism)

Lists file status at a certain level in the directory hierarchy.

E.g., given "/tmp/hoodie_table" as the rootPath, and 3 as the expected level, this method gives back the FileStatus of all files under "/tmp/hoodie_table/[*]/[*]/[*]/" folders.

Parameters:

hoodieEngineContext - HoodieEngineContext instance.

fs - FileSystem instance.

rootPath - Root path for the file listing.

expectLevel - Expected level of directory hierarchy for files to be added.

parallelism - Parallelism for the file listing.

Returns:

A list of file status of files at the level.

deleteFilesParallelize

public static Map<String,Boolean> deleteFilesParallelize(HoodieTableMetaClient metaClient,
                                                         List<String> paths,
                                                         HoodieEngineContext context,
                                                         int parallelism,
                                                         boolean ignoreFailed)

recoverDFSFileLease

public static boolean recoverDFSFileLease(org.apache.hadoop.hdfs.DistributedFileSystem dfs,
                                          org.apache.hadoop.fs.Path p)
                                   throws IOException,
                                          InterruptedException

When a file was opened and the task died without closing the stream, another task executor cannot open because the existing lease will be active. We will try to recover the lease, from HDFS. If a data node went down, it takes about 10 minutes for the lease to be recovered. But if the client dies, this should be instant.

Throws:

IOException

InterruptedException