org.apache.hudi.common.fs

Class FSUtils

java.lang.Object

org.apache.hudi.common.fs.FSUtils

public class FSUtils
extends Object

Utility functions related to accessing the file storage.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	FSUtils.SerializableFunction<T,R>

Field Summary

Fields

Modifier and Type	Field and Description
static Pattern	LOG_FILE_PATTERN

Constructor Summary

Constructors

Constructor and Description
FSUtils()

Method Summary

Modifier and Type	Method and Description
static org.apache.hadoop.fs.Path	addSchemeIfLocalPath(String path)
static int	computeNextLogVersion(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path partitionPath, String fileId, String logFileExtension, String baseCommitTime) computes the next log version for the specified fileId in the partition path.
static String	createNewFileId(String idPfx, int id)
static String	createNewFileIdPfx() Returns a new unique prefix for creating a file group.
static void	createPathIfNotExists(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path partitionPath)
static boolean	deleteDir(HoodieEngineContext hoodieEngineContext, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path dirPath, int parallelism) Deletes a directory by deleting sub-paths in parallel on the file system.
static boolean	deleteSubPath(String subPathStr, SerializableConfiguration conf, boolean recursive) Deletes a sub-path.
static org.apache.hadoop.fs.FileStatus[]	getAllDataFilesInPartition(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path partitionPath) Get the names of all the base and log files in the given partition path.
static List<String>	getAllFoldersWithPartitionMetaFile(org.apache.hadoop.fs.FileSystem fs, String basePathStr) Obtain all the partition paths, that are present in this table, denoted by presence of HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE_PREFIX.
static Stream<HoodieLogFile>	getAllLogFiles(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path partitionPath, String fileId, String logFileExtension, String baseCommitTime) Get all the log files for the passed in file-id in the partition path.
static List<String>	getAllPartitionFoldersThreeLevelsDown(org.apache.hadoop.fs.FileSystem fs, String basePath) Gets all partition paths assuming date partitioning (year, month, day) three levels down.
static List<String>	getAllPartitionPaths(HoodieEngineContext engineContext, HoodieMetadataConfig metadataConfig, String basePathStr)
static List<String>	getAllPartitionPaths(HoodieEngineContext engineContext, String basePathStr, boolean useFileListingFromMetadata, boolean assumeDatePartitioning)
static String	getBaseCommitTimeFromLogPath(org.apache.hadoop.fs.Path path) Get the first part of the file name in the log file.
static String	getCommitFromCommitFile(String commitFileName)
static String	getCommitTime(String fullFileName)
static int	getDefaultBufferSize(org.apache.hadoop.fs.FileSystem fs)
static Short	getDefaultReplication(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path)
static String	getDFSFullPartitionPath(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path fullPartitionPath) Get DFS full partition path (e.g.
static String	getFileExtension(String fullName)
static String	getFileExtensionFromLog(org.apache.hadoop.fs.Path logPath) Get the file extension from the log file.
static String	getFileId(String fullFileName)
static String	getFileIdFromFilePath(org.apache.hadoop.fs.Path filePath) Check if the file is a base file of a log file.
static String	getFileIdFromLogPath(org.apache.hadoop.fs.Path path) Get the first part of the file name in the log file.
static String	getFileName(String filePathWithPartition, String partition) Extracts the file name from the relative path based on the table base path.
static Map<String,org.apache.hadoop.fs.FileStatus[]>	getFilesInPartitions(HoodieEngineContext engineContext, HoodieMetadataConfig metadataConfig, String basePathStr, String[] partitionPaths, String spillableMapPath)
static long	getFileSize(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path)
static List<org.apache.hadoop.fs.FileStatus>	getFileStatusAtLevel(HoodieEngineContext hoodieEngineContext, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path rootPath, int expectLevel, int parallelism) Lists file status at a certain level in the directory hierarchy.
static int	getFileVersionFromLog(org.apache.hadoop.fs.Path logPath) Get the last part of the file name in the log file and convert to int.
static org.apache.hadoop.fs.FileSystem	getFs(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf)
static org.apache.hadoop.fs.FileSystem	getFs(String pathStr, org.apache.hadoop.conf.Configuration conf)
static org.apache.hadoop.fs.FileSystem	getFs(String pathStr, org.apache.hadoop.conf.Configuration conf, boolean localByDefault)
static HoodieWrapperFileSystem	getFs(String path, SerializableConfiguration hadoopConf, ConsistencyGuardConfig consistencyGuardConfig) Get the FS implementation for this table.
static List<org.apache.hadoop.fs.FileStatus>	getGlobStatusExcludingMetaFolder(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path globPath) Helper to filter out paths under metadata folder when running fs.globStatus.
static Option<HoodieLogFile>	getLatestLogFile(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path partitionPath, String fileId, String logFileExtension, String baseCommitTime) Get the latest log file for the passed in file-id in the partition path
static Option<Pair<Integer,String>>	getLatestLogVersion(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path partitionPath, String fileId, String logFileExtension, String baseCommitTime) Get the latest log version for the fileId in the partition path.
static org.apache.hadoop.fs.Path	getPartitionPath(org.apache.hadoop.fs.Path basePath, String partitionPath)
static org.apache.hadoop.fs.Path	getPartitionPath(String basePath, String partitionPath)
static String	getRelativePartitionPath(org.apache.hadoop.fs.Path basePath, org.apache.hadoop.fs.Path fullPartitionPath) Given a base partition and a partition path, return relative path of partition path to the base path.
static Long	getSizeInMB(long sizeInBytes)
static Integer	getStageIdFromLogPath(org.apache.hadoop.fs.Path path) Get StageId used in log-path.
static Integer	getTaskAttemptIdFromLogPath(org.apache.hadoop.fs.Path path) Get Task Attempt Id used in log-path.
static Integer	getTaskPartitionIdFromLogPath(org.apache.hadoop.fs.Path path) Get TaskPartitionId used in log-path.
static String	getWriteTokenFromLogPath(org.apache.hadoop.fs.Path path) Get Write-Token used in log-path.
static boolean	isBaseFile(org.apache.hadoop.fs.Path path)
static boolean	isCHDFileSystem(org.apache.hadoop.fs.FileSystem fs) Chdfs will throw IOException instead of EOFException.
static boolean	isDataFile(org.apache.hadoop.fs.Path path) Returns true if the given path is a Base file or a Log file.
static boolean	isGCSFileSystem(org.apache.hadoop.fs.FileSystem fs) This is due to HUDI-140 GCS has a different behavior for detecting EOF during seek().
static boolean	isLogFile(org.apache.hadoop.fs.Path logPath)
static boolean	isLogFile(String fileName)
static boolean	isTableExists(String path, org.apache.hadoop.fs.FileSystem fs) Check if table already exists in the given path.
static String	makeBaseFileName(String instantTime, String writeToken, String fileId)
static String	makeBaseFileName(String instantTime, String writeToken, String fileId, String fileExtension)
static String	makeBootstrapIndexFileName(String instantTime, String fileId, String ext)
static String	makeLogFileName(String fileId, String logFileExtension, String baseCommitTime, int version, String writeToken)
static org.apache.hadoop.fs.Path	makeQualified(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path) Makes path qualified w/ FileSystem's URI
static String	makeWriteToken(int taskPartitionId, int stageId, long taskAttemptId) A write token uniquely identifies an attempt at one of the IOHandle operations (Merge/Create/Append).
static String	maskWithoutFileId(String instantTime, int taskPartitionId)
static <T> Map<String,T>	parallelizeFilesProcess(HoodieEngineContext hoodieEngineContext, org.apache.hadoop.fs.FileSystem fs, int parallelism, FSUtils.SerializableFunction<Pair<String,SerializableConfiguration>,T> pairFunction, List<String> subPaths)
static <T> Map<String,T>	parallelizeSubPathProcess(HoodieEngineContext hoodieEngineContext, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path dirPath, int parallelism, Predicate<org.apache.hadoop.fs.FileStatus> subPathPredicate, FSUtils.SerializableFunction<Pair<String,SerializableConfiguration>,T> pairFunction) Processes sub-path in parallel.
static org.apache.hadoop.conf.Configuration	prepareHadoopConf(org.apache.hadoop.conf.Configuration conf)
static void	processFiles(org.apache.hadoop.fs.FileSystem fs, String basePathStr, Function<org.apache.hadoop.fs.FileStatus,Boolean> consumer, boolean excludeMetaFolder) Recursively processes all files in the base-path.
static boolean	recoverDFSFileLease(org.apache.hadoop.hdfs.DistributedFileSystem dfs, org.apache.hadoop.fs.Path p) When a file was opened and the task died without closing the stream, another task executor cannot open because the existing lease will be active.
static org.apache.hadoop.conf.Configuration	registerFileSystem(org.apache.hadoop.fs.Path file, org.apache.hadoop.conf.Configuration conf)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG_FILE_PATTERN

public static final Pattern LOG_FILE_PATTERN

Constructor Detail

FSUtils

public FSUtils()

Method Detail

prepareHadoopConf

public static org.apache.hadoop.conf.Configuration prepareHadoopConf(org.apache.hadoop.conf.Configuration conf)

getFs

public static org.apache.hadoop.fs.FileSystem getFs(String pathStr,
                                                    org.apache.hadoop.conf.Configuration conf)

getFs

public static org.apache.hadoop.fs.FileSystem getFs(org.apache.hadoop.fs.Path path,
                                                    org.apache.hadoop.conf.Configuration conf)

getFs

public static org.apache.hadoop.fs.FileSystem getFs(String pathStr,
                                                    org.apache.hadoop.conf.Configuration conf,
                                                    boolean localByDefault)

isTableExists

public static boolean isTableExists(String path,
                                    org.apache.hadoop.fs.FileSystem fs)
                             throws IOException

Check if table already exists in the given path.

Parameters:

path - base path of the table.

fs - instance of FileSystem.

Returns:

true if table exists. false otherwise.

Throws:

IOException

addSchemeIfLocalPath

public static org.apache.hadoop.fs.Path addSchemeIfLocalPath(String path)

makeQualified

public static org.apache.hadoop.fs.Path makeQualified(org.apache.hadoop.fs.FileSystem fs,
                                                      org.apache.hadoop.fs.Path path)

Makes path qualified w/ FileSystem's URI

Parameters:

fs - instance of FileSystem path belongs to

path - path to be qualified

Returns:

qualified path, prefixed w/ the URI of the target FS object provided

makeWriteToken

public static String makeWriteToken(int taskPartitionId,
                                    int stageId,
                                    long taskAttemptId)

A write token uniquely identifies an attempt at one of the IOHandle operations (Merge/Create/Append).

makeBaseFileName

public static String makeBaseFileName(String instantTime,
                                      String writeToken,
                                      String fileId)

makeBaseFileName

public static String makeBaseFileName(String instantTime,
                                      String writeToken,
                                      String fileId,
                                      String fileExtension)

makeBootstrapIndexFileName

public static String makeBootstrapIndexFileName(String instantTime,
                                                String fileId,
                                                String ext)

maskWithoutFileId

public static String maskWithoutFileId(String instantTime,
                                       int taskPartitionId)

getCommitFromCommitFile

public static String getCommitFromCommitFile(String commitFileName)

getCommitTime

public static String getCommitTime(String fullFileName)

getFileSize

public static long getFileSize(org.apache.hadoop.fs.FileSystem fs,
                               org.apache.hadoop.fs.Path path)
                        throws IOException

Throws:

IOException

getFileId

public static String getFileId(String fullFileName)

getAllPartitionFoldersThreeLevelsDown

public static List<String> getAllPartitionFoldersThreeLevelsDown(org.apache.hadoop.fs.FileSystem fs,
                                                                 String basePath)
                                                          throws IOException

Gets all partition paths assuming date partitioning (year, month, day) three levels down.

Throws:

IOException

getRelativePartitionPath

public static String getRelativePartitionPath(org.apache.hadoop.fs.Path basePath,
                                              org.apache.hadoop.fs.Path fullPartitionPath)

Given a base partition and a partition path, return relative path of partition path to the base path.

getAllFoldersWithPartitionMetaFile

public static List<String> getAllFoldersWithPartitionMetaFile(org.apache.hadoop.fs.FileSystem fs,
                                                              String basePathStr)
                                                       throws IOException

Obtain all the partition paths, that are present in this table, denoted by presence of HoodiePartitionMetadata.HOODIE_PARTITION_METAFILE_PREFIX. If the basePathStr is a subdirectory of .hoodie folder then we assume that the partitions of an internal table (a hoodie table within the .hoodie directory) are to be obtained.

Parameters:

fs - FileSystem instance

basePathStr - base directory

Throws:

IOException

processFiles

public static void processFiles(org.apache.hadoop.fs.FileSystem fs,
                                String basePathStr,
                                Function<org.apache.hadoop.fs.FileStatus,Boolean> consumer,
                                boolean excludeMetaFolder)
                         throws IOException

Recursively processes all files in the base-path. If excludeMetaFolder is set, the meta-folder and all its subdirs are skipped

Parameters:

fs - File System

basePathStr - Base-Path

consumer - Callback for processing

excludeMetaFolder - Exclude .hoodie folder

Throws:

IOException - -

getAllPartitionPaths

public static List<String> getAllPartitionPaths(HoodieEngineContext engineContext,
                                                String basePathStr,
                                                boolean useFileListingFromMetadata,
                                                boolean assumeDatePartitioning)

getAllPartitionPaths

public static List<String> getAllPartitionPaths(HoodieEngineContext engineContext,
                                                HoodieMetadataConfig metadataConfig,
                                                String basePathStr)

getFilesInPartitions

public static Map<String,org.apache.hadoop.fs.FileStatus[]> getFilesInPartitions(HoodieEngineContext engineContext,
                                                                                 HoodieMetadataConfig metadataConfig,
                                                                                 String basePathStr,
                                                                                 String[] partitionPaths,
                                                                                 String spillableMapPath)

getFileExtension

public static String getFileExtension(String fullName)

createNewFileIdPfx

public static String createNewFileIdPfx()

Returns a new unique prefix for creating a file group.

createNewFileId

public static String createNewFileId(String idPfx,
                                     int id)

getFileExtensionFromLog

public static String getFileExtensionFromLog(org.apache.hadoop.fs.Path logPath)

Get the file extension from the log file.

getFileIdFromLogPath

public static String getFileIdFromLogPath(org.apache.hadoop.fs.Path path)

Get the first part of the file name in the log file. That will be the fileId. Log file do not have instantTime in the file name.

getFileIdFromFilePath

public static String getFileIdFromFilePath(org.apache.hadoop.fs.Path filePath)

Check if the file is a base file of a log file. Then get the fileId appropriately.

getBaseCommitTimeFromLogPath

public static String getBaseCommitTimeFromLogPath(org.apache.hadoop.fs.Path path)

Get the first part of the file name in the log file. That will be the fileId. Log file do not have instantTime in the file name.

getTaskPartitionIdFromLogPath

public static Integer getTaskPartitionIdFromLogPath(org.apache.hadoop.fs.Path path)

Get TaskPartitionId used in log-path.

getWriteTokenFromLogPath

public static String getWriteTokenFromLogPath(org.apache.hadoop.fs.Path path)

Get Write-Token used in log-path.

getStageIdFromLogPath

public static Integer getStageIdFromLogPath(org.apache.hadoop.fs.Path path)

Get StageId used in log-path.

getTaskAttemptIdFromLogPath

public static Integer getTaskAttemptIdFromLogPath(org.apache.hadoop.fs.Path path)

Get Task Attempt Id used in log-path.

getFileVersionFromLog

public static int getFileVersionFromLog(org.apache.hadoop.fs.Path logPath)

Get the last part of the file name in the log file and convert to int.

makeLogFileName

public static String makeLogFileName(String fileId,
                                     String logFileExtension,
                                     String baseCommitTime,
                                     int version,
                                     String writeToken)

isBaseFile

public static boolean isBaseFile(org.apache.hadoop.fs.Path path)

isLogFile

public static boolean isLogFile(org.apache.hadoop.fs.Path logPath)

isLogFile

public static boolean isLogFile(String fileName)

isDataFile

public static boolean isDataFile(org.apache.hadoop.fs.Path path)

Returns true if the given path is a Base file or a Log file.

getAllDataFilesInPartition

public static org.apache.hadoop.fs.FileStatus[] getAllDataFilesInPartition(org.apache.hadoop.fs.FileSystem fs,
                                                                           org.apache.hadoop.fs.Path partitionPath)
                                                                    throws IOException

Get the names of all the base and log files in the given partition path.

Throws:

IOException

getLatestLogFile

public static Option<HoodieLogFile> getLatestLogFile(org.apache.hadoop.fs.FileSystem fs,
                                                     org.apache.hadoop.fs.Path partitionPath,
                                                     String fileId,
                                                     String logFileExtension,
                                                     String baseCommitTime)
                                              throws IOException

Get the latest log file for the passed in file-id in the partition path

Throws:

IOException

getAllLogFiles

public static Stream<HoodieLogFile> getAllLogFiles(org.apache.hadoop.fs.FileSystem fs,
                                                   org.apache.hadoop.fs.Path partitionPath,
                                                   String fileId,
                                                   String logFileExtension,
                                                   String baseCommitTime)
                                            throws IOException

Get all the log files for the passed in file-id in the partition path.

Throws:

IOException

getLatestLogVersion

public static Option<Pair<Integer,String>> getLatestLogVersion(org.apache.hadoop.fs.FileSystem fs,
                                                               org.apache.hadoop.fs.Path partitionPath,
                                                               String fileId,
                                                               String logFileExtension,
                                                               String baseCommitTime)
                                                        throws IOException

Get the latest log version for the fileId in the partition path.

Throws:

IOException

computeNextLogVersion

public static int computeNextLogVersion(org.apache.hadoop.fs.FileSystem fs,
                                        org.apache.hadoop.fs.Path partitionPath,
                                        String fileId,
                                        String logFileExtension,
                                        String baseCommitTime)
                                 throws IOException

computes the next log version for the specified fileId in the partition path.

Throws:

IOException

getDefaultBufferSize

public static int getDefaultBufferSize(org.apache.hadoop.fs.FileSystem fs)

getDefaultReplication

public static Short getDefaultReplication(org.apache.hadoop.fs.FileSystem fs,
                                          org.apache.hadoop.fs.Path path)

recoverDFSFileLease

public static boolean recoverDFSFileLease(org.apache.hadoop.hdfs.DistributedFileSystem dfs,
                                          org.apache.hadoop.fs.Path p)
                                   throws IOException,
                                          InterruptedException

When a file was opened and the task died without closing the stream, another task executor cannot open because the existing lease will be active. We will try to recover the lease, from HDFS. If a data node went down, it takes about 10 minutes for the lease to be rocovered. But if the client dies, this should be instant.

Throws:

IOException

InterruptedException

createPathIfNotExists

public static void createPathIfNotExists(org.apache.hadoop.fs.FileSystem fs,
                                         org.apache.hadoop.fs.Path partitionPath)
                                  throws IOException

Throws:

IOException

getSizeInMB

public static Long getSizeInMB(long sizeInBytes)

getPartitionPath

public static org.apache.hadoop.fs.Path getPartitionPath(String basePath,
                                                         String partitionPath)

getPartitionPath

public static org.apache.hadoop.fs.Path getPartitionPath(org.apache.hadoop.fs.Path basePath,
                                                         String partitionPath)

getFileName

public static String getFileName(String filePathWithPartition,
                                 String partition)

Extracts the file name from the relative path based on the table base path. For example: "/2022/07/29/file1.parquet", "/2022/07/29" -> "file1.parquet" "2022/07/29/file2.parquet", "2022/07/29" -> "file2.parquet" "/file3.parquet", "" -> "file3.parquet" "file4.parquet", "" -> "file4.parquet"

Parameters:

filePathWithPartition - the relative file path based on the table base path.

partition - the relative partition path. For partitioned table, `partition` contains the relative partition path; for non-partitioned table, `partition` is empty

Returns:

Extracted file name in String.

getDFSFullPartitionPath

public static String getDFSFullPartitionPath(org.apache.hadoop.fs.FileSystem fs,
                                             org.apache.hadoop.fs.Path fullPartitionPath)

Get DFS full partition path (e.g. hdfs://ip-address:8020:/)

isGCSFileSystem

public static boolean isGCSFileSystem(org.apache.hadoop.fs.FileSystem fs)

This is due to HUDI-140 GCS has a different behavior for detecting EOF during seek().

Parameters:

fs - fileSystem instance.

Returns:

true if the inputstream or the wrapped one is of type GoogleHadoopFSInputStream

isCHDFileSystem

public static boolean isCHDFileSystem(org.apache.hadoop.fs.FileSystem fs)

Chdfs will throw IOException instead of EOFException. It will cause error in isBlockCorrupted(). Wrapped by BoundedFsDataInputStream, to check whether the desired offset is out of the file size in advance.

registerFileSystem

public static org.apache.hadoop.conf.Configuration registerFileSystem(org.apache.hadoop.fs.Path file,
                                                                      org.apache.hadoop.conf.Configuration conf)

getFs

public static HoodieWrapperFileSystem getFs(String path,
                                            SerializableConfiguration hadoopConf,
                                            ConsistencyGuardConfig consistencyGuardConfig)

Get the FS implementation for this table.

Parameters:

path - Path String

hadoopConf - Serializable Hadoop Configuration

consistencyGuardConfig - Consistency Guard Config

Returns:

HoodieWrapperFileSystem

getGlobStatusExcludingMetaFolder

public static List<org.apache.hadoop.fs.FileStatus> getGlobStatusExcludingMetaFolder(org.apache.hadoop.fs.FileSystem fs,
                                                                                     org.apache.hadoop.fs.Path globPath)
                                                                              throws IOException

Helper to filter out paths under metadata folder when running fs.globStatus.

Parameters:

fs - File System

globPath - Glob Path

Returns:

the file status list of globPath exclude the meta folder

Throws:

IOException - when having trouble listing the path

deleteDir

public static boolean deleteDir(HoodieEngineContext hoodieEngineContext,
                                org.apache.hadoop.fs.FileSystem fs,
                                org.apache.hadoop.fs.Path dirPath,
                                int parallelism)

Deletes a directory by deleting sub-paths in parallel on the file system.

Parameters:

hoodieEngineContext - HoodieEngineContext instance

fs - file system

dirPath - directory path

parallelism - parallelism to use for sub-paths

Returns:

true if the directory is delete; false otherwise.

parallelizeSubPathProcess

public static <T> Map<String,T> parallelizeSubPathProcess(HoodieEngineContext hoodieEngineContext,
                                                          org.apache.hadoop.fs.FileSystem fs,
                                                          org.apache.hadoop.fs.Path dirPath,
                                                          int parallelism,
                                                          Predicate<org.apache.hadoop.fs.FileStatus> subPathPredicate,
                                                          FSUtils.SerializableFunction<Pair<String,SerializableConfiguration>,T> pairFunction)

Processes sub-path in parallel.

Type Parameters:

T - type of result to return for each sub-path

Parameters:

hoodieEngineContext - HoodieEngineContext instance

fs - file system

dirPath - directory path

parallelism - parallelism to use for sub-paths

subPathPredicate - predicate to use to filter sub-paths for processing

pairFunction - actual processing logic for each sub-path

Returns:

a map of sub-path to result of the processing

parallelizeFilesProcess

public static <T> Map<String,T> parallelizeFilesProcess(HoodieEngineContext hoodieEngineContext,
                                                        org.apache.hadoop.fs.FileSystem fs,
                                                        int parallelism,
                                                        FSUtils.SerializableFunction<Pair<String,SerializableConfiguration>,T> pairFunction,
                                                        List<String> subPaths)

deleteSubPath

public static boolean deleteSubPath(String subPathStr,
                                    SerializableConfiguration conf,
                                    boolean recursive)

Deletes a sub-path.

Parameters:

subPathStr - sub-path String

conf - serializable config

recursive - is recursive or not

Returns:

true if the sub-path is deleted; false otherwise.

getFileStatusAtLevel

public static List<org.apache.hadoop.fs.FileStatus> getFileStatusAtLevel(HoodieEngineContext hoodieEngineContext,
                                                                         org.apache.hadoop.fs.FileSystem fs,
                                                                         org.apache.hadoop.fs.Path rootPath,
                                                                         int expectLevel,
                                                                         int parallelism)

Lists file status at a certain level in the directory hierarchy.

E.g., given "/tmp/hoodie_table" as the rootPath, and 3 as the expected level, this method gives back the FileStatus of all files under "/tmp/hoodie_table/[*]/[*]/[*]/" folders.

Parameters:

hoodieEngineContext - HoodieEngineContext instance.

fs - FileSystem instance.

rootPath - Root path for the file listing.

expectLevel - Expected level of directory hierarchy for files to be added.

parallelism - Parallelism for the file listing.

Returns:

A list of file status of files at the level.