org.apache.hudi.common.fs

Class FSUtils

java.lang.Object

org.apache.hudi.common.fs.FSUtils

public class FSUtils
extends Object

Utility functions related to accessing the file storage.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	FSUtils.SerializableFunction<T,R> Serializable function interface.

Field Summary

Fields

Modifier and Type	Field and Description
static Pattern	LOG_FILE_PATTERN
static String	PATH_SEPARATOR
static Pattern	PREFIX_BY_FILE_ID_PATTERN

Constructor Summary

Constructors

Constructor and Description
FSUtils()

Method Summary

Modifier and Type	Method and Description
static int	computeNextLogVersion(HoodieStorage storage, StoragePath partitionPath, String fileId, String logFileExtension, String baseCommitTime) computes the next log version for the specified fileId in the partition path.
static StoragePath	constructAbsolutePath(StoragePath basePath, String relativePartitionPath)
static StoragePath	constructAbsolutePath(String basePath, String relativePartitionPath)
static String	createNewFileId(String idPfx, int id)
static String	createNewFileIdPfx() Returns a new unique prefix for creating a file group.
static void	createPathIfNotExists(HoodieStorage storage, StoragePath partitionPath)
static boolean	deleteDir(HoodieEngineContext hoodieEngineContext, HoodieStorage storage, StoragePath dirPath, int parallelism) Deletes a directory by deleting sub-paths in parallel on the file system.
static boolean	deleteSubPath(String subPathStr, StorageConfiguration<?> conf, boolean recursive) Deletes a sub-path.
static List<StoragePathInfo>	getAllDataFilesInPartition(HoodieStorage storage, StoragePath partitionPath)
static List<StoragePathInfo>	getAllDataPathInfo(HoodieStorage storage, StoragePath path)
static Stream<HoodieLogFile>	getAllLogFiles(HoodieStorage storage, StoragePath partitionPath, String fileId, String logFileExtension, String baseCommitTime) Get all the log files for the passed in file-id in the partition path.
static List<String>	getAllPartitionFoldersThreeLevelsDown(HoodieStorage storage, String basePath) Gets all partition paths assuming date partitioning (year, month, day) three levels down.
static List<String>	getAllPartitionPaths(HoodieEngineContext engineContext, HoodieStorage storage, HoodieMetadataConfig metadataConfig, String basePathStr)
static List<String>	getAllPartitionPaths(HoodieEngineContext engineContext, HoodieStorage storage, String basePathStr, boolean useFileListingFromMetadata, boolean assumeDatePartitioning)
static String	getBaseCommitTimeFromLogPath(StoragePath path) Get the first part of the file name in the log file.
static String	getCommitFromCommitFile(String commitFileName)
static String	getCommitTime(String fullFileName)
static String	getFileExtension(String fullName)
static String	getFileExtensionFromLog(StoragePath logPath) Get the file extension from the log file.
static String	getFileId(String fullFileName)
static String	getFileIdFromFilePath(StoragePath filePath)
static String	getFileIdFromLogPath(StoragePath path)
static String	getFileIdPfxFromFileId(String fileId) Returns prefix for a file group from fileId.
static String	getFileName(String filePathWithPartition, String partition) Extracts the file name from the relative path based on the table base path.
static Map<String,List<StoragePathInfo>>	getFilesInPartitions(HoodieEngineContext engineContext, HoodieStorage storage, HoodieMetadataConfig metadataConfig, String basePathStr, String[] partitionPaths)
static long	getFileSize(HoodieStorage storage, StoragePath path)
static int	getFileVersionFromLog(StoragePath logPath) Get the last part of the file name in the log file and convert to int.
static int	getFileVersionFromLog(String logFileName)
static List<StoragePathInfo>	getGlobStatusExcludingMetaFolder(HoodieStorage storage, StoragePath globPath) Helper to filter out paths under metadata folder when running fs.globStatus.
static Option<HoodieLogFile>	getLatestLogFile(HoodieStorage storage, StoragePath partitionPath, String fileId, String logFileExtension, String baseCommitTime) Get the latest log file for the passed in file-id in the partition path
static Option<Pair<Integer,String>>	getLatestLogVersion(HoodieStorage storage, StoragePath partitionPath, String fileId, String logFileExtension, String baseCommitTime) Get the latest log version for the fileId in the partition path.
static List<Option<StoragePathInfo>>	getPathInfoUnderPartition(HoodieStorage storage, StoragePath partitionPathIncludeBasePath, Set<String> filesNamesUnderThisPartition, boolean ignoreMissingFiles) Get all the files in the given partition path.
static StoragePath	getPathWithoutSchemeAndAuthority(StoragePath path)
static String	getRelativePartitionPath(StoragePath basePath, StoragePath fullPartitionPath)
static Long	getSizeInMB(long sizeInBytes)
static Integer	getStageIdFromLogPath(StoragePath path) Get StageId used in log-path.
static Integer	getTaskAttemptIdFromLogPath(StoragePath path) Get Task Attempt Id used in log-path.
static Integer	getTaskPartitionIdFromLogPath(StoragePath path) Get TaskPartitionId used in log-path.
static String	getWriteTokenFromLogPath(StoragePath path) Get Write-Token used in log-path.
static boolean	isBaseFile(StoragePath path)
static boolean	isDataFile(StoragePath path)
static boolean	isLogFile(StoragePath logPath)
static boolean	isLogFile(String fileName)
static boolean	isTableExists(String path, HoodieStorage storage) Check if table already exists in the given path.
static String	makeBaseFileName(String instantTime, String writeToken, String fileId, String fileExtension)
static String	makeBootstrapIndexFileName(String instantTime, String fileId, String ext)
static String	makeLogFileName(String fileId, String logFileExtension, String baseCommitTime, int version, String writeToken)
static StoragePath	makeQualified(HoodieStorage storage, StoragePath path) Makes path qualified with HoodieStorage's URI.
static String	makeWriteToken(int taskPartitionId, int stageId, long taskAttemptId) A write token uniquely identifies an attempt at one of the IOHandle operations (Merge/Create/Append).
static String	maskWithoutFileId(String instantTime, int taskPartitionId)
static <T> Map<String,T>	parallelizeFilesProcess(HoodieEngineContext hoodieEngineContext, HoodieStorage storage, int parallelism, FSUtils.SerializableFunction<Pair<String,StorageConfiguration<?>>,T> pairFunction, List<String> subPaths)
static <T> Map<String,T>	parallelizeSubPathProcess(HoodieEngineContext hoodieEngineContext, HoodieStorage storage, StoragePath dirPath, int parallelism, Predicate<StoragePathInfo> subPathPredicate, FSUtils.SerializableFunction<Pair<String,StorageConfiguration<?>>,T> pairFunction) Processes sub-path in parallel.
static void	processFiles(HoodieStorage storage, String basePathStr, Function<StoragePathInfo,Boolean> consumer, boolean excludeMetaFolder) Recursively processes all files in the base-path.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PATH_SEPARATOR

public static final String PATH_SEPARATOR

See Also:

Constant Field Values

LOG_FILE_PATTERN

public static final Pattern LOG_FILE_PATTERN

PREFIX_BY_FILE_ID_PATTERN

public static final Pattern PREFIX_BY_FILE_ID_PATTERN

Constructor Detail

FSUtils

public FSUtils()

Method Detail

isTableExists

public static boolean isTableExists(String path,
                                    HoodieStorage storage)
                             throws IOException

Check if table already exists in the given path.

Parameters:

path - base path of the table.

storage - instance of HoodieStorage.

Returns:

true if table exists. false otherwise.

Throws:

IOException

makeQualified

public static StoragePath makeQualified(HoodieStorage storage,
                                        StoragePath path)

Makes path qualified with HoodieStorage's URI.

Parameters:

storage - instance of HoodieStorage.

path - to be qualified.

Returns:

qualified path, prefixed with the URI of the target HoodieStorage object provided.

makeWriteToken

public static String makeWriteToken(int taskPartitionId,
                                    int stageId,
                                    long taskAttemptId)

A write token uniquely identifies an attempt at one of the IOHandle operations (Merge/Create/Append).

makeBaseFileName

public static String makeBaseFileName(String instantTime,
                                      String writeToken,
                                      String fileId,
                                      String fileExtension)

makeBootstrapIndexFileName

public static String makeBootstrapIndexFileName(String instantTime,
                                                String fileId,
                                                String ext)

maskWithoutFileId

public static String maskWithoutFileId(String instantTime,
                                       int taskPartitionId)

getCommitFromCommitFile

public static String getCommitFromCommitFile(String commitFileName)

getCommitTime

public static String getCommitTime(String fullFileName)

getFileSize

public static long getFileSize(HoodieStorage storage,
                               StoragePath path)
                        throws IOException

Throws:

IOException

getFileId

public static String getFileId(String fullFileName)

getAllPartitionFoldersThreeLevelsDown

public static List<String> getAllPartitionFoldersThreeLevelsDown(HoodieStorage storage,
                                                                 String basePath)
                                                          throws IOException

Gets all partition paths assuming date partitioning (year, month, day) three levels down.

Throws:

IOException

getRelativePartitionPath

public static String getRelativePartitionPath(StoragePath basePath,
                                              StoragePath fullPartitionPath)

getPathWithoutSchemeAndAuthority

public static StoragePath getPathWithoutSchemeAndAuthority(StoragePath path)

processFiles

public static void processFiles(HoodieStorage storage,
                                String basePathStr,
                                Function<StoragePathInfo,Boolean> consumer,
                                boolean excludeMetaFolder)
                         throws IOException

Recursively processes all files in the base-path. If excludeMetaFolder is set, the meta-folder and all its subdirs are skipped

Parameters:

storage - File System

basePathStr - Base-Path

consumer - Callback for processing

excludeMetaFolder - Exclude .hoodie folder

Throws:

IOException - -

getAllPartitionPaths

public static List<String> getAllPartitionPaths(HoodieEngineContext engineContext,
                                                HoodieStorage storage,
                                                String basePathStr,
                                                boolean useFileListingFromMetadata,
                                                boolean assumeDatePartitioning)

getAllPartitionPaths

public static List<String> getAllPartitionPaths(HoodieEngineContext engineContext,
                                                HoodieStorage storage,
                                                HoodieMetadataConfig metadataConfig,
                                                String basePathStr)

getFilesInPartitions

public static Map<String,List<StoragePathInfo>> getFilesInPartitions(HoodieEngineContext engineContext,
                                                                     HoodieStorage storage,
                                                                     HoodieMetadataConfig metadataConfig,
                                                                     String basePathStr,
                                                                     String[] partitionPaths)

getPathInfoUnderPartition

public static List<Option<StoragePathInfo>> getPathInfoUnderPartition(HoodieStorage storage,
                                                                      StoragePath partitionPathIncludeBasePath,
                                                                      Set<String> filesNamesUnderThisPartition,
                                                                      boolean ignoreMissingFiles)

Get all the files in the given partition path.

Parameters:

storage - HoodieStorage instance.

partitionPathIncludeBasePath - The full partition path including the base path

filesNamesUnderThisPartition - The names of the files under this partition for which file status is needed

ignoreMissingFiles - If true, missing files will be ignored and empty Option will be added to the result list

Returns:

List of file statuses for the files under this partition

getFileExtension

public static String getFileExtension(String fullName)

createNewFileIdPfx

public static String createNewFileIdPfx()

Returns a new unique prefix for creating a file group.

getFileIdPfxFromFileId

public static String getFileIdPfxFromFileId(String fileId)

Returns prefix for a file group from fileId.

createNewFileId

public static String createNewFileId(String idPfx,
                                     int id)

getFileExtensionFromLog

public static String getFileExtensionFromLog(StoragePath logPath)

Get the file extension from the log file.

getFileIdFromLogPath

public static String getFileIdFromLogPath(StoragePath path)

getFileIdFromFilePath

public static String getFileIdFromFilePath(StoragePath filePath)

getBaseCommitTimeFromLogPath

public static String getBaseCommitTimeFromLogPath(StoragePath path)

Get the first part of the file name in the log file. That will be the fileId. Log file do not have instantTime in the file name.

getTaskPartitionIdFromLogPath

public static Integer getTaskPartitionIdFromLogPath(StoragePath path)

Get TaskPartitionId used in log-path.

getWriteTokenFromLogPath

public static String getWriteTokenFromLogPath(StoragePath path)

Get Write-Token used in log-path.

getStageIdFromLogPath

public static Integer getStageIdFromLogPath(StoragePath path)

Get StageId used in log-path.

getTaskAttemptIdFromLogPath

public static Integer getTaskAttemptIdFromLogPath(StoragePath path)

Get Task Attempt Id used in log-path.

getFileVersionFromLog

public static int getFileVersionFromLog(StoragePath logPath)

Get the last part of the file name in the log file and convert to int.

getFileVersionFromLog

public static int getFileVersionFromLog(String logFileName)

makeLogFileName

public static String makeLogFileName(String fileId,
                                     String logFileExtension,
                                     String baseCommitTime,
                                     int version,
                                     String writeToken)

isBaseFile

public static boolean isBaseFile(StoragePath path)

isLogFile

public static boolean isLogFile(StoragePath logPath)

isLogFile

public static boolean isLogFile(String fileName)

isDataFile

public static boolean isDataFile(StoragePath path)

getAllDataFilesInPartition

public static List<StoragePathInfo> getAllDataFilesInPartition(HoodieStorage storage,
                                                               StoragePath partitionPath)
                                                        throws IOException

Throws:

IOException

getLatestLogFile

public static Option<HoodieLogFile> getLatestLogFile(HoodieStorage storage,
                                                     StoragePath partitionPath,
                                                     String fileId,
                                                     String logFileExtension,
                                                     String baseCommitTime)
                                              throws IOException

Get the latest log file for the passed in file-id in the partition path

Throws:

IOException

getAllLogFiles

public static Stream<HoodieLogFile> getAllLogFiles(HoodieStorage storage,
                                                   StoragePath partitionPath,
                                                   String fileId,
                                                   String logFileExtension,
                                                   String baseCommitTime)
                                            throws IOException

Get all the log files for the passed in file-id in the partition path.

Throws:

IOException

getLatestLogVersion

public static Option<Pair<Integer,String>> getLatestLogVersion(HoodieStorage storage,
                                                               StoragePath partitionPath,
                                                               String fileId,
                                                               String logFileExtension,
                                                               String baseCommitTime)
                                                        throws IOException

Get the latest log version for the fileId in the partition path.

Throws:

IOException

computeNextLogVersion

public static int computeNextLogVersion(HoodieStorage storage,
                                        StoragePath partitionPath,
                                        String fileId,
                                        String logFileExtension,
                                        String baseCommitTime)
                                 throws IOException

computes the next log version for the specified fileId in the partition path.

Throws:

IOException

createPathIfNotExists

public static void createPathIfNotExists(HoodieStorage storage,
                                         StoragePath partitionPath)
                                  throws IOException

Throws:

IOException

getSizeInMB

public static Long getSizeInMB(long sizeInBytes)

constructAbsolutePath

public static StoragePath constructAbsolutePath(String basePath,
                                                String relativePartitionPath)

constructAbsolutePath

public static StoragePath constructAbsolutePath(StoragePath basePath,
                                                String relativePartitionPath)

getFileName

public static String getFileName(String filePathWithPartition,
                                 String partition)

Extracts the file name from the relative path based on the table base path. For example: "/2022/07/29/file1.parquet", "/2022/07/29" -> "file1.parquet" "2022/07/29/file2.parquet", "2022/07/29" -> "file2.parquet" "/file3.parquet", "" -> "file3.parquet" "file4.parquet", "" -> "file4.parquet"

Parameters:

filePathWithPartition - the relative file path based on the table base path.

partition - the relative partition path. For partitioned table, `partition` contains the relative partition path; for non-partitioned table, `partition` is empty

Returns:

Extracted file name in String.

getGlobStatusExcludingMetaFolder

public static List<StoragePathInfo> getGlobStatusExcludingMetaFolder(HoodieStorage storage,
                                                                     StoragePath globPath)
                                                              throws IOException

Helper to filter out paths under metadata folder when running fs.globStatus.

Parameters:

storage - HoodieStorage instance.

globPath - Glob Path

Returns:

the file status list of globPath exclude the meta folder

Throws:

IOException - when having trouble listing the path

deleteDir

public static boolean deleteDir(HoodieEngineContext hoodieEngineContext,
                                HoodieStorage storage,
                                StoragePath dirPath,
                                int parallelism)

Deletes a directory by deleting sub-paths in parallel on the file system.

Parameters:

hoodieEngineContext - HoodieEngineContext instance

storage - HoodieStorage instance.

dirPath - directory path.

parallelism - parallelism to use for sub-paths

Returns:

true if the directory is delete; false otherwise.

parallelizeSubPathProcess

public static <T> Map<String,T> parallelizeSubPathProcess(HoodieEngineContext hoodieEngineContext,
                                                          HoodieStorage storage,
                                                          StoragePath dirPath,
                                                          int parallelism,
                                                          Predicate<StoragePathInfo> subPathPredicate,
                                                          FSUtils.SerializableFunction<Pair<String,StorageConfiguration<?>>,T> pairFunction)

Processes sub-path in parallel.

Type Parameters:

T - type of result to return for each sub-path

Parameters:

hoodieEngineContext - HoodieEngineContext instance

storage - HoodieStorage instance

dirPath - directory path

parallelism - parallelism to use for sub-paths

subPathPredicate - predicate to use to filter sub-paths for processing

pairFunction - actual processing logic for each sub-path

Returns:

a map of sub-path to result of the processing

parallelizeFilesProcess

public static <T> Map<String,T> parallelizeFilesProcess(HoodieEngineContext hoodieEngineContext,
                                                        HoodieStorage storage,
                                                        int parallelism,
                                                        FSUtils.SerializableFunction<Pair<String,StorageConfiguration<?>>,T> pairFunction,
                                                        List<String> subPaths)

deleteSubPath

public static boolean deleteSubPath(String subPathStr,
                                    StorageConfiguration<?> conf,
                                    boolean recursive)

Deletes a sub-path.

Parameters:

subPathStr - sub-path String

conf - storage config

recursive - is recursive or not

Returns:

true if the sub-path is deleted; false otherwise.

getAllDataPathInfo

public static List<StoragePathInfo> getAllDataPathInfo(HoodieStorage storage,
                                                       StoragePath path)
                                                throws IOException

Throws:

IOException