org.apache.hudi.common.util

Class FileIOUtils

java.lang.Object

org.apache.hudi.common.util.FileIOUtils

public class FileIOUtils
extends Object

Bunch of utility methods for working with files and byte streams.

Field Summary

Fields

Modifier and Type	Field and Description
static long	KB
static org.slf4j.Logger	LOG

Constructor Summary

Constructors

Constructor and Description
FileIOUtils()

Method Summary

Modifier and Type	Method and Description
static void	closeQuietly(Closeable closeable) Closes Closeable quietly.
static boolean	copy(HoodieStorage srcStorage, StoragePath src, HoodieStorage dstStorage, StoragePath dst, boolean deleteSource, boolean overwrite)
static boolean	copy(HoodieStorage srcStorage, StoragePathInfo srcPathInfo, HoodieStorage dstStorage, StoragePath dst, boolean deleteSource, boolean overwrite) Copy files between FileSystems.
static void	copy(HoodieStorage storage, StoragePath sourceFilePath, StoragePath destFilePath) Copies the file content from source path to destination path.
static void	copy(InputStream inputStream, OutputStream outputStream)
static void	createFileInPath(HoodieStorage storage, StoragePath fullPath, Option<byte[]> content)
static void	createFileInPath(HoodieStorage storage, StoragePath fullPath, Option<byte[]> content, boolean ignoreIOE)
static void	deleteDirectory(File directory)
static String[]	getConfiguredLocalDirs() Return the configured local directories where hudi can write files.
static String	getDefaultSpillableMapBasePath()
static void	mkdir(File directory)
static byte[]	readAsByteArray(InputStream input)
static byte[]	readAsByteArray(InputStream input, int outputSize)
static String	readAsUTFString(InputStream input)
static String	readAsUTFString(InputStream input, int length)
static List<String>	readAsUTFStringLines(InputStream input) Reads the input stream into String lines.
static Option<byte[]>	readDataFromPath(HoodieStorage storage, StoragePath detailPath)
static Option<byte[]>	readDataFromPath(HoodieStorage storage, StoragePath detailPath, boolean ignoreIOE)
static void	writeStringToFile(String str, String filePath)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

KB

public static final long KB

See Also:

Constant Field Values

Constructor Detail

FileIOUtils

public FileIOUtils()

Method Detail

deleteDirectory

public static void deleteDirectory(File directory)
                            throws IOException

Throws:

IOException

mkdir

public static void mkdir(File directory)
                  throws IOException

Throws:

IOException

readAsUTFString

public static String readAsUTFString(InputStream input)
                              throws IOException

Throws:

IOException

readAsUTFString

public static String readAsUTFString(InputStream input,
                                     int length)
                              throws IOException

Throws:

IOException

readAsUTFStringLines

public static List<String> readAsUTFStringLines(InputStream input)

Reads the input stream into String lines.

Parameters:

input - InputStream instance.

Returns:

String lines in a list.

copy

public static void copy(InputStream inputStream,
                        OutputStream outputStream)
                 throws IOException

Throws:

IOException

copy

public static void copy(HoodieStorage storage,
                        StoragePath sourceFilePath,
                        StoragePath destFilePath)

Copies the file content from source path to destination path.

Parameters:

storage - HoodieStorage instance.

sourceFilePath - Source file path.

destFilePath - Destination file path.

readAsByteArray

public static byte[] readAsByteArray(InputStream input)
                              throws IOException

Throws:

IOException

readAsByteArray

public static byte[] readAsByteArray(InputStream input,
                                     int outputSize)
                              throws IOException

Throws:

IOException

writeStringToFile

public static void writeStringToFile(String str,
                                     String filePath)
                              throws IOException

Throws:

IOException

closeQuietly

public static void closeQuietly(Closeable closeable)

Closes Closeable quietly.

Parameters:

closeable - Closeable to close

createFileInPath

public static void createFileInPath(HoodieStorage storage,
                                    StoragePath fullPath,
                                    Option<byte[]> content,
                                    boolean ignoreIOE)

createFileInPath

public static void createFileInPath(HoodieStorage storage,
                                    StoragePath fullPath,
                                    Option<byte[]> content)

copy

public static boolean copy(HoodieStorage srcStorage,
                           StoragePath src,
                           HoodieStorage dstStorage,
                           StoragePath dst,
                           boolean deleteSource,
                           boolean overwrite)
                    throws IOException

Throws:

IOException

copy

public static boolean copy(HoodieStorage srcStorage,
                           StoragePathInfo srcPathInfo,
                           HoodieStorage dstStorage,
                           StoragePath dst,
                           boolean deleteSource,
                           boolean overwrite)
                    throws IOException

Copy files between FileSystems.

Throws:

IOException

readDataFromPath

public static Option<byte[]> readDataFromPath(HoodieStorage storage,
                                              StoragePath detailPath,
                                              boolean ignoreIOE)

readDataFromPath

public static Option<byte[]> readDataFromPath(HoodieStorage storage,
                                              StoragePath detailPath)

getConfiguredLocalDirs

public static String[] getConfiguredLocalDirs()

Return the configured local directories where hudi can write files. This method does not create any directories on its own, it only encapsulates the logic of locating the local directories according to deployment mode.

getDefaultSpillableMapBasePath

public static String getDefaultSpillableMapBasePath()