org.apache.hudi.storage

Class HoodieStorage

java.lang.Object

org.apache.hudi.storage.HoodieStorage

All Implemented Interfaces:

Closeable, AutoCloseable

@PublicAPIClass(maturity=EVOLVING)
public abstract class HoodieStorage
extends Object
implements Closeable

Provides I/O APIs on files and directories on storage. The APIs are mainly based on org.apache.hadoop.fs.FileSystem class.

Field Summary

Fields

Modifier and Type	Field and Description
static org.slf4j.Logger	LOG
protected StorageConfiguration<?>	storageConf

Constructor Summary

Constructors

Constructor and Description
HoodieStorage(StorageConfiguration<?> storageConf)

Method Summary

Modifier and Type	Method and Description
abstract OutputStream	append(StoragePath path) Appends to an existing file (optional operation).
OutputStream	create(StoragePath path) Create an OutputStream at the indicated path.
abstract OutputStream	create(StoragePath path, boolean overwrite) Creates an OutputStream at the indicated path.
abstract OutputStream	create(StoragePath path, boolean overwrite, Integer bufferSize, Short replication, Long sizeThreshold) Creates an OutputStream at the indicated path.
abstract boolean	createDirectory(StoragePath path) Creates the directory and non-existent parent directories.
void	createImmutableFileInPath(StoragePath path, Option<HoodieInstantWriter> contentWriter) Creates a new file with overwrite set to false.
void	createImmutableFileInPath(StoragePath path, Option<HoodieInstantWriter> contentWriter, boolean needTempFile) Creates a new file with overwrite set to false.
boolean	createNewFile(StoragePath path) Creates an empty new file at the indicated path.
abstract boolean	deleteDirectory(StoragePath path) Deletes a directory at path.
abstract boolean	deleteFile(StoragePath path) Deletes a file at path.
abstract boolean	exists(StoragePath path) Checks if a path exists.
StorageConfiguration<?>	getConf()
abstract int	getDefaultBlockSize(StoragePath path)
abstract int	getDefaultBufferSize()
abstract short	getDefaultReplication(StoragePath path)
abstract Object	getFileSystem()
abstract StoragePathInfo	getPathInfo(StoragePath path) Returns a StoragePathInfo object that represents the path.
abstract HoodieStorage	getRawStorage()
abstract String	getScheme()
abstract URI	getUri() Returns a URI which identifies this HoodieStorage.
List<StoragePathInfo>	globEntries(StoragePath pathPattern) Returns all the files that match the pathPattern and are not checksum files.
abstract List<StoragePathInfo>	globEntries(StoragePath pathPattern, StoragePathFilter filter) Returns all the files that match the pathPattern and are not checksum files, and filters the results.
List<StoragePathInfo>	listDirectEntries(List<StoragePath> pathList) Lists the file info of the direct files/directories in the given list of paths, if the paths are directory.
List<StoragePathInfo>	listDirectEntries(List<StoragePath> pathList, StoragePathFilter filter) Lists the file info of the direct files/directories in the given list of paths and filters the results, if the paths are directory.
abstract List<StoragePathInfo>	listDirectEntries(StoragePath path) Lists the path info of the direct files/directories in the given path if the path is a directory.
abstract List<StoragePathInfo>	listDirectEntries(StoragePath path, StoragePathFilter filter) Lists the path info of the direct files/directories in the given path and filters the results, if the path is a directory.
abstract List<StoragePathInfo>	listFiles(StoragePath path) Lists the path info of all files under the give path recursively.
boolean	needCreateTempFile()
abstract HoodieStorage	newInstance(StoragePath path, StorageConfiguration<?> storageConf)
abstract InputStream	open(StoragePath path) Opens an InputStream at the indicated path.
SeekableDataInputStream	openSeekable(StoragePath path, boolean wrapStream) Opens an SeekableDataInputStream at the indicated path with seeks supported.
abstract SeekableDataInputStream	openSeekable(StoragePath path, int bufferSize, boolean wrapStream) Opens an SeekableDataInputStream at the indicated path with seeks supported.
abstract boolean	rename(StoragePath oldPath, StoragePath newPath) Renames the path from old to new.
abstract void	setModificationTime(StoragePath path, long modificationTimeInMillisEpoch) Sets Modification Time for the storage Path

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.io.Closeable

close

Field Detail

LOG

public static final org.slf4j.Logger LOG

storageConf

protected final StorageConfiguration<?> storageConf

Constructor Detail

HoodieStorage

public HoodieStorage(StorageConfiguration<?> storageConf)

Method Detail

newInstance

@PublicAPIMethod(maturity=EVOLVING)
public abstract HoodieStorage newInstance(StoragePath path,
                                                                              StorageConfiguration<?> storageConf)

Parameters:

path - path to instantiate the storage.

storageConf - new storage configuration.

Returns:

new HoodieStorage instance with the configuration.

getScheme

@PublicAPIMethod(maturity=EVOLVING)
public abstract String getScheme()

Returns:

the scheme of the storage.

getDefaultBlockSize

@PublicAPIMethod(maturity=EVOLVING)
public abstract int getDefaultBlockSize(StoragePath path)

Returns:

the default block size.

getDefaultBufferSize

@PublicAPIMethod(maturity=EVOLVING)
public abstract int getDefaultBufferSize()

Returns:

the default buffer size.

getDefaultReplication

@PublicAPIMethod(maturity=EVOLVING)
public abstract short getDefaultReplication(StoragePath path)

Returns:

the default block replication

getUri

@PublicAPIMethod(maturity=EVOLVING)
public abstract URI getUri()

Returns a URI which identifies this HoodieStorage.

Returns:

the URI of this storage instance.

create

@PublicAPIMethod(maturity=EVOLVING)
public abstract OutputStream create(StoragePath path,
                                                                        boolean overwrite)
                                                                 throws IOException

Creates an OutputStream at the indicated path.

Parameters:

path - the file to create.

overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an exception will be thrown.

Returns:

the OutputStream to write to.

Throws:

IOException - IO error.

create

@PublicAPIMethod(maturity=EVOLVING)
public abstract OutputStream create(StoragePath path,
                                                                        boolean overwrite,
                                                                        Integer bufferSize,
                                                                        Short replication,
                                                                        Long sizeThreshold)
                                                                 throws IOException

Creates an OutputStream at the indicated path.

Parameters:

path - the file to create

overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an exception will be thrown.

bufferSize - the size of the buffer to be used

replication - required block replication for the file

sizeThreshold - block size

Returns:

the OutputStream to write to.

Throws:

IOException - IO error.

open

@PublicAPIMethod(maturity=EVOLVING)
public abstract InputStream open(StoragePath path)
                                                              throws IOException

Opens an InputStream at the indicated path.

Parameters:

path - the file to open.

Returns:

the InputStream to read from.

Throws:

IOException - IO error.

openSeekable

@PublicAPIMethod(maturity=EVOLVING)
public abstract SeekableDataInputStream openSeekable(StoragePath path,
                                                                                         int bufferSize,
                                                                                         boolean wrapStream)
                                                                                  throws IOException

Opens an SeekableDataInputStream at the indicated path with seeks supported.

Parameters:

path - the file to open.

bufferSize - buffer size to use.

wrapStream - true if we want to wrap the inputstream based on filesystem specific criteria

Returns:

the InputStream to read from.

Throws:

IOException - IO error.

append

@PublicAPIMethod(maturity=EVOLVING)
public abstract OutputStream append(StoragePath path)
                                                                 throws IOException

Appends to an existing file (optional operation).

Parameters:

path - the file to append.

Returns:

the OutputStream to write to.

Throws:

IOException - IO error.

exists

@PublicAPIMethod(maturity=EVOLVING)
public abstract boolean exists(StoragePath path)
                                                            throws IOException

Checks if a path exists.

Parameters:

path - to check.

Returns:

true if the path exists.

Throws:

IOException - IO error.

getPathInfo

@PublicAPIMethod(maturity=EVOLVING)
public abstract StoragePathInfo getPathInfo(StoragePath path)
                                                                         throws IOException

Returns a StoragePathInfo object that represents the path.

Parameters:

path - to check.

Returns:

a StoragePathInfo object.

Throws:

FileNotFoundException - when the path does not exist.

IOException - IO error.

createDirectory

@PublicAPIMethod(maturity=EVOLVING)
public abstract boolean createDirectory(StoragePath path)
                                                                     throws IOException

Creates the directory and non-existent parent directories.

Parameters:

path - to create.

Returns:

true if the directory was created.

Throws:

IOException - IO error.

listDirectEntries

@PublicAPIMethod(maturity=EVOLVING)
public abstract List<StoragePathInfo> listDirectEntries(StoragePath path)
                                                                                     throws IOException

Lists the path info of the direct files/directories in the given path if the path is a directory.

Parameters:

path - given path.

Returns:

the list of path info of the files/directories in the given path.

Throws:

FileNotFoundException - when the path does not exist.

IOException - IO error.

listFiles

@PublicAPIMethod(maturity=EVOLVING)
public abstract List<StoragePathInfo> listFiles(StoragePath path)
                                                                             throws IOException

Lists the path info of all files under the give path recursively.

Parameters:

path - given path.

Returns:

the list of path info of the files under the given path.

Throws:

FileNotFoundException - when the path does not exist.

IOException - IO error.

listDirectEntries

@PublicAPIMethod(maturity=EVOLVING)
public abstract List<StoragePathInfo> listDirectEntries(StoragePath path,
                                                                                            StoragePathFilter filter)
                                                                                     throws IOException

Lists the path info of the direct files/directories in the given path and filters the results, if the path is a directory.

Parameters:

path - given path.

filter - filter to apply.

Returns:

the list of path info of the files/directories in the given path.

Throws:

FileNotFoundException - when the path does not exist.

IOException - IO error.

setModificationTime

@PublicAPIMethod(maturity=EVOLVING)
public abstract void setModificationTime(StoragePath path,
                                                                             long modificationTimeInMillisEpoch)
                                                                      throws IOException

Sets Modification Time for the storage Path

Parameters:

path -

modificationTimeInMillisEpoch - Millis since Epoch

Throws:

IOException

globEntries

@PublicAPIMethod(maturity=EVOLVING)
public abstract List<StoragePathInfo> globEntries(StoragePath pathPattern,
                                                                                      StoragePathFilter filter)
                                                                               throws IOException

Returns all the files that match the pathPattern and are not checksum files, and filters the results.

Parameters:

pathPattern - given pattern.

filter - filter to apply.

Returns:

the list of path info of the files.

Throws:

IOException - IO error.

rename

@PublicAPIMethod(maturity=EVOLVING)
public abstract boolean rename(StoragePath oldPath,
                                                                   StoragePath newPath)
                                                            throws IOException

Renames the path from old to new.

Parameters:

oldPath - source path.

newPath - destination path.

Returns:

if rename is successful.

Throws:

IOException - IO error.

deleteDirectory

@PublicAPIMethod(maturity=EVOLVING)
public abstract boolean deleteDirectory(StoragePath path)
                                                                     throws IOException

Deletes a directory at path.

Parameters:

path - directory to delete.

Returns:

true if successful.

Throws:

IOException - IO error.

deleteFile

@PublicAPIMethod(maturity=EVOLVING)
public abstract boolean deleteFile(StoragePath path)
                                                                throws IOException

Deletes a file at path.

Parameters:

path - file to delete.

Returns:

true if successful.

Throws:

IOException - IO error.

getFileSystem

@PublicAPIMethod(maturity=EVOLVING)
public abstract Object getFileSystem()

Returns:

the underlying file system instance if exists.

getRawStorage

@PublicAPIMethod(maturity=EVOLVING)
public abstract HoodieStorage getRawStorage()

Returns:

the raw storage.

getConf

@PublicAPIMethod(maturity=EVOLVING)
public final StorageConfiguration<?> getConf()

Returns:

the storage configuration.

createImmutableFileInPath

@PublicAPIMethod(maturity=EVOLVING)
public final void createImmutableFileInPath(StoragePath path,
                                                                                Option<HoodieInstantWriter> contentWriter)
                                                                         throws HoodieIOException

Creates a new file with overwrite set to false. This ensures files are created only once and never rewritten, also, here we take care if the content is not empty, will first write the content to a temp file if {needCreateTempFile} is true, and then rename it back after the content is written.

CAUTION: if this method is invoked in multi-threads for concurrent write of the same file, an existence check of the file is recommended.

Parameters:

path - File path.

contentWriter - handles writing the content to the outputstream

Throws:

HoodieIOException

createImmutableFileInPath

@PublicAPIMethod(maturity=EVOLVING)
public final void createImmutableFileInPath(StoragePath path,
                                                                                Option<HoodieInstantWriter> contentWriter,
                                                                                boolean needTempFile)
                                                                         throws HoodieIOException

Creates a new file with overwrite set to false. This ensures files are created only once and never rewritten, also, here we take care if the content is not empty, will first write the content to a temp file if {needCreateTempFile} is true, and then rename it back after the content is written.

CAUTION: if this method is invoked in multi-threads for concurrent write of the same file, an existence check of the file is recommended.

Parameters:

path - File path.

contentWriter - handles writing the content to the outputstream

needTempFile - Whether to create auxiliary temp file.

Throws:

HoodieIOException

needCreateTempFile

@PublicAPIMethod(maturity=EVOLVING)
public final boolean needCreateTempFile()

Returns:

whether a temporary file needs to be created for immutability.

create

@PublicAPIMethod(maturity=EVOLVING)
public OutputStream create(StoragePath path)
                                                        throws IOException

Create an OutputStream at the indicated path. The file is overwritten by default.

Parameters:

path - the file to create.

Returns:

the OutputStream to write to.

Throws:

IOException - IO error.

createNewFile

@PublicAPIMethod(maturity=EVOLVING)
public boolean createNewFile(StoragePath path)
                                                          throws IOException

Creates an empty new file at the indicated path.

Parameters:

path - the file to create.

Returns:

true if successfully created; false if already exists.

Throws:

IOException - IO error.

openSeekable

@PublicAPIMethod(maturity=EVOLVING)
public SeekableDataInputStream openSeekable(StoragePath path,
                                                                                boolean wrapStream)
                                                                         throws IOException

Opens an SeekableDataInputStream at the indicated path with seeks supported.

Parameters:

path - the file to open.

wrapStream - true if we want to wrap the inputstream based on filesystem specific criteria

Returns:

the InputStream to read from.

Throws:

IOException - IO error.

listDirectEntries

@PublicAPIMethod(maturity=EVOLVING)
public List<StoragePathInfo> listDirectEntries(List<StoragePath> pathList)
                                                                            throws IOException

Lists the file info of the direct files/directories in the given list of paths, if the paths are directory.

Parameters:

pathList - given path list.

Returns:

the list of path info of the files/directories in the given paths.

Throws:

FileNotFoundException - when the path does not exist.

IOException - IO error.

listDirectEntries

@PublicAPIMethod(maturity=EVOLVING)
public List<StoragePathInfo> listDirectEntries(List<StoragePath> pathList,
                                                                                   StoragePathFilter filter)
                                                                            throws IOException

Lists the file info of the direct files/directories in the given list of paths and filters the results, if the paths are directory.

Parameters:

pathList - given path list.

filter - filter to apply.

Returns:

the list of path info of the files/directories in the given paths.

Throws:

FileNotFoundException - when the path does not exist.

IOException - IO error.

globEntries

@PublicAPIMethod(maturity=EVOLVING)
public List<StoragePathInfo> globEntries(StoragePath pathPattern)
                                                                      throws IOException

Returns all the files that match the pathPattern and are not checksum files.

Parameters:

pathPattern - given pattern.

Returns:

the list of file info of the files.

Throws:

IOException - IO error.