Package org.apache.pinot.spi.filesystem

Interface PinotFS

All Superinterfaces:

AutoCloseable, Closeable, Serializable

All Known Implementing Classes:

BasePinotFS, LocalPinotFS, NoClosePinotFS

@Public
@Stable
public interface PinotFS
extends Closeable, Serializable

PinotFS is a restricted FS API that exposes functionality that is required for Pinot to use different FS implementations. The restrictions are in place due to 2 driving factors: 1. Prevent unexpected performance hit when a broader API is implemented - especially, we would like to reduce calls to remote filesystems that might be needed for a broader API, but not necessarily required by Pinot (see the documentation for move() method below). 2. Provide an interface that is simple to be implemented across different FS types. The contract that developers have to adhere to will be simpler. Please read the method level docs carefully to note the exceptions while using the APIs.

Method Summary

Modifier and Type	Method	Description
default void	close()	For certain filesystems, we may need to close the filesystem and do relevant operations to prevent leaks.
default boolean	copy(URI srcUri, URI dstUri)	Copies the file from the src to dst.
boolean	copyDir(URI srcUri, URI dstUri)	Copies the file or directory from the src to dst.
default void	copyFromLocalDir(File srcFile, URI dstUri)
void	copyFromLocalFile(File srcFile, URI dstUri)	The src file is on the local disk.
void	copyToLocalFile(URI srcUri, File dstFile)	Copies a file from a remote filesystem to the local one.
boolean	delete(URI segmentUri, boolean forceDelete)	Deletes the file at the location provided.
boolean	exists(URI fileUri)	Checks whether the file or directory at the provided location exists.
void	init(PinotConfiguration config)	Initializes the configurations specific to that filesystem.
boolean	isDirectory(URI uri)	Allows us the ability to determine whether the uri is a directory.
long	lastModified(URI uri)	Returns the age of the file
long	length(URI fileUri)	Returns the length of the file at the provided location.
String[]	listFiles(URI fileUri, boolean recursive)	Lists all the files and directories at the location provided.
default List<FileMetadata>	listFilesWithMetadata(URI fileUri, boolean recursive)	Like listFiles but also return some file metadata so no need to call length(), isDirectory(), and lastModified() separately, which can be slow and costly for remote file system.
boolean	mkdir(URI uri)	Creates a new directory.
boolean	move(URI srcUri, URI dstUri, boolean overwrite)	Moves the file or directory from the src to dst.
InputStream	open(URI uri)	Opens a file in the underlying filesystem and returns an InputStream to read it.
boolean	touch(URI uri)	Updates the last modified time of an existing file or directory to be current time.

Method Detail

init

void init(PinotConfiguration config)

Initializes the configurations specific to that filesystem. For instance, any security related parameters can be initialized here and will not be logged.

mkdir

boolean mkdir(URI uri)
       throws IOException

Creates a new directory. If parent directories are not created, it will create them. If the directory exists, it will return true without doing anything.

Returns:

true if mkdir is successful

Throws:

IOException - on IO failure

delete

boolean delete(URI segmentUri,
               boolean forceDelete)
        throws IOException

Deletes the file at the location provided. If the segmentUri is a directory, it will delete the entire directory.

Parameters:

segmentUri - URI of the segment

forceDelete - true if we want the uri and any sub-uris to always be deleted, false if we want delete to fail when we want to delete a directory and that directory is not empty

Returns:

true if delete is successful else false

Throws:

IOException - on IO failure, e.g Uri is not present or not valid

move

boolean move(URI srcUri,
             URI dstUri,
             boolean overwrite)
      throws IOException

Moves the file or directory from the src to dst. Does not keep the original file. If the dst has parent directories that haven't been created, this method will create all the necessary parent directories. Note: In Pinot we recommend the full paths of both src and dst be specified. For example, if a file /a/b/c is moved to a file /x/y/z, in the case of overwrite, the directory /a/b still exists, but will not contain the file 'c'. Instead, /x/y/z will contain the contents of 'c'. If src is a directory /a/b which contains two files /a/b/c and /a/b/d, and the dst is /x/y, the result would be that the directory /a/b under /a gets removed and dst directory contains two files which is /x/y/c and /x/y/d. If src is a directory /a/b needs to be moved under another directory /x/y, please specify the dst to /x/y/b.

Parameters:

srcUri - URI of the original file

dstUri - URI of the final file location

overwrite - true if we want to overwrite the dstURI, false otherwise

Returns:

true if move is successful

Throws:

IOException - on IO failure

copy

default boolean copy(URI srcUri,
                     URI dstUri)
              throws IOException

Copies the file from the src to dst. The original file is retained. If the dst has parent directories that haven't been created, this method will create all the necessary parent directories. If dst already exists, this will overwrite the existing file in the path. Note: In Pinot we recommend the full paths of both src and dst be specified. For example, if a file /a/b/c is copied to a file /x/y/z, the directory /a/b still exists containing the file 'c'. The dst file /x/y/z will contain the contents of 'c'.

Parameters:

srcUri - URI of the original file

dstUri - URI of the final file location

Returns:

true if copy is successful

Throws:

IOException - on IO failure

copyDir

boolean copyDir(URI srcUri,
                URI dstUri)
         throws IOException

Copies the file or directory from the src to dst. The original file is retained. If the dst has parent directories that haven't been created, this method will create all the necessary parent directories. If dst already exists, this will overwrite the existing file/directory in the path. Note: In Pinot we recommend the full paths of both src and dst be specified. For example, if a file /a/b/c is copied to a file /x/y/z, the directory /a/b still exists containing the file 'c'. The dst file /x/y/z will contain the contents of 'c'. If a directory /a/b is copied to another directory /x/y, the directory /x/y will contain the content of /a/b. If a directory /a/b is copied under the directory /x/y, the dst needs to be specify as /x/y/b.

Parameters:

srcUri - URI of the original file

dstUri - URI of the final file location

Returns:

true if copy is successful

Throws:

IOException - on IO failure

exists

boolean exists(URI fileUri)
        throws IOException

Checks whether the file or directory at the provided location exists.

Parameters:

fileUri - URI of file

Returns:

true if path exists

Throws:

IOException - on IO failure

length

long length(URI fileUri)
     throws IOException

Returns the length of the file at the provided location.

Parameters:

fileUri - location of file

Returns:

the number of bytes

Throws:

IOException - on IO failure, e.g if it's a directory.

listFiles

String[] listFiles(URI fileUri,
                   boolean recursive)
            throws IOException

Lists all the files and directories at the location provided. Lists recursively if recursive is set to true. Throws IOException if this abstract pathname is not valid, or if an I/O error occurs.

Parameters:

fileUri - location of file

recursive - if we want to list files recursively

Returns:

an array of strings that contains file paths

Throws:

IOException - on IO failure. See specific implementation

listFilesWithMetadata

default List<FileMetadata> listFilesWithMetadata(URI fileUri,
                                                 boolean recursive)
                                          throws IOException

Like listFiles but also return some file metadata so no need to call length(), isDirectory(), and lastModified() separately, which can be slow and costly for remote file system.

Parameters:

fileUri - location of file

recursive - if we want to list files recursively

Returns:

a list of FileMetadata that contains file path, and a few file metadata.

Throws:

IOException - on IO failure. See specific implementation

copyToLocalFile

void copyToLocalFile(URI srcUri,
                     File dstFile)
              throws Exception

Copies a file from a remote filesystem to the local one. Keeps the original file.

Parameters:

srcUri - location of current file on remote filesystem (must not be a directory)

dstFile - location of destination on local filesystem

Throws:

Exception - if srcUri is not valid or not present, or timeout when downloading file to local

copyFromLocalDir

default void copyFromLocalDir(File srcFile,
                              URI dstUri)
                       throws Exception

Parameters:

srcFile - location of src file on local disk (must be a directory)

dstUri - location of dst on remote filesystem

Throws:

Exception - if fileUri is not valid or not present, or timeout when uploading file from local

copyFromLocalFile

void copyFromLocalFile(File srcFile,
                       URI dstUri)
                throws Exception

The src file is on the local disk. Add it to filesystem at the given dst name and the source is kept intact afterwards.

Parameters:

srcFile - location of src file on local disk (must not be a directory)

dstUri - location of dst on remote filesystem

Throws:

Exception - if fileUri is not valid or not present, or timeout when uploading file from local

isDirectory

boolean isDirectory(URI uri)
             throws IOException

Allows us the ability to determine whether the uri is a directory.

Parameters:

uri - location of file or directory

Returns:

true if uri is a directory, false otherwise.

Throws:

IOException - on IO failure, e.g uri is not valid or not present

lastModified

long lastModified(URI uri)
           throws IOException

Returns the age of the file

Parameters:

uri - location of file or directory

Returns:

A long value representing the time the file was last modified, measured in milliseconds since epoch (00:00:00 GMT, January 1, 1970) or 0L if the file does not exist or if an I/O error occurs

Throws:

IOException - if uri is not valid or not present

touch

boolean touch(URI uri)
       throws IOException

Updates the last modified time of an existing file or directory to be current time. If the file system object does not exist, creates an empty file.

Parameters:

uri - location of file or directory

Throws:

IOException - if the parent directory doesn't exist

open

InputStream open(URI uri)
          throws IOException

Opens a file in the underlying filesystem and returns an InputStream to read it. Note that the caller can invoke close on this inputstream. Some implementations can choose to copy the original file to local temp file and return the inputstream. In this case, the implementation it should delete the temp file when close is invoked.

Parameters:

uri - location of the file to open

Returns:

a new InputStream

Throws:

IOException - on any IO error - missing file, not a file etc

close

default void close()
            throws IOException

For certain filesystems, we may need to close the filesystem and do relevant operations to prevent leaks. By default, this method does nothing.

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Throws:

IOException - on IO failure