tachyon.underfs

Class UnderFileSystem

java.lang.Object

tachyon.underfs.UnderFileSystem

public abstract class UnderFileSystem
extends Object

Tachyon stores data into an under layer file system. Any file system implementing this interface can be a valid under layer file system

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	UnderFileSystem.SpaceType
static class	UnderFileSystem.UnderFSType Type of under filesystem, to be used by getUnderFSType() to determine which concrete under filesystem implementation is being used.

Field Summary

Fields

Modifier and Type	Field and Description
protected TachyonConf	mTachyonConf

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	UnderFileSystem(TachyonConf tachyonConf)

Method Summary

Methods

Modifier and Type	Method and Description
abstract void	close() Closes this under file system.
abstract void	connectFromMaster(TachyonConf conf, String hostname) Takes any necessary actions required to establish a connection to the under file system from the given master host e.g.
abstract void	connectFromWorker(TachyonConf conf, String hostname) Takes any necessary actions required to establish a connection to the under file system from the given worker host e.g.
abstract OutputStream	create(String path) Creates a file in the under file system with the indicated name.
abstract OutputStream	create(String path, int blockSizeByte) Creates a file in the under file system with the indicated name and block size.
abstract OutputStream	create(String path, short replication, int blockSizeByte) Creates a file in the under file system with the indicated name, replication number and block size.
abstract boolean	delete(String path, boolean recursive) Deletes a file or folder from the under file system with the indicated name.
abstract boolean	exists(String path) Checks if a file or folder exists in under file system.
static UnderFileSystem	get(String path, Object ufsConf, TachyonConf tachyonConf) Gets the UnderFileSystem instance according to its scheme and configuration.
static UnderFileSystem	get(String path, TachyonConf tachyonConf) Gets the UnderFileSystem instance according to its schema.
abstract long	getBlockSizeByte(String path) Gets the block size of a file in under file system, in bytes.
abstract Object	getConf() Gets the configuration object for UnderFileSystem.
abstract List<String>	getFileLocations(String path) Gets the list of locations of the indicated path.
abstract List<String>	getFileLocations(String path, long offset) Gets the list of locations of the indicated path given its offset.
abstract long	getFileSize(String path) Gets the file size in bytes.
abstract long	getModificationTimeMs(String path) Gets the UTC time of when the indicated path was modified recently in ms.
abstract long	getSpace(String path, UnderFileSystem.SpaceType type) Queries the under file system about the space of the indicated path (e.g., space left, space used and etc).
abstract UnderFileSystem.UnderFSType	getUnderFSType()
abstract boolean	isFile(String path) Checks if the indicated path is a file or not.
static boolean	isHadoopUnderFS(String path, TachyonConf tachyonConf) Determines if the given path is on a Hadoop under file system To decide if a path should use the hadoop implementation, we check String.startsWith(String) to see if the configured schemas are found.
abstract String[]	list(String path) Returns an array of strings naming the files and directories in the directory denoted by this abstract pathname.
abstract boolean	mkdirs(String path, boolean createParent) Creates the directory named by this abstract pathname.
abstract InputStream	open(String path) Opens an InputStream at the indicated path.
static Pair<String,String>	parse(TachyonURI path, TachyonConf tachyonConf) Transform an input string like hdfs://host:port/dir, hdfs://host:port, file:///dir, /dir into a pair of address and path.
boolean	providesStorage() Checks whether the underFS provides storage
abstract boolean	rename(String src, String dst) Renames a file or folder from src to dst in under file system
abstract void	setConf(Object conf) Sets the configuration object for UnderFileSystem.
abstract void	setPermission(String path, String posixPerm) Changes posix file permission

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

mTachyonConf

protected final TachyonConf mTachyonConf

Constructor Detail

UnderFileSystem

protected UnderFileSystem(TachyonConf tachyonConf)

Method Detail

get

public static UnderFileSystem get(String path,
                  TachyonConf tachyonConf)

Gets the UnderFileSystem instance according to its schema.

Parameters:

path - file path storing over the ufs.

tachyonConf - the TachyonConf instance.

Returns:

instance of the under layer file system

Throws:

IllegalArgumentException - for unknown scheme

get

public static UnderFileSystem get(String path,
                  Object ufsConf,
                  TachyonConf tachyonConf)

Gets the UnderFileSystem instance according to its scheme and configuration.

Parameters:

path - file path storing over the ufs

ufsConf - the configuration object for ufs only

tachyonConf - the TachyonConf instance.

Returns:

instance of the under layer file system

Throws:

IllegalArgumentException - for unknown scheme

getUnderFSType

public abstract UnderFileSystem.UnderFSType getUnderFSType()

Returns:

type of concrete under filesystem implementation

isHadoopUnderFS

public static boolean isHadoopUnderFS(String path,
                      TachyonConf tachyonConf)

Determines if the given path is on a Hadoop under file system To decide if a path should use the hadoop implementation, we check String.startsWith(String) to see if the configured schemas are found.

providesStorage

public boolean providesStorage()

Checks whether the underFS provides storage

parse

public static Pair<String,String> parse(TachyonURI path,
                        TachyonConf tachyonConf)

Transform an input string like hdfs://host:port/dir, hdfs://host:port, file:///dir, /dir into a pair of address and path. The returned pairs are ("hdfs://host:port", "/dir"), ("hdfs://host:port", "/"), and ("/", "/dir"), respectively.

Parameters:

path - the input path string

Returns:

null if path does not start with tachyon://, tachyon-ft://, hdfs://, s3://, s3n://, file://, /. Or a pair of strings denoting the under FS address and the relative path relative to that address. For local FS (with prefixes file:// or /), the under FS address is "/" and the path starts with "/".

connectFromMaster

public abstract void connectFromMaster(TachyonConf conf,
                     String hostname)
                                throws IOException

Takes any necessary actions required to establish a connection to the under file system from the given master host e.g. logging in

Depending on the implementation this may be a no-op

Parameters:

conf - Tachyon configuration

hostname - The host that wants to connect to the under file system

Throws:

IOException

connectFromWorker

public abstract void connectFromWorker(TachyonConf conf,
                     String hostname)
                                throws IOException

Takes any necessary actions required to establish a connection to the under file system from the given worker host e.g. logging in

Depending on the implementation this may be a no-op

Parameters:

conf - Tachyon configuration

hostname - The host that wants to connect to the under file system

Throws:

IOException

close

public abstract void close()
                    throws IOException

Closes this under file system.

Throws:

IOException

create

public abstract OutputStream create(String path)
                             throws IOException

Creates a file in the under file system with the indicated name.

Parameters:

path - The file name

Returns:

A OutputStream object

Throws:

IOException

create

public abstract OutputStream create(String path,
                  int blockSizeByte)
                             throws IOException

Creates a file in the under file system with the indicated name and block size.

Parameters:

path - The file name

blockSizeByte - The block size in bytes

Returns:

A OutputStream object

Throws:

IOException

create

public abstract OutputStream create(String path,
                  short replication,
                  int blockSizeByte)
                             throws IOException

Creates a file in the under file system with the indicated name, replication number and block size.

Parameters:

path - The file name

replication - The number of replications for each block

blockSizeByte - The block size in bytes

Returns:

A OutputStream object

Throws:

IOException

delete

public abstract boolean delete(String path,
             boolean recursive)
                        throws IOException

Deletes a file or folder from the under file system with the indicated name.

Parameters:

path - The file or folder name

recursive - Whether we delete folder and its children

Returns:

true if succeed, false otherwise

Throws:

IOException

exists

public abstract boolean exists(String path)
                        throws IOException

Checks if a file or folder exists in under file system.

Parameters:

path - The file name

Returns:

true if succeed, false otherwise

Throws:

IOException

getBlockSizeByte

public abstract long getBlockSizeByte(String path)
                               throws IOException

Gets the block size of a file in under file system, in bytes.

Parameters:

path - The file name

Returns:

file size in bytes

Throws:

IOException

getConf

public abstract Object getConf()

Gets the configuration object for UnderFileSystem.

Returns:

configuration object used for concrete ufs instance

getFileLocations

public abstract List<String> getFileLocations(String path)
                                       throws IOException

Gets the list of locations of the indicated path.

Parameters:

path - The file name

Returns:

The list of locations

Throws:

IOException

getFileLocations

public abstract List<String> getFileLocations(String path,
                            long offset)
                                       throws IOException

Gets the list of locations of the indicated path given its offset.

Parameters:

path - The file name

offset - Offset in bytes

Returns:

The list of locations

Throws:

IOException

getFileSize

public abstract long getFileSize(String path)
                          throws IOException

Gets the file size in bytes.

Parameters:

path - The file name

Returns:

the file size in bytes

Throws:

IOException

getModificationTimeMs

public abstract long getModificationTimeMs(String path)
                                    throws IOException

Gets the UTC time of when the indicated path was modified recently in ms.

Parameters:

path - The file or folder name

Returns:

modification time in milliseconds

Throws:

IOException

getSpace

public abstract long getSpace(String path,
            UnderFileSystem.SpaceType type)
                       throws IOException

Queries the under file system about the space of the indicated path (e.g., space left, space used and etc).

Parameters:

path - The path to query

type - The type of queries

Returns:

The space in bytes

Throws:

IOException

isFile

public abstract boolean isFile(String path)
                        throws IOException

Checks if the indicated path is a file or not.

Parameters:

path - The path name

Returns:

true if this is a file, false otherwise

Throws:

IOException

list

public abstract String[] list(String path)
                       throws IOException

Returns an array of strings naming the files and directories in the directory denoted by this abstract pathname.

If this abstract pathname does not denote a directory, then this method returns null. Otherwise an array of strings is returned, one for each file or directory in the directory. Names denoting the directory itself and the directory's parent directory are not included in the result. Each string is a file name rather than a complete path.

There is no guarantee that the name strings in the resulting array will appear in any specific order; they are not, in particular, guaranteed to appear in alphabetical order.

Parameters:

path - the path to list.

Returns:

An array of strings naming the files and directories in the directory denoted by this abstract pathname. The array will be empty if the directory is empty. Returns null if this abstract pathname does not denote a directory, or if an I/O error occurs.

Throws:

IOException

mkdirs

public abstract boolean mkdirs(String path,
             boolean createParent)
                        throws IOException

Creates the directory named by this abstract pathname. If the folder already exists, the method returns false.

Parameters:

path - the folder to create

createParent - If true, the method creates any necessary but nonexistent parent directories. Otherwise, the method does not create nonexistent parent directories.

Returns:

true if and only if the directory was created; false otherwise

Throws:

IOException

open

public abstract InputStream open(String path)
                          throws IOException

Opens an InputStream at the indicated path.

Parameters:

path - The file name

Returns:

The InputStream object

Throws:

IOException

rename

public abstract boolean rename(String src,
             String dst)
                        throws IOException

Renames a file or folder from src to dst in under file system

Parameters:

src - The source file or folder name

dst - The destination file or folder name

Returns:

true if succeed, false otherwise

Throws:

IOException

setConf

public abstract void setConf(Object conf)

Sets the configuration object for UnderFileSystem. The conf object is understood by the concrete underfs's implementation.

Parameters:

conf - The configuration object accepted by ufs.

setPermission

public abstract void setPermission(String path,
                 String posixPerm)
                            throws IOException

Changes posix file permission

Parameters:

path - path of the file

posixPerm - standard posix permission like "777", "775", etc.

Throws:

IOException