com.google.cloud.hadoop.fs.gcs

Class GoogleHadoopFS

java.lang.Object

org.apache.hadoop.fs.AbstractFileSystem

com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS

public class GoogleHadoopFS
extends org.apache.hadoop.fs.AbstractFileSystem

GoogleHadoopFS provides a YARN compatible Abstract File System on top of Google Cloud Storage (GCS). It is implemented as a thin abstraction layer on top of GoogleHadoopFileSystem, but will soon be refactored to share a common base.

Field Summary

Fields

Modifier and Type	Field and Description
static LogUtil	log

Fields inherited from class org.apache.hadoop.fs.AbstractFileSystem

statistics

Constructor Summary

Constructors

Constructor and Description
GoogleHadoopFS(GoogleHadoopFileSystem ghfs, URI uri, org.apache.hadoop.conf.Configuration conf)
GoogleHadoopFS(URI uri, org.apache.hadoop.conf.Configuration conf)

Method Summary

Methods

Modifier and Type	Method and Description
void	checkPath(org.apache.hadoop.fs.Path path) Only accept valid AbstractFileSystem and GoogleHadoopFileSystem Paths.
org.apache.hadoop.fs.FSDataOutputStream	createInternal(org.apache.hadoop.fs.Path file, EnumSet<org.apache.hadoop.fs.CreateFlag> flag, org.apache.hadoop.fs.permission.FsPermission absolutePermission, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress, org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt, boolean createParent)
boolean	delete(org.apache.hadoop.fs.Path f, boolean recursive)
org.apache.hadoop.fs.BlockLocation[]	getFileBlockLocations(org.apache.hadoop.fs.Path f, long start, long len)
org.apache.hadoop.fs.FileChecksum	getFileChecksum(org.apache.hadoop.fs.Path f)
org.apache.hadoop.fs.FileStatus	getFileStatus(org.apache.hadoop.fs.Path f)
org.apache.hadoop.fs.FsStatus	getFsStatus()
org.apache.hadoop.fs.FsServerDefaults	getServerDefaults()
URI	getUri() This is overridden to use GoogleHadoopFileSystem's URI, because AbstractFileSystem appends the default port to the authority.
int	getUriDefaultPort()
boolean	isValidName(String src) Follow HDFS conventions except allow for ':' in paths.
org.apache.hadoop.fs.FileStatus[]	listStatus(org.apache.hadoop.fs.Path f)
void	mkdir(org.apache.hadoop.fs.Path dir, org.apache.hadoop.fs.permission.FsPermission permission, boolean createParent)
org.apache.hadoop.fs.FSDataInputStream	open(org.apache.hadoop.fs.Path f, int bufferSize)
void	renameInternal(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst)
void	setOwner(org.apache.hadoop.fs.Path f, String username, String groupname)
void	setPermission(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsPermission permission)
boolean	setReplication(org.apache.hadoop.fs.Path f, short replication)
void	setTimes(org.apache.hadoop.fs.Path f, long mtime, long atime)
void	setVerifyChecksum(boolean verifyChecksum)

Methods inherited from class org.apache.hadoop.fs.AbstractFileSystem

checkScheme, clearStatistics, create, createFileSystem, createSymlink, equals, get, getAclStatus, getAllStatistics, getCanonicalServiceName, getDelegationTokens, getFileLinkStatus, getFsStatus, getHomeDirectory, getInitialWorkingDirectory, getLinkTarget, getStatistics, getStatistics, getUriPath, hashCode, listCorruptFileBlocks, listLocatedStatus, listStatusIterator, makeQualified, modifyAclEntries, open, printStatistics, removeAcl, removeAclEntries, removeDefaultAcl, rename, renameInternal, resolvePath, setAcl, supportsSymlinks

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Field Detail

log

public static final LogUtil log

Constructor Detail

GoogleHadoopFS

public GoogleHadoopFS(URI uri,
              org.apache.hadoop.conf.Configuration conf)
               throws URISyntaxException,
                      IOException

Throws:

URISyntaxException

IOException

GoogleHadoopFS

public GoogleHadoopFS(GoogleHadoopFileSystem ghfs,
              URI uri,
              org.apache.hadoop.conf.Configuration conf)
               throws URISyntaxException,
                      IOException

Throws:

URISyntaxException

IOException

Method Detail

createInternal

public org.apache.hadoop.fs.FSDataOutputStream createInternal(org.apache.hadoop.fs.Path file,
                                                     EnumSet<org.apache.hadoop.fs.CreateFlag> flag,
                                                     org.apache.hadoop.fs.permission.FsPermission absolutePermission,
                                                     int bufferSize,
                                                     short replication,
                                                     long blockSize,
                                                     org.apache.hadoop.util.Progressable progress,
                                                     org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt,
                                                     boolean createParent)
                                                       throws IOException

Specified by:

createInternal in class org.apache.hadoop.fs.AbstractFileSystem

Throws:

IOException

getUriDefaultPort

public int getUriDefaultPort()

Specified by:

getUriDefaultPort in class org.apache.hadoop.fs.AbstractFileSystem

getUri

public URI getUri()

This is overridden to use GoogleHadoopFileSystem's URI, because AbstractFileSystem appends the default port to the authority.

Overrides:

getUri in class org.apache.hadoop.fs.AbstractFileSystem

isValidName

public boolean isValidName(String src)

Follow HDFS conventions except allow for ':' in paths.

Overrides:

isValidName in class org.apache.hadoop.fs.AbstractFileSystem

checkPath

public void checkPath(org.apache.hadoop.fs.Path path)

Only accept valid AbstractFileSystem and GoogleHadoopFileSystem Paths.

Overrides:

checkPath in class org.apache.hadoop.fs.AbstractFileSystem

getServerDefaults

public org.apache.hadoop.fs.FsServerDefaults getServerDefaults()
                                                        throws IOException

Specified by:

getServerDefaults in class org.apache.hadoop.fs.AbstractFileSystem

Throws:

IOException

mkdir

public void mkdir(org.apache.hadoop.fs.Path dir,
         org.apache.hadoop.fs.permission.FsPermission permission,
         boolean createParent)
           throws IOException

Specified by:

mkdir in class org.apache.hadoop.fs.AbstractFileSystem

Throws:

IOException

delete

public boolean delete(org.apache.hadoop.fs.Path f,
             boolean recursive)
               throws IOException

Specified by:

delete in class org.apache.hadoop.fs.AbstractFileSystem

Throws:

IOException

open

public org.apache.hadoop.fs.FSDataInputStream open(org.apache.hadoop.fs.Path f,
                                          int bufferSize)
                                            throws IOException

Specified by:

open in class org.apache.hadoop.fs.AbstractFileSystem

Throws:

IOException

setReplication

public boolean setReplication(org.apache.hadoop.fs.Path f,
                     short replication)
                       throws IOException

Specified by:

setReplication in class org.apache.hadoop.fs.AbstractFileSystem

Throws:

IOException

renameInternal

public void renameInternal(org.apache.hadoop.fs.Path src,
                  org.apache.hadoop.fs.Path dst)
                    throws IOException

Specified by:

renameInternal in class org.apache.hadoop.fs.AbstractFileSystem

Throws:

IOException

setPermission

public void setPermission(org.apache.hadoop.fs.Path f,
                 org.apache.hadoop.fs.permission.FsPermission permission)
                   throws IOException

Specified by:

setPermission in class org.apache.hadoop.fs.AbstractFileSystem

Throws:

IOException

setOwner

public void setOwner(org.apache.hadoop.fs.Path f,
            String username,
            String groupname)
              throws IOException

Specified by:

setOwner in class org.apache.hadoop.fs.AbstractFileSystem

Throws:

IOException

setTimes

public void setTimes(org.apache.hadoop.fs.Path f,
            long mtime,
            long atime)
              throws IOException

Specified by:

setTimes in class org.apache.hadoop.fs.AbstractFileSystem

Throws:

IOException

getFileChecksum

public org.apache.hadoop.fs.FileChecksum getFileChecksum(org.apache.hadoop.fs.Path f)
                                                  throws IOException

Specified by:

getFileChecksum in class org.apache.hadoop.fs.AbstractFileSystem

Throws:

IOException

getFileStatus

public org.apache.hadoop.fs.FileStatus getFileStatus(org.apache.hadoop.fs.Path f)
                                              throws IOException

Specified by:

getFileStatus in class org.apache.hadoop.fs.AbstractFileSystem

Throws:

IOException

getFileBlockLocations

public org.apache.hadoop.fs.BlockLocation[] getFileBlockLocations(org.apache.hadoop.fs.Path f,
                                                         long start,
                                                         long len)
                                                           throws IOException

Specified by:

getFileBlockLocations in class org.apache.hadoop.fs.AbstractFileSystem

Throws:

IOException

getFsStatus

public org.apache.hadoop.fs.FsStatus getFsStatus()
                                          throws IOException

Specified by:

getFsStatus in class org.apache.hadoop.fs.AbstractFileSystem

Throws:

IOException

listStatus

public org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.fs.Path f)
                                             throws IOException

Specified by:

listStatus in class org.apache.hadoop.fs.AbstractFileSystem

Throws:

IOException

setVerifyChecksum

public void setVerifyChecksum(boolean verifyChecksum)

Specified by:

setVerifyChecksum in class org.apache.hadoop.fs.AbstractFileSystem