org.gridgain.grid.ggfs.hadoop.v2

Class GridGgfsHadoopFileSystem

java.lang.Object

org.apache.hadoop.fs.AbstractFileSystem

org.gridgain.grid.ggfs.hadoop.v2.GridGgfsHadoopFileSystem

All Implemented Interfaces:

Closeable, AutoCloseable

public class GridGgfsHadoopFileSystem
extends org.apache.hadoop.fs.AbstractFileSystem
implements Closeable

GGFS Hadoop 2.x file system driver over file system API. To use GGFS as Hadoop file system, you should configure this class in Hadoop's core-site.xml as follows:

  <property>
      <name>fs.default.name</name>
      <value>ggfs://ipc</value>
  </property>

  <property>
      <name>fs.ggfs.impl</name>
      <value>org.gridgain.grid.ggfs.hadoop.GridGgfsHadoopFileSystem</value>
  </property>
 

You should also add GridGain JAR and all libraries to Hadoop classpath. To do this, add following lines to conf/hadoop-env.sh script in Hadoop distribution:

 export GRIDGAIN_HOME=/path/to/GridGain/distribution
 export HADOOP_CLASSPATH=$GRIDGAIN_HOME/gridgain*.jar

 for f in $GRIDGAIN_HOME/libs/*.jar; do
  export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f;
 done
 

Data vs Clients Nodes

Hadoop needs to use its FileSystem remotely from client nodes as well as directly on data nodes. Client nodes are responsible for basic file system operations as well as accessing data nodes remotely. Usually, client nodes are started together with job-submitter or job-scheduler processes, while data nodes are usually started together with Hadoop task-tracker processes.

For sample client and data node configuration refer to config/hadoop/default-config-client.xml and config/hadoop/default-config.xml configuration files in GridGain installation.

Field Summary

Fields inherited from class org.apache.hadoop.fs.AbstractFileSystem

statistics

Constructor Summary

Constructors

Constructor and Description
GridGgfsHadoopFileSystem(URI name, org.apache.hadoop.conf.Configuration cfg)

Method Summary

Methods

Modifier and Type	Method and Description
void	checkPath(org.apache.hadoop.fs.Path path)
void	close()
void	colocateFileWrites(boolean colocateFileWrites) Public setter that can be used by direct users of FS or Visor.
org.apache.hadoop.fs.FSDataOutputStream	createInternal(org.apache.hadoop.fs.Path f, EnumSet<org.apache.hadoop.fs.CreateFlag> flag, org.apache.hadoop.fs.permission.FsPermission perm, int bufSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress, org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt, boolean createParent)
boolean	delete(org.apache.hadoop.fs.Path f, boolean recursive)
protected void	finalize()
org.apache.hadoop.fs.BlockLocation[]	getFileBlockLocations(org.apache.hadoop.fs.Path path, long start, long len)
org.apache.hadoop.fs.FileChecksum	getFileChecksum(org.apache.hadoop.fs.Path f)
org.apache.hadoop.fs.FileStatus	getFileStatus(org.apache.hadoop.fs.Path f)
org.apache.hadoop.fs.FsStatus	getFsStatus()
org.apache.hadoop.fs.FsServerDefaults	getServerDefaults()
URI	getUri()
int	getUriDefaultPort()
org.apache.hadoop.fs.FileStatus[]	listStatus(org.apache.hadoop.fs.Path f)
void	mkdir(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsPermission perm, boolean createParent)
org.gridgain.grid.ggfs.GridGgfsMode	mode(org.apache.hadoop.fs.Path path) Resolve path mode.
org.apache.hadoop.fs.FSDataInputStream	open(org.apache.hadoop.fs.Path f, int bufSize)
void	renameInternal(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst)
void	setOwner(org.apache.hadoop.fs.Path p, String usr, String grp)
void	setPermission(org.apache.hadoop.fs.Path p, org.apache.hadoop.fs.permission.FsPermission perm)
boolean	setReplication(org.apache.hadoop.fs.Path f, short replication)
void	setTimes(org.apache.hadoop.fs.Path f, long mtime, long atime)
void	setVerifyChecksum(boolean verifyChecksum)
boolean	supportsSymlinks()
String	toString()

Methods inherited from class org.apache.hadoop.fs.AbstractFileSystem

checkScheme, clearStatistics, create, createFileSystem, createSymlink, equals, get, getAclStatus, getAllStatistics, getCanonicalServiceName, getDelegationTokens, getFileLinkStatus, getFsStatus, getHomeDirectory, getInitialWorkingDirectory, getLinkTarget, getStatistics, getStatistics, getUriPath, hashCode, isValidName, listCorruptFileBlocks, listLocatedStatus, listStatusIterator, makeQualified, modifyAclEntries, open, printStatistics, removeAcl, removeAclEntries, removeDefaultAcl, rename, renameInternal, resolvePath, setAcl

Methods inherited from class java.lang.Object

clone, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

GridGgfsHadoopFileSystem

public GridGgfsHadoopFileSystem(URI name,
                        org.apache.hadoop.conf.Configuration cfg)
                         throws URISyntaxException,
                                IOException

Parameters:

name - URI for file system.

cfg - Configuration.

Throws:

URISyntaxException - if name has invalid syntax.

IOException - If initialization failed.

Method Detail

checkPath

public void checkPath(org.apache.hadoop.fs.Path path)

Overrides:

checkPath in class org.apache.hadoop.fs.AbstractFileSystem

colocateFileWrites

public void colocateFileWrites(boolean colocateFileWrites)

Public setter that can be used by direct users of FS or Visor.

Parameters:

colocateFileWrites - Whether all ongoing file writes should be colocated.

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

finalize

protected void finalize()
                 throws Throwable

Overrides:

finalize in class Object

Throws:

Throwable

getUri

public URI getUri()

Overrides:

getUri in class org.apache.hadoop.fs.AbstractFileSystem

getUriDefaultPort

public int getUriDefaultPort()

Specified by:

getUriDefaultPort in class org.apache.hadoop.fs.AbstractFileSystem

getServerDefaults

public org.apache.hadoop.fs.FsServerDefaults getServerDefaults()
                                                        throws IOException

Specified by:

getServerDefaults in class org.apache.hadoop.fs.AbstractFileSystem

Throws:

IOException

setReplication

public boolean setReplication(org.apache.hadoop.fs.Path f,
                     short replication)
                       throws IOException

Specified by:

setReplication in class org.apache.hadoop.fs.AbstractFileSystem

Throws:

IOException

setTimes

public void setTimes(org.apache.hadoop.fs.Path f,
            long mtime,
            long atime)
              throws IOException

Specified by:

setTimes in class org.apache.hadoop.fs.AbstractFileSystem

Throws:

IOException

getFsStatus

public org.apache.hadoop.fs.FsStatus getFsStatus()
                                          throws IOException

Specified by:

getFsStatus in class org.apache.hadoop.fs.AbstractFileSystem

Throws:

IOException

setPermission

public void setPermission(org.apache.hadoop.fs.Path p,
                 org.apache.hadoop.fs.permission.FsPermission perm)
                   throws IOException

Specified by:

setPermission in class org.apache.hadoop.fs.AbstractFileSystem

Throws:

IOException

setOwner

public void setOwner(org.apache.hadoop.fs.Path p,
            String usr,
            String grp)
              throws IOException

Specified by:

setOwner in class org.apache.hadoop.fs.AbstractFileSystem

Throws:

IOException

open

public org.apache.hadoop.fs.FSDataInputStream open(org.apache.hadoop.fs.Path f,
                                          int bufSize)
                                            throws IOException

Specified by:

open in class org.apache.hadoop.fs.AbstractFileSystem

Throws:

IOException

createInternal

public org.apache.hadoop.fs.FSDataOutputStream createInternal(org.apache.hadoop.fs.Path f,
                                                     EnumSet<org.apache.hadoop.fs.CreateFlag> flag,
                                                     org.apache.hadoop.fs.permission.FsPermission perm,
                                                     int bufSize,
                                                     short replication,
                                                     long blockSize,
                                                     org.apache.hadoop.util.Progressable progress,
                                                     org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt,
                                                     boolean createParent)
                                                       throws IOException

Specified by:

createInternal in class org.apache.hadoop.fs.AbstractFileSystem

Throws:

IOException

supportsSymlinks

public boolean supportsSymlinks()

Overrides:

supportsSymlinks in class org.apache.hadoop.fs.AbstractFileSystem

renameInternal

public void renameInternal(org.apache.hadoop.fs.Path src,
                  org.apache.hadoop.fs.Path dst)
                    throws IOException

Specified by:

renameInternal in class org.apache.hadoop.fs.AbstractFileSystem

Throws:

IOException

delete

public boolean delete(org.apache.hadoop.fs.Path f,
             boolean recursive)
               throws IOException

Specified by:

delete in class org.apache.hadoop.fs.AbstractFileSystem

Throws:

IOException

setVerifyChecksum

public void setVerifyChecksum(boolean verifyChecksum)
                       throws IOException

Specified by:

setVerifyChecksum in class org.apache.hadoop.fs.AbstractFileSystem

Throws:

IOException

getFileChecksum

public org.apache.hadoop.fs.FileChecksum getFileChecksum(org.apache.hadoop.fs.Path f)
                                                  throws IOException

Specified by:

getFileChecksum in class org.apache.hadoop.fs.AbstractFileSystem

Throws:

IOException

listStatus

public org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.fs.Path f)
                                             throws IOException

Specified by:

listStatus in class org.apache.hadoop.fs.AbstractFileSystem

Throws:

IOException

mkdir

public void mkdir(org.apache.hadoop.fs.Path f,
         org.apache.hadoop.fs.permission.FsPermission perm,
         boolean createParent)
           throws IOException

Specified by:

mkdir in class org.apache.hadoop.fs.AbstractFileSystem

Throws:

IOException

getFileStatus

public org.apache.hadoop.fs.FileStatus getFileStatus(org.apache.hadoop.fs.Path f)
                                              throws IOException

Specified by:

getFileStatus in class org.apache.hadoop.fs.AbstractFileSystem

Throws:

IOException

getFileBlockLocations

public org.apache.hadoop.fs.BlockLocation[] getFileBlockLocations(org.apache.hadoop.fs.Path path,
                                                         long start,
                                                         long len)
                                                           throws IOException

Specified by:

getFileBlockLocations in class org.apache.hadoop.fs.AbstractFileSystem

Throws:

IOException

mode

public org.gridgain.grid.ggfs.GridGgfsMode mode(org.apache.hadoop.fs.Path path)

Resolve path mode.

Parameters:

path - HDFS path.

Returns:

Path mode.

toString

public String toString()

Overrides:

toString in class Object