org.gridgain.grid.ggfs.hadoop.v1

Class GridGgfsHadoopFileSystem

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.FileSystem

org.gridgain.grid.ggfs.hadoop.v1.GridGgfsHadoopFileSystem

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable

public class GridGgfsHadoopFileSystem
extends org.apache.hadoop.fs.FileSystem

GGFS Hadoop 1.x file system driver over file system API. To use GGFS as Hadoop file system, you should configure this class in Hadoop's core-site.xml as follows:

  <property>
      <name>fs.default.name</name>
      <value>ggfs://ipc</value>
  </property>

  <property>
      <name>fs.ggfs.impl</name>
      <value>org.gridgain.grid.ggfs.hadoop.GridGgfsHadoopFileSystem</value>
  </property>
 

You should also add GridGain JAR and all libraries to Hadoop classpath. To do this, add following lines to conf/hadoop-env.sh script in Hadoop distribution:

 export GRIDGAIN_HOME=/path/to/GridGain/distribution
 export HADOOP_CLASSPATH=$GRIDGAIN_HOME/gridgain*.jar

 for f in $GRIDGAIN_HOME/libs/*.jar; do
  export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f;
 done
 

Data vs Clients Nodes

Hadoop needs to use its FileSystem remotely from client nodes as well as directly on data nodes. Client nodes are responsible for basic file system operations as well as accessing data nodes remotely. Usually, client nodes are started together with job-submitter or job-scheduler processes, while data nodes are usually started together with Hadoop task-tracker processes.

For sample client and data node configuration refer to config/hadoop/default-config-client.xml and config/hadoop/default-config.xml configuration files in GridGain installation.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.FileSystem.Statistics

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.hadoop.fs.FileStatus[]	EMPTY_FILE_STATUS Empty array of file statuses.
static String	GGFS_MANAGEMENT Internal property to indicate management connection.

Fields inherited from class org.apache.hadoop.fs.FileSystem

DEFAULT_FS, FS_DEFAULT_NAME_KEY, LOG, SHUTDOWN_HOOK_PRIORITY, statistics

Constructor Summary

Constructors

Constructor and Description
GridGgfsHadoopFileSystem()

Method Summary

Methods

Modifier and Type	Method and Description
org.apache.hadoop.fs.FSDataOutputStream	append(org.apache.hadoop.fs.Path f, int bufSize, org.apache.hadoop.util.Progressable progress)
protected void	checkPath(org.apache.hadoop.fs.Path path)
void	close()
void	colocateFileWrites(boolean colocateFileWrites) Public setter that can be used by direct users of FS or Visor.
org.apache.hadoop.fs.FSDataOutputStream	create(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsPermission perm, boolean overwrite, int bufSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress)
boolean	delete(org.apache.hadoop.fs.Path f)
boolean	delete(org.apache.hadoop.fs.Path f, boolean recursive)
protected void	finalize()
org.apache.hadoop.fs.ContentSummary	getContentSummary(org.apache.hadoop.fs.Path f)
long	getDefaultBlockSize()
short	getDefaultReplication()
org.apache.hadoop.fs.BlockLocation[]	getFileBlockLocations(org.apache.hadoop.fs.FileStatus status, long start, long len)
org.apache.hadoop.fs.FileStatus	getFileStatus(org.apache.hadoop.fs.Path f)
URI	getUri()
org.apache.hadoop.fs.Path	getWorkingDirectory()
void	initialize(URI name, org.apache.hadoop.conf.Configuration cfg)
org.apache.hadoop.fs.FileStatus[]	listStatus(org.apache.hadoop.fs.Path f)
boolean	mkdirs(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsPermission perm)
org.gridgain.grid.ggfs.GridGgfsMode	mode(org.gridgain.grid.ggfs.GridGgfsPath path) Resolve path mode.
org.gridgain.grid.ggfs.GridGgfsMode	mode(org.apache.hadoop.fs.Path path) Resolve path mode.
org.apache.hadoop.fs.FSDataInputStream	open(org.apache.hadoop.fs.Path f, int bufSize)
boolean	rename(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst)
void	setOwner(org.apache.hadoop.fs.Path p, String username, String grpName)
void	setPermission(org.apache.hadoop.fs.Path p, org.apache.hadoop.fs.permission.FsPermission perm)
void	setTimes(org.apache.hadoop.fs.Path p, long mtime, long atime)
void	setWorkingDirectory(org.apache.hadoop.fs.Path newPath)
String	toString()

Methods inherited from class org.apache.hadoop.fs.FileSystem

addDelegationTokens, append, append, areSymlinksEnabled, cancelDeleteOnExit, canonicalizeUri, clearStatistics, closeAll, closeAllForUGI, completeLocalOutput, concat, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, create, createNewFile, createNonRecursive, createNonRecursive, createNonRecursive, createSnapshot, createSnapshot, createSymlink, deleteOnExit, deleteSnapshot, enableSymlinks, exists, fixRelativePart, get, get, get, getAclStatus, getAllStatistics, getBlockSize, getCanonicalServiceName, getCanonicalUri, getChildFileSystems, getDefaultBlockSize, getDefaultPort, getDefaultReplication, getDefaultUri, getDelegationToken, getFileBlockLocations, getFileChecksum, getFileLinkStatus, getFileSystemClass, getFSofPath, getHomeDirectory, getInitialWorkingDirectory, getLength, getLinkTarget, getLocal, getName, getNamed, getReplication, getScheme, getServerDefaults, getServerDefaults, getStatistics, getStatistics, getStatus, getStatus, getUsed, globStatus, globStatus, isDirectory, isFile, listCorruptFileBlocks, listFiles, listLocatedStatus, listLocatedStatus, listStatus, listStatus, listStatus, makeQualified, mkdirs, mkdirs, modifyAclEntries, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, newInstance, newInstance, newInstance, newInstanceLocal, open, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, processDeleteOnExit, removeAcl, removeAclEntries, removeDefaultAcl, rename, renameSnapshot, resolveLink, resolvePath, setAcl, setDefaultUri, setDefaultUri, setReplication, setVerifyChecksum, setWriteChecksum, startLocalOutput, supportsSymlinks

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

GGFS_MANAGEMENT

public static final String GGFS_MANAGEMENT

Internal property to indicate management connection.

See Also:

Constant Field Values

EMPTY_FILE_STATUS

public static final org.apache.hadoop.fs.FileStatus[] EMPTY_FILE_STATUS

Empty array of file statuses.

Constructor Detail

GridGgfsHadoopFileSystem

public GridGgfsHadoopFileSystem()

Method Detail

getUri

public URI getUri()

Specified by:

getUri in class org.apache.hadoop.fs.FileSystem

colocateFileWrites

public void colocateFileWrites(boolean colocateFileWrites)

Public setter that can be used by direct users of FS or Visor.

Parameters:

colocateFileWrites - Whether all ongoing file writes should be colocated.

initialize

public void initialize(URI name,
              org.apache.hadoop.conf.Configuration cfg)
                throws IOException

Overrides:

initialize in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

checkPath

protected void checkPath(org.apache.hadoop.fs.Path path)

Overrides:

checkPath in class org.apache.hadoop.fs.FileSystem

getDefaultReplication

public short getDefaultReplication()

Overrides:

getDefaultReplication in class org.apache.hadoop.fs.FileSystem

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

finalize

protected void finalize()
                 throws Throwable

Overrides:

finalize in class Object

Throws:

Throwable

setTimes

public void setTimes(org.apache.hadoop.fs.Path p,
            long mtime,
            long atime)
              throws IOException

Overrides:

setTimes in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

setPermission

public void setPermission(org.apache.hadoop.fs.Path p,
                 org.apache.hadoop.fs.permission.FsPermission perm)
                   throws IOException

Overrides:

setPermission in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

setOwner

public void setOwner(org.apache.hadoop.fs.Path p,
            String username,
            String grpName)
              throws IOException

Overrides:

setOwner in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

open

public org.apache.hadoop.fs.FSDataInputStream open(org.apache.hadoop.fs.Path f,
                                          int bufSize)
                                            throws IOException

Specified by:

open in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

create

public org.apache.hadoop.fs.FSDataOutputStream create(org.apache.hadoop.fs.Path f,
                                             org.apache.hadoop.fs.permission.FsPermission perm,
                                             boolean overwrite,
                                             int bufSize,
                                             short replication,
                                             long blockSize,
                                             org.apache.hadoop.util.Progressable progress)
                                               throws IOException

Specified by:

create in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

append

public org.apache.hadoop.fs.FSDataOutputStream append(org.apache.hadoop.fs.Path f,
                                             int bufSize,
                                             org.apache.hadoop.util.Progressable progress)
                                               throws IOException

Specified by:

append in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

rename

public boolean rename(org.apache.hadoop.fs.Path src,
             org.apache.hadoop.fs.Path dst)
               throws IOException

Specified by:

rename in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

delete

public boolean delete(org.apache.hadoop.fs.Path f)
               throws IOException

Overrides:

delete in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

delete

public boolean delete(org.apache.hadoop.fs.Path f,
             boolean recursive)
               throws IOException

Specified by:

delete in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

listStatus

public org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.fs.Path f)
                                             throws IOException

Specified by:

listStatus in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

setWorkingDirectory

public void setWorkingDirectory(org.apache.hadoop.fs.Path newPath)

Specified by:

setWorkingDirectory in class org.apache.hadoop.fs.FileSystem

getWorkingDirectory

public org.apache.hadoop.fs.Path getWorkingDirectory()

Specified by:

getWorkingDirectory in class org.apache.hadoop.fs.FileSystem

mkdirs

public boolean mkdirs(org.apache.hadoop.fs.Path f,
             org.apache.hadoop.fs.permission.FsPermission perm)
               throws IOException

Specified by:

mkdirs in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

getFileStatus

public org.apache.hadoop.fs.FileStatus getFileStatus(org.apache.hadoop.fs.Path f)
                                              throws IOException

Specified by:

getFileStatus in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

getContentSummary

public org.apache.hadoop.fs.ContentSummary getContentSummary(org.apache.hadoop.fs.Path f)
                                                      throws IOException

Overrides:

getContentSummary in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

getFileBlockLocations

public org.apache.hadoop.fs.BlockLocation[] getFileBlockLocations(org.apache.hadoop.fs.FileStatus status,
                                                         long start,
                                                         long len)
                                                           throws IOException

Overrides:

getFileBlockLocations in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

getDefaultBlockSize

public long getDefaultBlockSize()

Overrides:

getDefaultBlockSize in class org.apache.hadoop.fs.FileSystem

mode

public org.gridgain.grid.ggfs.GridGgfsMode mode(org.apache.hadoop.fs.Path path)

Resolve path mode.

Parameters:

path - HDFS path.

Returns:

Path mode.

mode

public org.gridgain.grid.ggfs.GridGgfsMode mode(org.gridgain.grid.ggfs.GridGgfsPath path)

Resolve path mode.

Parameters:

path - GGFS path.

Returns:

Path mode.

toString

public String toString()

Overrides:

toString in class Object