org.gridgain.grid.ggfs.hadoop

Class GridGgfsHadoopParameters

java.lang.Object

org.gridgain.grid.ggfs.hadoop.GridGgfsHadoopParameters

public class GridGgfsHadoopParameters
extends Object

This class lists parameters that can be specified in Hadoop configuration. Hadoop configuration can be specified in core-site.xml file or passed to map-reduce task directly when using Hadoop driver for GGFS file system:

• fs.ggfs.[name].open.sequential_reads_before_prefetch - this parameter overrides the one specified in GridGgfsConfiguration.getSequentialReadsBeforePrefetch() GGFS data node configuration property.
• fs.ggfs.[name].log.enabled - specifies whether GGFS sampling logger is enabled. If true, then all file system operations will be logged to a file.
• fs.ggfs.[name].log.dir - specifies log directory where sampling log files should be placed.
• fs.ggfs.[name].log.batch_size - specifies how many log entries are accumulated in a batch before it gets flushed to log file. Higher values will imply greater performance, but will increase delay before record appears in the log file.
• fs.ggfs.[name].colocated.writes - specifies whether written files should be colocated on data node to which client is connected. If true, file will not be distributed and will be written to a single data node. Default value is true.

Where [name] is file system endpoint which you specify in file system URI authority part. E.g. in case your file system URI is ggfs://127.0.0.1:10500 then name will be 127.0.0.1:10500.

Sample configuration that can be placed to core-site.xml file:

     <property>
         <name>fs.ggfs.127.0.0.1:10500.log.enabled</name>
         <value>true</value>
     </property>
     <property>
         <name>fs.ggfs.127.0.0.1:10500.log.dir</name>
         <value>/home/gridgain/log/sampling</value>
     </property>
     <property>
         <name>fs.ggfs.127.0.0.1:10500.log.batch_size</name>
         <value>16</value>
     </property>
 

Parameters could also be specified per mapreduce job, e.g.

 hadoop jar myjarfile.jar MyMapReduceJob -Dfs.ggfs.open.sequential_reads_before_prefetch=4
 

If you want to use these parameters in code, then you have to substitute you file system name in it. The easiest way to do that is String.format(PARAM_GGFS_COLOCATED_WRITES, [name]).

Field Summary

Fields

Modifier and Type	Field and Description
static String	PARAM_GGFS_COLOCATED_WRITES Parameter name for control over file colocation write mode.
static String	PARAM_GGFS_LOG_BATCH_SIZE Parameter name for log batch size.
static String	PARAM_GGFS_LOG_DIR Parameter name for client logger directory.
static String	PARAM_GGFS_LOG_ENABLED Parameter name for log enabled flag.
static String	PARAM_GGFS_SEQ_READS_BEFORE_PREFETCH Parameter name for custom sequential reads before prefetch value.

Constructor Summary

Constructors

Constructor and Description
GridGgfsHadoopParameters()

Method Summary

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PARAM_GGFS_COLOCATED_WRITES

public static final String PARAM_GGFS_COLOCATED_WRITES

Parameter name for control over file colocation write mode.

See Also:

Constant Field Values

PARAM_GGFS_SEQ_READS_BEFORE_PREFETCH

public static final String PARAM_GGFS_SEQ_READS_BEFORE_PREFETCH

Parameter name for custom sequential reads before prefetch value.

See Also:

Constant Field Values

PARAM_GGFS_LOG_DIR

public static final String PARAM_GGFS_LOG_DIR

Parameter name for client logger directory.

See Also:

Constant Field Values

PARAM_GGFS_LOG_BATCH_SIZE

public static final String PARAM_GGFS_LOG_BATCH_SIZE

Parameter name for log batch size.

See Also:

Constant Field Values

PARAM_GGFS_LOG_ENABLED

public static final String PARAM_GGFS_LOG_ENABLED

Parameter name for log enabled flag.

See Also:

Constant Field Values

Constructor Detail

GridGgfsHadoopParameters

public GridGgfsHadoopParameters()