edu.umd.cloud9.webgraph.driver

Class ClueWebDriver

java.lang.Object

org.apache.hadoop.conf.Configured

edu.umd.cloud9.webgraph.driver.ClueWebDriver

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, org.apache.hadoop.util.Tool

public class ClueWebDriver
extends org.apache.hadoop.conf.Configured
implements org.apache.hadoop.util.Tool

Main driver program for extracting the web graph, reverse web graph, and lines of anchor text. Command-line arguments are as follows:

• [-input collection-base-path]: the base path to the collection, collection file will be searched recursively
• [-output output-base-path]: the base path under which the output would be stored
• [-docno docno-mapping-file]: the path to the docno-mapping.dat file
• [-begin from-segment]: starting segment number
• [-end to-segment]: ending segment number
• [-il]: include internal links (i.e., links within a domain)
• [-caw]: compute the default weights for lines of external anchor text
• [-normalizer normalizer] A normalizer class used to normalize the lines of anchor text, must extend edu.umd.cloud9.webgraph.normalize.AnchorTextNormalizer.

The starting and ending segments will correspond to paths /[collection-base-path]/en.XX to /[collection-base-path]/en.YY.

The default weight used in this program was originally proposed by Metzler et. al in the following paper:
D. Metzler, J. Novak, H. Cui, and S. Reddy. Building enriched document representations using aggregated anchor text. In Proc. 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 219{226, New York, NY, USA, 2009. ACM.

Author:

Nima Asadi

Constructor Summary

Constructors

Constructor and Description
ClueWebDriver()

Method Summary

Methods

Modifier and Type	Method and Description
static void	main(String[] args)
int	run(String[] args)

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Constructor Detail

ClueWebDriver

public ClueWebDriver()

Method Detail

run

public int run(String[] args)
        throws Exception

Specified by:

run in interface org.apache.hadoop.util.Tool

Throws:

Exception

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception