edu.umd.cloud9.collection

Class ExtractHTMLFieldCollection

java.lang.Object

org.apache.hadoop.conf.Configured

edu.umd.cloud9.util.PowerTool

edu.umd.cloud9.collection.ExtractHTMLFieldCollection

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, org.apache.hadoop.util.Tool

public class ExtractHTMLFieldCollection
extends PowerTool

Tool for generating 'per-field' collections from HTML documents. The output of this tool is a new collection, in TREC format (in the form of a SequenceFile) that only consists of the text contained within the target tag. This is useful for various document structure and/or field-based retrieval tasks.

Author:

fangyue, metzler

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ExtractHTMLFieldCollection.MyMapper

Field Summary

Fields

Modifier and Type	Field and Description
static String[]	RequiredParameters

Constructor Summary

Constructors

Constructor and Description
ExtractHTMLFieldCollection(org.apache.hadoop.conf.Configuration conf)

Method Summary

Methods

Modifier and Type	Method and Description
String[]	getRequiredParameters()
static void	main(String[] args)
static void	recursivelyAddInputPaths(org.apache.hadoop.mapreduce.Job job, String path)
int	runTool()

Methods inherited from class edu.umd.cloud9.util.PowerTool

run, run

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Field Detail

RequiredParameters

public static final String[] RequiredParameters

Constructor Detail

ExtractHTMLFieldCollection

public ExtractHTMLFieldCollection(org.apache.hadoop.conf.Configuration conf)

Method Detail

getRequiredParameters

public String[] getRequiredParameters()

Specified by:

getRequiredParameters in class PowerTool

runTool

public int runTool()
            throws Exception

Specified by:

runTool in class PowerTool

Throws:

Exception

recursivelyAddInputPaths

public static void recursivelyAddInputPaths(org.apache.hadoop.mapreduce.Job job,
                            String path)
                                     throws IOException

Throws:

IOException

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception