com.wcohen.ss

Class AbstractStatisticalTokenDistance

java.lang.Object

com.wcohen.ss.AbstractStringDistance

com.wcohen.ss.AbstractTokenizedStringDistance

com.wcohen.ss.AbstractStatisticalTokenDistance

All Implemented Interfaces:

StringDistance, StringDistanceLearner

Direct Known Subclasses:

AbbreviationAlignment, Mixture, SoftTokenFelligiSunter, TagLink, TFIDF, TokenFelligiSunter

public abstract class AbstractStatisticalTokenDistance
extends AbstractTokenizedStringDistance

Abstract token distance metric that uses frequency statistics.

Field Summary

Fields

Modifier and Type	Field and Description
protected int	collectionSize
protected Map	documentFrequency
protected int	totalTokenCount

Fields inherited from class com.wcohen.ss.AbstractTokenizedStringDistance

tokenizer

Constructor Summary

Constructors

Constructor and Description
AbstractStatisticalTokenDistance()
AbstractStatisticalTokenDistance(Tokenizer tokenizer)

Method Summary

Modifier and Type	Method and Description
protected void	checkTrainingHasHappened(StringWrapper s, StringWrapper t)
int	getDocumentFrequency(Token tok)
Iterator	tokenIterator()
void	train(StringWrapperIterator i) Accumulate statistics on how often each token value occurs

Methods inherited from class com.wcohen.ss.AbstractTokenizedStringDistance

asBagOfTokens, prepare, setStringWrapperPool

Methods inherited from class com.wcohen.ss.AbstractStringDistance

addExample, doMain, explainScore, explainScore, getDistance, hasNextQuery, nextQuery, prepare, prepare, score, score, setDistanceInstancePool

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

documentFrequency

protected Map documentFrequency

collectionSize

protected int collectionSize

totalTokenCount

protected int totalTokenCount

Constructor Detail

AbstractStatisticalTokenDistance

public AbstractStatisticalTokenDistance(Tokenizer tokenizer)

AbstractStatisticalTokenDistance

public AbstractStatisticalTokenDistance()

Method Detail

train

public void train(StringWrapperIterator i)

Accumulate statistics on how often each token value occurs

Specified by:

train in class AbstractTokenizedStringDistance

checkTrainingHasHappened

protected void checkTrainingHasHappened(StringWrapper s,
                                        StringWrapper t)

getDocumentFrequency

public int getDocumentFrequency(Token tok)

tokenIterator

public Iterator tokenIterator()