com.wcohen.ss

Class SourcedSoftTFIDF

java.lang.Object

com.wcohen.ss.AbstractStringDistance

com.wcohen.ss.AbstractSourcedTokenizedStringDistance

com.wcohen.ss.AbstractSourcedStatisticalTokenDistance

com.wcohen.ss.SourcedTFIDF

com.wcohen.ss.SourcedSoftTFIDF

All Implemented Interfaces:

StringDistance, StringDistanceLearner

public class SourcedSoftTFIDF
extends SourcedTFIDF

TFIDF-based distance metric, extended to use "soft" token-matching. Specifically, tokens are considered a partial match if they get a good score using an inner string comparator.

On the WHIRL datasets, thresholding JaroWinkler at 0.9 or 0.95 seems to be about right.

Nested Class Summary

Nested classes/interfaces inherited from class com.wcohen.ss.SourcedTFIDF

SourcedTFIDF.UnitVector

Field Summary

Fields inherited from class com.wcohen.ss.AbstractSourcedStatisticalTokenDistance

collectionSize, documentFrequency, totalTokenCount

Fields inherited from class com.wcohen.ss.AbstractSourcedTokenizedStringDistance

tokenizer

Constructor Summary

Constructors

Constructor and Description
SourcedSoftTFIDF()
SourcedSoftTFIDF(SourcedTokenizer tokenizer, StringDistance tokenDistance, double tokenMatchThreshold)
SourcedSoftTFIDF(StringDistance tokenDistance)
SourcedSoftTFIDF(StringDistance tokenDistance, double tokenMatchThreshold)

Method Summary

Modifier and Type	Method and Description
String	explainScore(StringWrapper s, StringWrapper t) Explain how the distance was computed.
double	getTokenMatchThreshold()
double	score(StringWrapper s0, StringWrapper t0) This method needs to be implemented by subclasses.
void	setTokenMatchThreshold(double d)
void	setTokenMatchThreshold(Double d)
String	toString()

Methods inherited from class com.wcohen.ss.SourcedTFIDF

asUnitVector, getCollectionSize, getDocumentFrequency, getTokens, getWeight, main, prepare, setCollectionSize, setDocumentFrequency

Methods inherited from class com.wcohen.ss.AbstractSourcedStatisticalTokenDistance

checkTrainingHasHappened, train

Methods inherited from class com.wcohen.ss.AbstractSourcedTokenizedStringDistance

asBagOfSourcedTokens, prepare, setStringWrapperPool

Methods inherited from class com.wcohen.ss.AbstractStringDistance

addExample, doMain, explainScore, getDistance, hasNextQuery, nextQuery, prepare, score, setDistanceInstancePool

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

SourcedSoftTFIDF

public SourcedSoftTFIDF(SourcedTokenizer tokenizer,
                        StringDistance tokenDistance,
                        double tokenMatchThreshold)

SourcedSoftTFIDF

public SourcedSoftTFIDF(StringDistance tokenDistance,
                        double tokenMatchThreshold)

SourcedSoftTFIDF

public SourcedSoftTFIDF(StringDistance tokenDistance)

SourcedSoftTFIDF

public SourcedSoftTFIDF()

Method Detail

setTokenMatchThreshold

public void setTokenMatchThreshold(double d)

setTokenMatchThreshold

public void setTokenMatchThreshold(Double d)

getTokenMatchThreshold

public double getTokenMatchThreshold()

score

public double score(StringWrapper s0,
                    StringWrapper t0)

Description copied from class: AbstractStringDistance

This method needs to be implemented by subclasses.

Specified by:

score in interface StringDistance

Overrides:

score in class SourcedTFIDF

explainScore

public String explainScore(StringWrapper s,
                           StringWrapper t)

Explain how the distance was computed. In the output, the tokens in S and T are listed, and the common tokens are marked with an asterisk.

Specified by:

explainScore in interface StringDistance

Overrides:

explainScore in class SourcedTFIDF

toString

public String toString()

Overrides:

toString in class SourcedTFIDF