com.wcohen.ss

Class SourcedTFIDF

java.lang.Object

com.wcohen.ss.AbstractStringDistance

com.wcohen.ss.AbstractSourcedTokenizedStringDistance

com.wcohen.ss.AbstractSourcedStatisticalTokenDistance

com.wcohen.ss.SourcedTFIDF

All Implemented Interfaces:

StringDistance, StringDistanceLearner

Direct Known Subclasses:

SourcedSoftTFIDF

public class SourcedTFIDF
extends AbstractSourcedStatisticalTokenDistance

Sourced-based distance metric.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected class	SourcedTFIDF.UnitVector Marker class extending BagOfTokens

Field Summary

Fields inherited from class com.wcohen.ss.AbstractSourcedStatisticalTokenDistance

collectionSize, documentFrequency, totalTokenCount

Fields inherited from class com.wcohen.ss.AbstractSourcedTokenizedStringDistance

tokenizer

Constructor Summary

Constructors

Constructor and Description
SourcedTFIDF()
SourcedTFIDF(SourcedTokenizer tokenizer)

Method Summary

Modifier and Type	Method and Description
protected SourcedTFIDF.UnitVector	asUnitVector(SourcedStringWrapper w)
String	explainScore(StringWrapper s, StringWrapper t) Explain how the distance was computed.
int	getCollectionSize()
int	getDocumentFrequency(Token token) Get the document frequency of the token.
Token[]	getTokens() Access the tokens of the last prepare()-ed string.
double	getWeight(Token token) Access the weight of a token in the vector created for the last prepare()-ed string.
static void	main(String[] argv)
StringWrapper	prepare(String s) Preprocess a string by finding tokens and giving them TFIDF weights
double	score(StringWrapper s0, StringWrapper t0) This method needs to be implemented by subclasses.
void	setCollectionSize(int n) Setting the collectionSize and alsoSet the size of the collection that this TFIDF measure was trained on to some value.
void	setDocumentFrequency(Token token, int df) Set the document frequency of the token to some value.
String	toString()

Methods inherited from class com.wcohen.ss.AbstractSourcedStatisticalTokenDistance

checkTrainingHasHappened, train

Methods inherited from class com.wcohen.ss.AbstractSourcedTokenizedStringDistance

asBagOfSourcedTokens, prepare, setStringWrapperPool

Methods inherited from class com.wcohen.ss.AbstractStringDistance

addExample, doMain, explainScore, getDistance, hasNextQuery, nextQuery, prepare, score, setDistanceInstancePool

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

SourcedTFIDF

public SourcedTFIDF(SourcedTokenizer tokenizer)

SourcedTFIDF

public SourcedTFIDF()

Method Detail

score

public double score(StringWrapper s0,
                    StringWrapper t0)

Description copied from class: AbstractStringDistance

This method needs to be implemented by subclasses.

Specified by:

score in interface StringDistance

Specified by:

score in class AbstractStringDistance

asUnitVector

protected SourcedTFIDF.UnitVector asUnitVector(SourcedStringWrapper w)

prepare

public StringWrapper prepare(String s)

Preprocess a string by finding tokens and giving them TFIDF weights

Specified by:

prepare in interface StringDistance

Overrides:

prepare in class AbstractStringDistance

getTokens

public Token[] getTokens()

Access the tokens of the last prepare()-ed string.

getWeight

public double getWeight(Token token)

Access the weight of a token in the vector created for the last prepare()-ed string.

getDocumentFrequency

public int getDocumentFrequency(Token token)

Get the document frequency of the token.

Overrides:

getDocumentFrequency in class AbstractSourcedStatisticalTokenDistance

setDocumentFrequency

public void setDocumentFrequency(Token token,
                                 int df)

Set the document frequency of the token to some value. Setting the collectionSize and also setting the document frequency of every token is an alternative to explicit training.

getCollectionSize

public int getCollectionSize()

setCollectionSize

public void setCollectionSize(int n)

Setting the collectionSize and alsoSet the size of the collection that this TFIDF measure was trained on to some value. setting the document frequency of every token is an alternative to explicit training.

explainScore

public String explainScore(StringWrapper s,
                           StringWrapper t)

Explain how the distance was computed. In the output, the tokens in S and T are listed, and the common tokens are marked with an asterisk.

Specified by:

explainScore in interface StringDistance

Specified by:

explainScore in class AbstractStringDistance

toString

public String toString()

Overrides:

toString in class Object

main

public static void main(String[] argv)