com.wcohen.ss

Class TFIDF

java.lang.Object

com.wcohen.ss.AbstractStringDistance

com.wcohen.ss.AbstractTokenizedStringDistance

com.wcohen.ss.AbstractStatisticalTokenDistance

com.wcohen.ss.TFIDF

All Implemented Interfaces:

StringDistance, StringDistanceLearner

Direct Known Subclasses:

SoftTFIDF

public class TFIDF
extends AbstractStatisticalTokenDistance

TFIDF-based distance metric.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected class	TFIDF.UnitVector Marker class extending BagOfTokens

Field Summary

Fields inherited from class com.wcohen.ss.AbstractStatisticalTokenDistance

collectionSize, documentFrequency, totalTokenCount

Fields inherited from class com.wcohen.ss.AbstractTokenizedStringDistance

tokenizer

Constructor Summary

Constructors

Constructor and Description
TFIDF()
TFIDF(Tokenizer tokenizer)

Method Summary

Modifier and Type	Method and Description
protected TFIDF.UnitVector	asUnitVector(StringWrapper w)
String	explainScore(StringWrapper s, StringWrapper t) Explain how the distance was computed.
int	getCollectionSize()
int	getDocumentFrequency(Token token) Get the document frequency of the token.
Token[]	getTokens() Access the tokens of the last prepare()-ed string.
int	getVocabularySize()
double	getWeight(Token token) Access the weight of a token in the vector created for the last prepare()-ed string.
static void	main(String[] argv)
StringWrapper	prepare(String s) Preprocess a string by finding tokens and giving them TFIDF weights
double	score(StringWrapper s, StringWrapper t) This method needs to be implemented by subclasses.
void	setCollectionSize(int n) Setting the collectionSize and alsoSet the size of the collection that this TFIDF measure was trained on to some value.
void	setDocumentFrequency(Token token, int df) Set the document frequency of the token to some value.
void	setTokenCount(int tc)
String	toString()

Methods inherited from class com.wcohen.ss.AbstractStatisticalTokenDistance

checkTrainingHasHappened, tokenIterator, train

Methods inherited from class com.wcohen.ss.AbstractTokenizedStringDistance

asBagOfTokens, prepare, setStringWrapperPool

Methods inherited from class com.wcohen.ss.AbstractStringDistance

addExample, doMain, explainScore, getDistance, hasNextQuery, nextQuery, prepare, score, setDistanceInstancePool

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

TFIDF

public TFIDF(Tokenizer tokenizer)

TFIDF

public TFIDF()

Method Detail

score

public double score(StringWrapper s,
                    StringWrapper t)

Description copied from class: AbstractStringDistance

This method needs to be implemented by subclasses.

Specified by:

score in interface StringDistance

Specified by:

score in class AbstractStringDistance

asUnitVector

protected TFIDF.UnitVector asUnitVector(StringWrapper w)

prepare

public StringWrapper prepare(String s)

Preprocess a string by finding tokens and giving them TFIDF weights

Specified by:

prepare in interface StringDistance

Overrides:

prepare in class AbstractStringDistance

getTokens

public Token[] getTokens()

Access the tokens of the last prepare()-ed string.

getWeight

public double getWeight(Token token)

Access the weight of a token in the vector created for the last prepare()-ed string.

getDocumentFrequency

public int getDocumentFrequency(Token token)

Get the document frequency of the token.

Overrides:

getDocumentFrequency in class AbstractStatisticalTokenDistance

setDocumentFrequency

public void setDocumentFrequency(Token token,
                                 int df)

Set the document frequency of the token to some value. Setting the collectionSize and also setting the document frequency of every token is an alternative to explicit training.

setTokenCount

public void setTokenCount(int tc)

getCollectionSize

public int getCollectionSize()

setCollectionSize

public void setCollectionSize(int n)

Setting the collectionSize and alsoSet the size of the collection that this TFIDF measure was trained on to some value. setting the document frequency of every token is an alternative to explicit training.

getVocabularySize

public int getVocabularySize()

explainScore

public String explainScore(StringWrapper s,
                           StringWrapper t)

Explain how the distance was computed. In the output, the tokens in S and T are listed, and the common tokens are marked with an asterisk.

Specified by:

explainScore in interface StringDistance

Specified by:

explainScore in class AbstractStringDistance

toString

public String toString()

Overrides:

toString in class Object

main

public static void main(String[] argv)