info.debatty.java.stringsimilarity

Class KShingling

java.lang.Object

java.util.AbstractCollection<E>

java.util.AbstractSet<E>

java.util.HashSet<String>

info.debatty.java.stringsimilarity.KShingling

All Implemented Interfaces:

Serializable, Cloneable, Iterable<String>, Collection<String>, Set<String>

public class KShingling
extends HashSet<String>
implements Serializable

A k-shingling is a set of unique k-grams, used to measure the similarity of two documents. Generally speaking, a k-gram is any sequence of k tokens. We use here the definition from Leskovec, Rajaraman & Ullman (2014), "Mining of Massive Datasets", Cambridge University Press: Multiple subsequent spaces are replaced by a single space, and a k-gram is a sequence of k characters.

Author:

Thibault Debatty http://www.debatty.info

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected int	k

Constructor Summary

Constructors

Constructor and Description
KShingling()
KShingling(int k)

Method Summary

Methods

Modifier and Type	Method and Description
boolean	add(String s) Add a k-shingle s to the list of possible shingles
boolean[]	booleanVectorOf(String s) Compute and return the boolean vector representation of string s.
static int	countOccurences(String substring, String str)
int	getK()
Set<Integer>	integerSetOf(String s) Compute the boolean representation of string s, returned as a set of position integers.
static void	main(String[] args)
boolean	parse(String s) Extract all k-singles from sting s and add them to the list of possible shingles
static void	printArray(boolean[] a)
static void	printArray(int[] a)
int[]	profileOf(String s) Compute and return the profile of s, as defined by Ukkonen "Approximate string-matching with q-grams and maximal matches".
void	setK(int k) Set the size of k-grams.

Methods inherited from class java.util.HashSet

clear, clone, contains, isEmpty, iterator, remove, size

Methods inherited from class java.util.AbstractSet

equals, hashCode, removeAll

Methods inherited from class java.util.AbstractCollection

addAll, containsAll, retainAll, toArray, toArray, toString

Methods inherited from class java.lang.Object

finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface java.util.Set

addAll, containsAll, equals, hashCode, removeAll, retainAll, toArray, toArray

Field Detail

k

protected int k

Constructor Detail

KShingling

public KShingling()

KShingling

public KShingling(int k)

Method Detail

main

public static void main(String[] args)

countOccurences

public static int countOccurences(String substring,
                  String str)

printArray

public static void printArray(boolean[] a)

printArray

public static void printArray(int[] a)

getK

public int getK()

setK

public final void setK(int k)

Set the size of k-grams. Default value is 5 (recommended for emails). A good rule of thumb is to imagine that there are only 20 characters and estimate the number of k-shingles as 20^k. For large documents, such as research articles, choice k = 9 is considered safe.

Parameters:

k -

parse

public boolean parse(String s)

Extract all k-singles from sting s and add them to the list of possible shingles

Parameters:

s -

Returns:

true

add

public boolean add(String s)

Add a k-shingle s to the list of possible shingles

Specified by:

add in interface Collection<String>

Specified by:

add in interface Set<String>

Overrides:

add in class HashSet<String>

Parameters:

s -

Returns:

booleanVectorOf

public boolean[] booleanVectorOf(String s)

Compute and return the boolean vector representation of string s. E.g. if this set contains the shingles [AB, BC, CD, DE] and s is ABCD This will return [true, true, true, false]

Parameters:

s -

Returns:

integerSetOf

public Set<Integer> integerSetOf(String s)

Compute the boolean representation of string s, returned as a set of position integers. E.g. if this set contains the shingles [AB, BC, CD, DE] and s is ABCD This will return (0, 1, 2)

Parameters:

s -

Returns:

profileOf

public int[] profileOf(String s)

Compute and return the profile of s, as defined by Ukkonen "Approximate string-matching with q-grams and maximal matches". https://www.cs.helsinki.fi/u/ukkonen/TCS92.pdf The profile is the number of occurences of k-shingles, and is used to compute q-gram similarity. E.g. if this set contains the shingles [AB, BC, CD, DE] and s is ABCDAB This will return [2, 1, 1, 0]

Parameters:

s -

Returns: