com.clearspring.experimental.stream.cardinality

Class HyperBitBit

java.lang.Object

com.clearspring.experimental.stream.cardinality.HyperBitBit

All Implemented Interfaces:

ICardinality

public class HyperBitBit
extends Object
implements ICardinality

Java implementation of HyperBitBit (HBB) algorithm as seen on the presentation by Robert Sedgewick:

https://www.cs.princeton.edu/~rs/talks/AC11-Cardinality.pdf

HBB aims to beat HyperLogLog. From the talk, on practical data: - HyperBitBit, for N < 2^64, - Uses 128 + 6 bits. (in this implementation case 128 + 8) - Estimates cardinality within 10% of the actual.

The algorithm still need some improvements. - If you insert twice the same element the structure can change (not as in HLL) - For small cardinalities it does not work AT ALL. - The constatn 5.4 used in the cardinality estimation formula should be refined with real world applications feedback

Even so, HyperBitBit has the necessary characteristics to become a better algorithm than HyperLogLog: - Makes one pass through the stream. - Uses a few dozen machine instructions per value - Uses a few hundred bits - Achieves 10% relative accuracy or better

Any feedback to improve the algorithm in its weak points will be welcome.

Constructor Summary

Constructors

Constructor and Description
HyperBitBit() Create a new HyperBitBit instance.

Method Summary

Modifier and Type	Method and Description
long	cardinality()
byte[]	getBytes()
ICardinality	merge(ICardinality... estimators) Merges estimators to produce a new estimator for the combined streams of this estimator and those passed as arguments.
boolean	offer(Object o)
boolean	offerHashed(int hashedInt) Offer the value as a hashed long value
boolean	offerHashed(long hashedLong) Offer the value as a hashed long value
int	sizeof()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HyperBitBit

public HyperBitBit()

Create a new HyperBitBit instance. Remember that it does not work well for small cardinalities!

Method Detail

offer

public boolean offer(Object o)

Specified by:

offer in interface ICardinality

Parameters:

o - stream element

Returns:

false if the value returned by cardinality() is unaffected by the appearance of o in the stream.

offerHashed

public boolean offerHashed(long hashedLong)

Description copied from interface: ICardinality

Offer the value as a hashed long value

Specified by:

offerHashed in interface ICardinality

Parameters:

hashedLong - - the hash of the item to offer to the estimator

Returns:

false if the value returned by cardinality() is unaffected by the appearance of hashedLong in the stream

offerHashed

public boolean offerHashed(int hashedInt)

Description copied from interface: ICardinality

Offer the value as a hashed long value

Specified by:

offerHashed in interface ICardinality

Parameters:

hashedInt - - the hash of the item to offer to the estimator

Returns:

false if the value returned by cardinality() is unaffected by the appearance of hashedInt in the stream

cardinality

public long cardinality()

Specified by:

cardinality in interface ICardinality

Returns:

the number of unique elements in the stream or an estimate thereof

sizeof

public int sizeof()

Specified by:

sizeof in interface ICardinality

Returns:

size in bytes needed for serialization

getBytes

public byte[] getBytes()
                throws IOException

Specified by:

getBytes in interface ICardinality

Returns:

Throws:

IOException

merge

public ICardinality merge(ICardinality... estimators)
                   throws CardinalityMergeException

Description copied from interface: ICardinality

Merges estimators to produce a new estimator for the combined streams of this estimator and those passed as arguments.

Nor this estimator nor the one passed as parameters are modified.

Specified by:

merge in interface ICardinality

Parameters:

estimators - Zero or more compatible estimators

Throws:

CardinalityMergeException - If at least one of the estimators is not compatible with this one