com.datadoghq.sketch.ddsketch

Class DDSketch

java.lang.Object

com.datadoghq.sketch.ddsketch.DDSketch

All Implemented Interfaces:

QuantileSketch<DDSketch>, java.util.function.DoubleConsumer

public class DDSketch
extends java.lang.Object
implements QuantileSketch<DDSketch>

A QuantileSketch with relative-error guarantees. This sketch computes quantile values with an approximation error that is relative to the actual quantile value. It works with both positive and negative input values.

For instance, using DDSketch with a relative accuracy guarantee set to 1%, if the expected quantile value is 100, the computed quantile value is guaranteed to be between 99 and 101. If the expected quantile value is 1000, the computed quantile value is guaranteed to be between 990 and 1010.

DDSketch works by mapping floating-point input values to bins and counting the number of values for each bin. The mapping to bins is handled by IndexMapping, while the underlying structure that keeps track of bin counts is Store. The IndexMapping and the Store supplier are provided when constructing the sketch and can be adjusted based on use-case requirements. See DDSketches for preset sketches and some explanation about the involved tradeoffs.

Note that negative values are inverted before being mapped to the store. That means that if you use a store that collapses the lowest (resp., the highest) indexes, it will affect the input values that are the closest to (resp., the farthest away from) zero.

Note that this implementation is not thread-safe.

Constructor Summary

Constructors

Constructor and Description
DDSketch(double relativeAccuracy) Constructs a simple instance of DDSketch with the provided relative accuracy guarantee.
DDSketch(IndexMapping indexMapping, java.util.function.Supplier<Store> storeSupplier) Constructs an initially empty quantile sketch using the specified IndexMapping and Store supplier.
DDSketch(IndexMapping indexMapping, java.util.function.Supplier<Store> negativeValueStoreSupplier, java.util.function.Supplier<Store> positiveValueStoreSupplier) Constructs an initially empty quantile sketch using the specified IndexMapping and Store suppliers.
DDSketch(IndexMapping indexMapping, java.util.function.Supplier<Store> negativeValueStoreSupplier, java.util.function.Supplier<Store> positiveValueStoreSupplier, double minIndexedValue) Constructs an initially empty quantile sketch using the specified IndexMapping and Store supplier.

Method Summary

Modifier and Type	Method and Description
void	accept(double value) Adds a value to the sketch.
void	accept(double value, double count) Adds a value to the sketch with a floating-point count.
static DDSketch	balanced(double relativeAccuracy) Deprecated. Use DDSketches.unboundedDense(double).
static DDSketch	balancedCollapsingHighest(double relativeAccuracy, int maxNumBins) Deprecated. Use DDSketches.collapsingHighestDense(double, int).
static DDSketch	balancedCollapsingLowest(double relativeAccuracy, int maxNumBins) Deprecated. Use DDSketches.collapsingLowestDense(double, int).
void	clear() Sets all counts in the sketch to zero.
DDSketch	convert(IndexMapping newIndexMapping, java.util.function.Supplier<Store> storeSupplier) Builds a new DDSketch that encodes the content of this sketch with the specified index mapping and the specified stores.
DDSketch	copy()
static DDSketch	decode(Input input, java.util.function.Supplier<Store> storeSupplier)
static DDSketch	decode(Input input, java.util.function.Supplier<Store> storeSupplier, IndexMapping indexMapping)
static DDSketch	decode(Input input, java.util.function.Supplier<Store> storeSupplier, IndexMapping indexMapping, com.datadoghq.sketch.ddsketch.DDSketch.Decoder fallback)
void	decodeAndMergeWith(Input input)
void	encode(Output output, boolean omitIndexMapping)
static DDSketch	fast(double relativeAccuracy) Deprecated. Use new DDSketch(new BitwiseLinearlyInterpolatedMapping(relativeAccuracy), UnboundedSizeDenseStore::new).
static DDSketch	fastCollapsingHighest(double relativeAccuracy, int maxNumBins) Deprecated. Use new DDSketch(new BitwiseLinearlyInterpolatedMapping(relativeAccuracy), () -> new CollapsingHighestDenseStore(maxNumBins)).
static DDSketch	fastCollapsingLowest(double relativeAccuracy, int maxNumBins) Deprecated. Use new DDSketch(new BitwiseLinearlyInterpolatedMapping(relativeAccuracy), () -> new CollapsingLowestDenseStore(maxNumBins).
double	getCount()
IndexMapping	getIndexMapping()
double	getMaxValue()
double	getMinValue()
Store	getNegativeValueStore()
Store	getPositiveValueStore()
double	getSum() Returns an approximation of the sum of the values that have been added to the sketch.
double	getValueAtQuantile(double quantile)
double[]	getValuesAtQuantiles(double[] quantiles)
boolean	isEmpty()
static DDSketch	memoryOptimal(double relativeAccuracy) Deprecated. Use DDSketches.logarithmicCollapsingLowestDense(double, int).
static DDSketch	memoryOptimalCollapsingHighest(double relativeAccuracy, int maxNumBins) Deprecated. Use DDSketches.logarithmicCollapsingHighestDense(double, int).
static DDSketch	memoryOptimalCollapsingLowest(double relativeAccuracy, int maxNumBins) Deprecated. Use DDSketches.logarithmicCollapsingLowestDense(double, int).
void	mergeWith(DDSketch other) Merges the other sketch into this one.
static DDSketch	of(IndexMapping indexMapping, Store negativeValueStore, Store positiveValueStore, double zeroCount) Constructs an instance of DDSketch from a mapping, possibly non-empty stores and a non-negative count for the zero bucket.
java.nio.ByteBuffer	serialize() Produces protobuf encoded bytes which are equivalent to using the official protobuf bindings, without requiring a runtime dependency on protobuf-java.
int	serializedSize()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.datadoghq.sketch.QuantileSketch

getAverage

Methods inherited from interface java.util.function.DoubleConsumer

andThen

Constructor Detail

DDSketch

public DDSketch(IndexMapping indexMapping,
                java.util.function.Supplier<Store> storeSupplier)

Constructs an initially empty quantile sketch using the specified IndexMapping and Store supplier.

Parameters:

indexMapping - the mapping between floating-point values and integer indices to be used by the sketch

storeSupplier - the store constructor for keeping track of added values

See Also:

DDSketches

DDSketch

public DDSketch(IndexMapping indexMapping,
                java.util.function.Supplier<Store> negativeValueStoreSupplier,
                java.util.function.Supplier<Store> positiveValueStoreSupplier)

Constructs an initially empty quantile sketch using the specified IndexMapping and Store suppliers.

Parameters:

indexMapping - the mapping between floating-point values and integer indices to be used by the sketch

negativeValueStoreSupplier - the store constructor for keeping track of added negative values

positiveValueStoreSupplier - the store constructor for keeping track of added positive values

See Also:

DDSketches

DDSketch

public DDSketch(IndexMapping indexMapping,
                java.util.function.Supplier<Store> negativeValueStoreSupplier,
                java.util.function.Supplier<Store> positiveValueStoreSupplier,
                double minIndexedValue)

Constructs an initially empty quantile sketch using the specified IndexMapping and Store supplier.

Parameters:

indexMapping - the mapping between floating-point values and integer indices to be used by the sketch

negativeValueStoreSupplier - the store constructor for keeping track of added negative values

positiveValueStoreSupplier - the store constructor for keeping track of added positive values

minIndexedValue - the least value that should be distinguished from zero

See Also:

DDSketches

DDSketch

public DDSketch(double relativeAccuracy)

Constructs a simple instance of DDSketch with the provided relative accuracy guarantee.

This is a convenience constructor, mostly for testing purposes. For preset sketches that are well-suited to a production setting and to situations where performance and memory usage constraints are involved, see DDSketches.

Parameters:

relativeAccuracy - the relative accuracy guarantee of the constructed sketch

See Also:

DDSketches

Method Detail

of

public static DDSketch of(IndexMapping indexMapping,
                          Store negativeValueStore,
                          Store positiveValueStore,
                          double zeroCount)

Constructs an instance of DDSketch from a mapping, possibly non-empty stores and a non-negative count for the zero bucket.

Parameters:

indexMapping - the index mapping

negativeValueStore - the store for negative values, possibly non-empty

positiveValueStore - the store for positive values, possibly non-empty

zeroCount - the count for the zero bucket, which should be non-negative (and non-NaN)

Returns:

a new instance of DDSketch

Throws:

java.lang.IllegalArgumentException - if the zeroCount is negative or NaN

java.lang.NullPointerException - if one of the arguments is null

getIndexMapping

public IndexMapping getIndexMapping()

getNegativeValueStore

public Store getNegativeValueStore()

getPositiveValueStore

public Store getPositiveValueStore()

accept

public void accept(double value)

Adds a value to the sketch.

Specified by:

accept in interface QuantileSketch<DDSketch>

Specified by:

accept in interface java.util.function.DoubleConsumer

Parameters:

value - the value to be added

Throws:

java.lang.IllegalArgumentException - if the value is outside the range that is tracked by the sketch

accept

public void accept(double value,
                   double count)

Adds a value to the sketch with a floating-point count.

If count is an integer, calling accept(value, count) is equivalent to calling accept(value) count times.

Specified by:

accept in interface QuantileSketch<DDSketch>

Parameters:

value - the value to be added

count - the weight associated with the value to be added

Throws:

java.lang.IllegalArgumentException - if the value is outside the range that is tracked by the sketch

mergeWith

public void mergeWith(DDSketch other)

Merges the other sketch into this one. After this operation, this sketch encodes the values that were added to both this and the other sketches.

Specified by:

mergeWith in interface QuantileSketch<DDSketch>

Parameters:

other - the sketch to be merged into this one

Throws:

java.lang.IllegalArgumentException - if the other sketch does not use the same index mapping

copy

public DDSketch copy()

Specified by:

copy in interface QuantileSketch<DDSketch>

Returns:

a (deep) copy of this sketch

isEmpty

public boolean isEmpty()

Specified by:

isEmpty in interface QuantileSketch<DDSketch>

Returns:

iff no value has been added to this sketch

clear

public void clear()

Description copied from interface: QuantileSketch

Sets all counts in the sketch to zero. The sketch behaves as if it were empty after this call, but any allocated memory is not released.

Specified by:

clear in interface QuantileSketch<DDSketch>

getCount

public double getCount()

Specified by:

getCount in interface QuantileSketch<DDSketch>

Returns:

the total number of values that have been added to this sketch

getSum

public double getSum()

Returns an approximation of the sum of the values that have been added to the sketch. If the values that have been added to the sketch all have the same sign, the approximation error has the relative accuracy guarantees of the IndexMapping used for this sketch.

Specified by:

getSum in interface QuantileSketch<DDSketch>

Returns:

an approximation of the sum of the values that have been added to the sketch

getMinValue

public double getMinValue()

Specified by:

getMinValue in interface QuantileSketch<DDSketch>

Returns:

the minimum value that has been added to this sketch

getMaxValue

public double getMaxValue()

Specified by:

getMaxValue in interface QuantileSketch<DDSketch>

Returns:

the maximum value that has been added to this sketch

getValueAtQuantile

public double getValueAtQuantile(double quantile)

Specified by:

getValueAtQuantile in interface QuantileSketch<DDSketch>

Parameters:

quantile - a number between 0 and 1 (both included)

Returns:

the value at the specified quantile

getValuesAtQuantiles

public double[] getValuesAtQuantiles(double[] quantiles)

Specified by:

getValuesAtQuantiles in interface QuantileSketch<DDSketch>

Parameters:

quantiles - number between 0 and 1 (both included)

Returns:

the values at the respective specified quantiles

convert

public DDSketch convert(IndexMapping newIndexMapping,
                        java.util.function.Supplier<Store> storeSupplier)

Builds a new DDSketch that encodes the content of this sketch with the specified index mapping and the specified stores. This DDSketch is not modified by the operation.

Note that this conversion degrades the accuracy of the new sketch to the extent that it is not necessarily upper-bounded by the accuracy of the provided index mapping. See IndexMappingConverter.distributingUniformly(IndexMapping, IndexMapping) for details.

Parameters:

newIndexMapping - the index mapping that the new sketch should use to map values to bins

storeSupplier - a constructor of an initially empty store to be used to encode bins

Returns:

a new DDSketch

encode

public void encode(Output output,
                   boolean omitIndexMapping)
            throws java.io.IOException

Throws:

java.io.IOException

decodeAndMergeWith

public void decodeAndMergeWith(Input input)
                        throws java.io.IOException

Throws:

java.io.IOException

decode

public static DDSketch decode(Input input,
                              java.util.function.Supplier<Store> storeSupplier)
                       throws java.io.IOException

Throws:

java.io.IOException

decode

public static DDSketch decode(Input input,
                              java.util.function.Supplier<Store> storeSupplier,
                              IndexMapping indexMapping)
                       throws java.io.IOException

Throws:

java.io.IOException

decode

public static DDSketch decode(Input input,
                              java.util.function.Supplier<Store> storeSupplier,
                              IndexMapping indexMapping,
                              com.datadoghq.sketch.ddsketch.DDSketch.Decoder fallback)
                       throws java.io.IOException

Throws:

java.io.IOException

serializedSize

public int serializedSize()

Returns:

the size of the sketch when serialized in protobuf

serialize

public java.nio.ByteBuffer serialize()

Produces protobuf encoded bytes which are equivalent to using the official protobuf bindings, without requiring a runtime dependency on protobuf-java.

Currently this API is asymmetric in that there is not an equivalent method to deserialize a sketch from a protobuf message. DDSketchProtoBinding can be used for this purpose, but requires a runtime dependency on protobuf-java. This API may be made symmetric in the future.

Returns:

the sketch serialized as a ByteBuffer.

balanced

@Deprecated
public static DDSketch balanced(double relativeAccuracy)

Deprecated. Use DDSketches.unboundedDense(double).

balancedCollapsingLowest

@Deprecated
public static DDSketch balancedCollapsingLowest(double relativeAccuracy,
                                                            int maxNumBins)

Deprecated. Use DDSketches.collapsingLowestDense(double, int).

balancedCollapsingHighest

@Deprecated
public static DDSketch balancedCollapsingHighest(double relativeAccuracy,
                                                             int maxNumBins)

Deprecated. Use DDSketches.collapsingHighestDense(double, int).

fast

@Deprecated
public static DDSketch fast(double relativeAccuracy)

Deprecated. Use new DDSketch(new BitwiseLinearlyInterpolatedMapping(relativeAccuracy), UnboundedSizeDenseStore::new).

fastCollapsingLowest

@Deprecated
public static DDSketch fastCollapsingLowest(double relativeAccuracy,
                                                        int maxNumBins)

Deprecated. Use new DDSketch(new BitwiseLinearlyInterpolatedMapping(relativeAccuracy), () -> new CollapsingLowestDenseStore(maxNumBins).

fastCollapsingHighest

@Deprecated
public static DDSketch fastCollapsingHighest(double relativeAccuracy,
                                                         int maxNumBins)

Deprecated. Use new DDSketch(new BitwiseLinearlyInterpolatedMapping(relativeAccuracy), () -> new CollapsingHighestDenseStore(maxNumBins)).

memoryOptimal

@Deprecated
public static DDSketch memoryOptimal(double relativeAccuracy)

Deprecated. Use DDSketches.logarithmicCollapsingLowestDense(double, int).

memoryOptimalCollapsingLowest

@Deprecated
public static DDSketch memoryOptimalCollapsingLowest(double relativeAccuracy,
                                                                 int maxNumBins)

Deprecated. Use DDSketches.logarithmicCollapsingLowestDense(double, int).

memoryOptimalCollapsingHighest

@Deprecated
public static DDSketch memoryOptimalCollapsingHighest(double relativeAccuracy,
                                                                  int maxNumBins)

Deprecated. Use DDSketches.logarithmicCollapsingHighestDense(double, int).