Package io.airlift.stats

Class QuantileDigest

java.lang.Object
io.airlift.stats.QuantileDigest
public class QuantileDigest extends Object
Implements http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.132.7343, a data structure for approximating quantiles by trading off error with memory requirements.

The size of the digest is adjusted dynamically to achieve the error bound and requires O(log2(U) / maxError) space, where U is the number of bits needed to represent the domain of the values added to the digest. The error is defined as the discrepancy between the real rank of the value returned in a quantile query and the rank corresponding to the queried quantile.

Thus, for a query for quantile q that returns value v, the error is |rank(v) - q * N| / N, where N is the number of elements added to the digest and rank(v) is the real rank of v

This class also supports exponential decay. The implementation is based on the ideas laid out in http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.159.3978

This class is NOT thread safe.

  • Constructor Details

    • QuantileDigest

      public QuantileDigest(double maxError)

      Create a QuantileDigest with a maximum error guarantee of "maxError" and no decay.

      Parameters:
      maxError - the max error tolerance

    • QuantileDigest

      public QuantileDigest(double maxError, double alpha)

      Create a QuantileDigest with a maximum error guarantee of "maxError" and exponential decay with factor "alpha".

      Parameters:
      maxError - the max error tolerance
      alpha - the exponential decay factor

    • QuantileDigest

      public QuantileDigest(QuantileDigest other)

    • QuantileDigest

      public QuantileDigest(io.airlift.slice.Slice serialized)

  • Method Details

    • getMaxError

      public double getMaxError()

    • getAlpha

      public double getAlpha()

    • add

      public void add(long value)

    • add

      public void add(long value, double weight)
      Adds a value to this digest. The value must be >= 0

    • add

      public void add(long value, long weight)

    • merge

      public void merge(QuantileDigest other)

    • getQuantilesLowerBound

      public List<Long> getQuantilesLowerBound(List<Double> quantiles)
      Get a lower bound on the quantiles for the given proportions. A returned q quantile is guaranteed to be within the q - maxError and q quantiles.

      The input list of quantile proportions must be sorted in increasing order, and each value must be in the range [0, 1]

    • getQuantilesUpperBound

      public List<Long> getQuantilesUpperBound(List<Double> quantiles)
      Get an upper bound on the quantiles for the given proportions. A returned q quantile is guaranteed to be within the q and q + maxError quantiles.

      The input list of quantile proportions must be sorted in increasing order, and each value must be in the range [0, 1]

    • getQuantiles

      public List<Long> getQuantiles(List<Double> quantiles)

    • getQuantile

      public long getQuantile(double quantile)
      Gets the value at the specified quantile +/- maxError. The quantile must be in the range [0, 1]

    • getQuantileLowerBound

      public long getQuantileLowerBound(double quantile)

    • getQuantileUpperBound

      public long getQuantileUpperBound(double quantile)

    • getCount

      public double getCount()
      Number (decayed) of elements added to this quantile digest

    • getHistogram

      public List<QuantileDigest.Bucket> getHistogram(List<Long> bucketUpperBounds)

    • getHistogram

      public List<QuantileDigest.Bucket> getHistogram(List<Long> bucketUpperBounds, QuantileDigest.MiddleFunction middleFunction)

    • getMin

      public long getMin()

    • getMax

      public long getMax()

    • estimatedInMemorySizeInBytes

      public int estimatedInMemorySizeInBytes()

    • estimatedSerializedSizeInBytes

      public int estimatedSerializedSizeInBytes()

    • serialize

      public io.airlift.slice.Slice serialize()

    • getConfidenceFactor

      public double getConfidenceFactor()
      Computes the maximum error of the current digest

    • toGraphviz

      public String toGraphviz()