pyinterp.StreamingHistogram

class pyinterp.StreamingHistogram(values: dask.array.core.Array | numpy.ndarray, weights: None | dask.array.core.Array | numpy.ndarray = None, axis: int | Iterable[int] | None = None, bin_count: int | None = None, dtype: numpy.dtype | None = None)[source]

Bases: object

Streaming histogram.

The bins in the histogram have no predefined size, so that as values are pushed into the histogram, bins are added and merged as soon as their numbers exceed the maximum allowed capacity. A particularly interesting feature of streaming histograms is that they can be used to approximate quantiles without sorting (or even storing) values individually. The histograms can be constructed independently and merged, making them usable with Dask.

Parameters:

  • values –

    Array containing numbers whose statistics are desired.

    Note

    NaNs are automatically ignored.

  • weights – An array of weights associated with the values. If not provided, all values are assumed to have equal weight.

  • axis – Axis or axes along which to compute the statistics. If not provided, the statistics are computed over the flattened array.

  • bin_count – The maximum number of bins to use in the histogram. If the number of bins exceeds the number of values, the histogram will be trimmed. Default is None, which will set the number of bins to 100.

  • dtype – Data type of the returned array. By default, the data type is numpy.float64.

See also

Yael Ben-Haim and Elad Tom-Tov, A Streaming Parallel Decision Tree Algorithm, Journal of Machine Learning Research, 11, 28, 849-872, http://jmlr.org/papers/v11/ben-haim10a.html

Note

If you do not want to estimate the quantiles of the dataset, use the class DescriptiveStatistics which will give you more accurate results.

Public Methods

bins()

Returns the histogram bins.

count()

Returns the count of samples.

kurtosis()

Returns the kurtosis of samples.

max()

Returns the maximum of samples.

mean()

Returns the mean of samples.

min()

Returns the minimum of samples.

quantile([q])

Returns the q quantile of samples.

size()

Returns the number of bins allocated to calculate the histogram.

skewness()

Returns the skewness of samples.

std()

Returns the standard deviation of samples.

sum_of_weights()

Returns the sum of weights.

var()

Returns the variance of samples.

Special Methods

__iadd__(other)

Adds a new histogram to the current one.