Histograms

A histogram summarises how one score is distributed. GAIn computes histograms as part of a resource’s statistics build, stores them in the resource, and renders them on the resource’s HTML summary page. In Python they are the objects you get back when you read that stored statistic.

Three kinds, and two aliases

There is one histogram class per kind of distribution, and one configuration class for each:

Kind	Histogram	Configuration
continuous, binned	NumberHistogram	NumberHistogramConfig
discrete, counted per value	CategoricalHistogram	CategoricalHistogramConfig
none — deliberately not computed	NullHistogram	NullHistogramConfig

Histogram and HistogramConfig are type aliases, not classes:

Histogram = NullHistogram | CategoricalHistogram | NumberHistogram
HistogramConfig = (
    NullHistogramConfig | CategoricalHistogramConfig | NumberHistogramConfig
)

They are the names the type annotations use, so a function documented as returning a Histogram returns one of the three concrete classes above. Because they are aliases there is nothing to autoclass and no page anchor to link to — check type on the object, or match on the class, to find out which one you have.

The null histogram is not an absence

NullHistogram is a real object that records why no histogram exists — a score whose values cannot be binned, a statistics build that was told to skip it, or a read that failed. The reason is a plain string — NullHistogramConfig.reason, copied onto the histogram as NullHistogram.reason — and it is required, not optional: there is no such thing as an unexplained null histogram, though default_config() supplies "Unspecified reason" for the few callers that genuinely have nothing better to say.

A null histogram answers the same interface as the other two, so consumers do not branch; its plot() draws nothing and the summary page renders the reason instead.

It is also what load_histogram() returns instead of failing, for the cases it handles: a missing file, an unrecognised file extension, and a body that fails to deserialise each come back as an explained NullHistogram. It is not a blanket guarantee — the YAML or JSON parse and the config/type lookups sit outside that handling, so a syntactically broken file or one missing its config key raises.

Histograms are stored as .yaml on older resources and .json on newer ones, so prefer to let the score resolve the name rather than spelling it yourself:

from gain.genomic_resources.histogram import load_histogram

res = grr.get_resource("hg38/scores/phastCons100way")
hist = load_histogram(res, "statistics/histogram_phastCons100way.json")
print(hist.type, hist.values_domain())     # number_histogram [0, 1]

ScoreResource.get_histogram_filename(score_id) is that resolver: it returns the .yaml name when the resource’s manifest carries one and the .json name otherwise. Passing a name the resource does not have is not an error — it is exactly the “file not found” case above, so the mistake shows up as a NullHistogram rather than an exception.

Configuration

The histogram: block of a score’s scores: entry is written on the curator-facing YAML side: its keys — bin counts, log scales, view ranges, value ordering, and the custom plot_function hook — are documented on Histogram configuration. That section also carries the one Python example that belongs on the user page: how to write a plot_function.

What that block parses into is a HistogramConfig, and that is described here. The chain is short and worth knowing, because it explains where a default comes from when the YAML says nothing:

1. parse_scoredef_config reads the scores: block and produces one GenomicScoreDef per entry.

2. For each entry, build_histogram_config reads that entry’s histogram: key and returns the matching *HistogramConfig. When the key is absent it returns None — it does not substitute a default.

3. The default is chosen later, at statistics-build time, by build_default_histogram_conf from the score’s value type. That is why an unconfigured score still gets a histogram, and why which kind it gets depends on the declared value_type rather than on the data.

4. The statistics build uses the config to construct the histogram, fills it with add_value() or the vectorised add_batch(), and stores the serialised result in the resource.

5. load_histogram() reads it back.

Merging is what makes step 3 parallelisable: each task histograms a slice of the genome and merge() folds the partial results together. A merge requires compatible configurations — two number histograms with different bin edges cannot be added — so the configuration is fixed before the build starts, not derived from the data as it arrives.

Cardinality, and truncation

CategoricalHistogram counts one bucket per distinct value, and two separate mechanisms keep that from getting out of hand. They are easy to confuse, so it is worth separating them.

The cardinality limit is a refusal, not a cap. Past UNIQUE_VALUES_LIMIT (100), add_value raises HistogramError and the statistics build replaces the whole histogram with a NullHistogram carrying that message — nothing is kept and nothing is truncated. The limit applies only when the score was not explicitly configured as categorical: see default_config() for why the flag that gates it, enforce_type, reads backwards from its name.

Truncation is about what gets written and drawn, and applies to a histogram that was built successfully. display_values is the ordered subset the summary page draws, controlled by the configuration’s displayed_values_count, displayed_values_percent and value_order keys, and serialize_truncated() writes that subset as a small sidecar alongside the full histogram. Because the sidecar also carries unique_values and total_count, a histogram loaded from one still reports the totals of the full data even though its own counts are the truncated set.

API

gain.genomic_resources.histogram.load_histogram(resource: GenomicResource, filename: str) → NullHistogram | CategoricalHistogram | NumberHistogram

Load and return a histogram in a resource.

On an error or missing histogram, an appropriate NullHistogram is returned.

class gain.genomic_resources.histogram.NumberHistogram(config: NumberHistogramConfig, bins: ndarray | None = None, bars: ndarray | None = None)

Class to represent a histogram.

add_batch(values: ndarray, weights: ndarray) → None

Add a batch of (value, weight) pairs, vectorized.

Bit-for-bit equivalent to calling add_value() over each pair in order: the same bin selection for both x-scales (the truncation of choose_bin_lin / choose_bin_log and the clamp to number_of_bins - 1), the same below/above out_of_range_bins split, the same min_value/max_value tracking, and the same nan-skip. This is the hot path for the statistics scan, where a per-value Python call dominates the cost.

The equivalence covers dtype as well as arithmetic: the float64 coercion below is what add_value reproduces by normalizing a numpy scalar, so a narrow column – float32, float16, bool – folds the same through either arm (gain#1338).

Where it does not hold: this arm also folds values add_value refuses outright, because asarray converts them silently – a complex loses its imaginary part, and an object array of Decimal converts. Nothing in a scan produces either; they are reachable only by calling this directly.

Both x-scales are vectorized. The log one is bit-exact for the same reason the linear one is and one more: np.log10 returns the same float for a scalar as for that scalar inside an array, at every array width numpy dispatches differently on – checked over many decades by test_add_batch_matches_add_value_loop_log_fuzz, which varies batch size precisely to cover numpy’s scalar-loop and SIMD kernels.

add_value(value: float | generic | None, count: int = 1) → None

Add value to the histogram.

np.generic is in the signature because a numpy scalar is a real caller’s value, not a curiosity (gain#1338).

choose_bin_lin(value: float) → int

Compute bin index for a passed value for linear x-scale.

choose_bin_log(value: float) → int

Compute bin index for a passed value for log x-scale.

static deserialize(content: str) → NumberHistogram

Rebuild a number histogram from serialize() output.

static from_dict(data: dict[str, Any]) → NumberHistogram

Build a number histogram from a dict.

merge(other: Statistic) → None

Merge two histograms.

plot(outfile: IO, score_id: str, y_axis_label: str | None = None, small_values_description: str | None = None, large_values_description: str | None = None) → None

Plot histogram and save it into outfile.

serialize() → str

Render this histogram as the JSON stored in the resource.

to_dict() → dict[str, Any]

Render this histogram as the mapping from_dict() reads back.

Bin edges and bar counts are carried as plain lists rather than arrays, so the result is JSON-serialisable as it stands.

values_domain() → str

The observed value range, rendered for the summary page.

Unlike view_min() / view_max() this reports the values actually seen, so a score whose configured view range is wider than its data still shows the narrower true extent.

view_max() → float

The high edge of the last bin.

The counterpart of view_min(); values above it are counted as out of range.

view_min() → float

The low edge of the first bin.

This is the histogram’s view range, not the score’s observed minimum: values below it are counted in out_of_range_bins rather than binned.

class gain.genomic_resources.histogram.CategoricalHistogram(config: CategoricalHistogramConfig, counter: dict[str | int, int] | None = None, *, truncated: bool = False, unique_values: int | None = None, total_count: int | None = None)

Class for categorical data histograms.

add_batch(values: ndarray, weights: ndarray) → None

Add a batch of (value, weight) pairs, vectorized.

Equivalent to calling add_value() over each pair in order: the same None skip, the same per-value counts, the same TypeError naming the first value that is neither str nor int, and the same HistogramError when the batch takes the histogram past UNIQUE_VALUES_LIMIT. This is the hot path for the statistics scan, where a per-value Python call dominates the cost.

The limit’s message reports UNIQUE_VALUES_LIMIT + 1 because that is the count add_value() always raises at: it tests after every single add, so the first add that exceeds the limit is the one that raises, and the histogram holds exactly one value too many. Which values the counter holds when it raises is not otherwise observable – a histogram that raises is replaced by a NullHistogram carrying the message.

A batch containing BOTH a value of an unusable type and enough new values to trip the limit reports the type failure, wherever the two sit relative to each other; the per-record path reports whichever comes first. The distinction is unreachable through the scan, which batches a str score’s column – every cell of which is a str or the None this skips.

add_value(value: str | int | None, count: int = 1) → None

Add a value to the categorical histogram.

Returns true if successfully added and false if failed. Will fail if too many values are accumulated.

static deserialize(content: str) → CategoricalHistogram

Rebuild a categorical histogram from serialize() output.

property display_values: dict[str | int, int]

Return categorical histogram display values in order.

A truncated instance carries exactly the values its config selected for display at serialization time, so they are returned verbatim – re-running the selection against the truncated counter would compute percentages and orderings against the wrong totals.

static from_dict(data: dict[str, Any]) → CategoricalHistogram

Build a categorical histogram from a dict.

Reads both forms to_dict() and serialize_truncated() produce: the truncation flag and the two totals are absent from a full histogram and default accordingly.

merge(other: Statistic) → None

Merge with other histogram.

plot(outfile: IO, score_id: str, *, y_axis_label: str | None = None, small_values_description: str | None = None, large_values_description: str | None = None) → None

Plot histogram and save it into outfile.

property raw_values: dict[str | int, int]

Every counted value with its count, unordered.

This is the histogram’s own content, as distinct from display_values, which is the ordered subset the summary page draws. A custom plot_function receives the histogram itself and so can read either.

Note that on a histogram loaded back from a truncated sidecar (serialize_truncated()) the counter holds only the values that sidecar carried, so this returns the truncated set – unique_values and total_count are the ones that still describe the full data.

serialize() → str

Render the full histogram as the JSON stored in the resource.

serialize_truncated() → str

Serialize the truncated sidecar form of this histogram.

The sidecar carries the config, the values this histogram’s config selects for display, and the unique_values/total_count totals of the full histogram, marked with "truncated": true. It is the small, always-readable companion of a full histogram whose values file may be absent from a checkout (DVC-tracked, not pulled).

The carried values follow the display selection so the sidecar renders what the full histogram would: every value_order key with its real count for an ordered config, the values covering displayed_values_percent for a percent config, and the displayed_values_count most common values otherwise.

to_dict() → dict[str, Any]

Render this histogram as the mapping from_dict() reads back.

Carries every counted value, not the displayed subset; the truncated companion form is serialize_truncated().

property total_count: int

Sum of all value counts, across truncation.

On a truncated instance this is the total of the full histogram the sidecar was derived from, not the sum of the top-N counter it carries.

property unique_values: int

Number of distinct values, across truncation.

On a truncated instance this is the distinct-value count of the full histogram the sidecar was derived from, not the length of the top-N counter it carries.

values_domain() → str

Render the displayed values, noting truncation when present.

class gain.genomic_resources.histogram.NullHistogram(config: NullHistogramConfig | None)

Class for annulled histograms.

add_value(value: Any, count: int = 1) → None

Discard the value.

A null histogram counts nothing by design, so that a statistics build can feed every score the same way without first asking whether this one has a histogram.

static deserialize(content: str) → NullHistogram

Rebuild a null histogram from serialize() output.

static from_dict(data: dict[str, Any]) → NullHistogram

Build a null histogram from a dict.

merge(other: Any) → None

Do nothing: there are no counts to fold together.

Merging is a no-op rather than an error so that a parallel build can reduce its partial results uniformly, null histograms included.

plot(_outfile: IO, _score_id: str) → None

Draw nothing, leaving outfile untouched.

The caller is expected to render reason in place of the image, rather than to link an image file this never wrote.

serialize() → str

Render this histogram as the JSON stored in the resource.

to_dict() → dict[str, Any]

Render this histogram as the mapping from_dict() reads back.

Only the config survives a round trip, because the reason is the whole of a null histogram’s content.

values_domain() → str

Report that there is no domain, in the other kinds’ place.

class gain.genomic_resources.histogram.NumberHistogramConfig(view_range: tuple[float | None, float | None], number_of_bins: int = 100, x_log_scale: bool = False, y_log_scale: bool = False, x_min_log: float | None = None, plot_function: str | None = None)

Configuration class for number histograms.

static default_config(min_max: MinMaxValue | None) → NumberHistogramConfig

Build a number histogram config from a parsed yaml file.

static from_dict(parsed: dict[str, Any]) → NumberHistogramConfig

Build a number histogram config from a parsed yaml file.

has_view_range() → bool

Whether both ends of the view range are pinned.

A histogram can only be built once its bin edges are known, so a config whose range is half-open still needs the score’s min/max before it can be used.

to_dict() → dict[str, Any]

Transform number histogram config to dict.

class gain.genomic_resources.histogram.CategoricalHistogramConfig(displayed_values_count: int | None = 20, displayed_values_percent: float | None = None, value_order: list[str | int] | None = None, y_log_scale: bool = False, label_rotation: int = 0, plot_function: str | None = None, enforce_type: bool = True, natural_order: bool = False, allow_only_whole_values_y: bool = False)

Configuration class for categorical histograms.

static default_config() → CategoricalHistogramConfig

The config for a score that declares no categorical histogram.

enforce_type=False here does NOT relax value-type checking – CategoricalHistogram.add_value() refuses a non-str/int value either way. What the flag gates is CategoricalHistogram.UNIQUE_VALUES_LIMIT, and it reads backwards: it is the default config that enforces the limit, so a score nobody declared categorical cannot silently histogram thousands of distinct values. A curator who writes an explicit categorical block has asserted the score really is categorical, and that config carries no limit.

static from_dict(parsed: dict[str, Any]) → CategoricalHistogramConfig

Create categorical histogram config from configuratin dict.

to_dict() → dict[str, Any]

Transform categorical histogram config to dict.

class gain.genomic_resources.histogram.NullHistogramConfig(reason: str)

Configuration class for null histograms.

static default_config() → NullHistogramConfig

A null config for a caller with no reason of its own to give.

reason is required rather than optional, so that every null histogram on a summary page can say why it is null; this is the placeholder for the few call sites that genuinely have nothing to add.

static from_dict(parsed: dict[str, Any]) → NullHistogramConfig

Create Null histogram from configuration dict.

to_dict() → dict[str, Any]

Render this config as the mapping from_dict reads back.