gain.genomic_resources package

Subpackages

Submodules

gain.genomic_resources.aggregators module

Score aggregator classes and factory utilities.

class Aggregator[source]

Bases: ABC

Base class for score aggregators.

An accumulator is not shared. An aggregator is mutable state, and every caller in gain builds a fresh one per fold: the folding reads build one per query per call, and the annotators that reduce their own values build one per attribute per call. Nothing outlives the fold it was built for, and build() is cheap enough for that to be the default – a name resolves through a memo (gain#1157).

aggregate() still clears its state first, so an instance CAN be reused single-threaded. It is not thread-safe either way: two threads folding through one accumulator would interleave their values.

add(value: Any, count: int = 1) → None[source]

Add a value to the aggregator, weighted by count.

count is the number of times the value is deemed to occur – the number of base pairs a position-score record spans, for instance. GenomicScore.record_weight is where each kind states its own rule. The weight is applied in closed form: adding a value with a weight of n produces the same result as adding it n times, without doing n units of work, which is what makes folding a region proportional to its records rather than to its length in base pairs. The one exception is mean, which is more accurate weighted than replicated: it rounds once per record rather than once per base. See _add_internal().

aggregate(values: list[Any] | None) → Any[source]: Clear state, add all values, and return the final result.

static build(source: AggregatorDefinition | str | dict[str, Any]) → Aggregator[source]

Build a FRESH aggregator from a definition, string, or dict.

A string is the hot spelling: every aggregating read builds its accumulators anew per call, so the same few names arrive here millions of times per run, and parsing one was ~70% of building it (gain#1157). A name is therefore resolved through _class_and_parameters(), which remembers what a string resolves TO – a class and its parameters, nothing mutable – and never the accumulator built from it.

clear() → None[source]: Reset the aggregator to its initial state.

default_parameter: ClassVar[str | None] = None

get_final() → Any[source]: Return the aggregated result.

get_total_count() → int[source]: Return the total weight seen, None values included.

get_used_count() → int[source]

Return the total weight of the non-None values added.

A weighted total, not a number of records: it is the denominator of the mean, so a value added with a weight of n contributes n to it.

output_value_type: ClassVar[str | None] = None

parametrized: ClassVar[bool] = False

classmethod preserves_domain(*, value_type: str | None = None) → bool[source]

Return True if the output stays in the source value domain.

The rule is element-wise: the output, or each element of a list output, is a value of the source domain. list and most_common return original values, so they preserve it.

static resolve_class(source: AggregatorDefinition | str | dict[str, Any]) → type[Aggregator][source]

The aggregator CLASS a definition, string, or dict names.

For the callers that want what an aggregator WOULD answer rather than an accumulator to answer it with: output_value_type and preserves_domain() are both class-level, so an attribute that only knows an aggregator’s name can describe its output without building one (gain#1133).

It resolves through the same _resolve() as build(), for every spelling and not just the memoised one, so the two cannot disagree about what a source names and a source refused there is refused here, in the same words.

class AggregatorDefinition(aggregator_type: str, parameters: list[Any] = <factory>)[source]

Bases: object

Parsed representation of an aggregator type string.

aggregator_type: str

as_dict() → dict[str, Any][source]: Serialize to a dictionary.

classmethod coerce(source: AggregatorDefinition | str | dict[str, Any]) → AggregatorDefinition[source]

Whichever of the three spellings arrived, as a definition.

An aggregator reaches this module written three ways – a name, a {aggregator_type, parameters} mapping, or an already parsed definition – and every consumer wants the last of those. The cascade that gets there is stated once here so a fourth spelling, or a fix to the parsing rules, is one edit rather than a hunt for the copies. (Aggregator.build() takes the string arm through a memo of its own, but that memo parses with from_string() too – a parsing fix is still one edit.)

classmethod from_dict(data: dict[str, Any]) → AggregatorDefinition[source]: Construct an aggregator definition from a dictionary.

classmethod from_string(raw: str) → AggregatorDefinition[source]

Parse an aggregator definition from a string.

Format: name or name(parameter).

parameters: list[Any]

class BoolAggregator[source]

Bases: Aggregator

Aggregator that returns True if any non-None value was added.

get_final() → bool[source]: Return the aggregated result.

output_value_type: ClassVar[str | None] = 'bool'

class ConcatAggregator[source]

Bases: Aggregator

Aggregator that concatenates all passed values.

One of the three aggregators whose output is genuinely proportional to the aggregated weight (see also join and list). The weight is kept run-length encoded during the scan and expanded only in get_final().

get_final() → Any[source]: Return the aggregated result.

output_value_type: ClassVar[str | None] = 'str'

values: list[tuple[str, int]]

class CountAggregator[source]

Bases: Aggregator

Aggregator that counts values.

get_final() → Any[source]: Return the aggregated result.

output_value_type: ClassVar[str | None] = 'int'

class CounterAggregator[source]

Bases: Aggregator

Aggregator that counts values.

counter: Counter

get_final() → Any[source]: Return the aggregated result.

output_value_type: ClassVar[str | None] = 'object'

class JoinAggregator(separator: str)[source]

Bases: Aggregator

Aggregator that joins all passed values using a separator.

default_parameter: ClassVar[str | None] = ','

get_final() → Any[source]: Return the aggregated result.

output_value_type: ClassVar[str | None] = 'str'

parametrized: ClassVar[bool] = True

values: list[tuple[str, int]]

class ListAggregator[source]

Bases: Aggregator

Aggregator that builds a list of all passed values.

get_final() → Any[source]: Return the aggregated result.

output_value_type: ClassVar[str | None] = 'list'

classmethod preserves_domain(*, value_type: str | None = None) → bool[source]

Return True if the output stays in the source value domain.

The rule is element-wise: the output, or each element of a list output, is a value of the source domain. list and most_common return original values, so they preserve it.

values: list[tuple[Any, int]]

class MaxAggregator[source]

Bases: Aggregator

Maximum value aggregator for genomic scores.

The result is one of the added values, so it has the type of the input. Strings are ordered lexicographically.

get_final() → Any[source]: Return the aggregated result.

output_value_type: ClassVar[str | None] = None

classmethod preserves_domain(*, value_type: str | None = None) → bool[source]

Return True if the output stays in the source value domain.

The rule is element-wise: the output, or each element of a list output, is a value of the source domain. list and most_common return original values, so they preserve it.

class MeanAggregator[source]

Bases: Aggregator

Aggregator for genomic scores that calculates mean value.

get_final() → Any[source]: Return the aggregated result.

output_value_type: ClassVar[str | None] = 'float'

classmethod preserves_domain(*, value_type: str | None = None) → bool[source]

Return True if the output stays in the source value domain.

The rule is element-wise: the output, or each element of a list output, is a value of the source domain. list and most_common return original values, so they preserve it.

class MedianAggregator[source]

Bases: Aggregator

Aggregator for genomic scores that calculates median value.

get_final() → Any[source]: Return the aggregated result.

output_value_type: ClassVar[str | None] = 'float'

classmethod preserves_domain(*, value_type: str | None = None) → bool[source]

Return True if the output stays in the source value domain.

The rule is element-wise: the output, or each element of a list output, is a value of the source domain. list and most_common return original values, so they preserve it.

values: list[tuple[Any, int]]

class MinAggregator[source]

Bases: Aggregator

Minimum value aggregator for genomic scores.

The result is one of the added values, so it has the type of the input. Strings are ordered lexicographically.

get_final() → Any[source]: Return the aggregated result.

output_value_type: ClassVar[str | None] = None

classmethod preserves_domain(*, value_type: str | None = None) → bool[source]

Return True if the output stays in the source value domain.

The rule is element-wise: the output, or each element of a list output, is a value of the source domain. list and most_common return original values, so they preserve it.

class ModeAggregator[source]

Bases: Aggregator

Aggregator for genomic scores that calculates mode value.

get_final() → Any[source]: Return the aggregated result.

classmethod preserves_domain(*, value_type: str | None = None) → bool[source]

Return True if the output stays in the source value domain.

The rule is element-wise: the output, or each element of a list output, is a value of the source domain. list and most_common return original values, so they preserve it.

value_counts: dict[Any, int]

class MostCommonAggregator(k: str | int)[source]

Bases: Aggregator

The k most frequent values, most frequent first.

Configured as most_common(k); k is required and must be a positive integer, given as a string or, from the dict form’s parameters, as an int. Values rank by their weighted frequency, and a tie keeps the order in which the values first appeared – the order of collections.Counter.most_common(). A multi-valued input is flattened as list flattens it, and None is skipped, so the answer is the first k keys of Counter(<the list result>).most_common(k). No values give [].

counter: Counter

default_parameter: ClassVar[str | None] = '3'

get_final() → list[Any][source]: Return the aggregated result.

output_value_type: ClassVar[str | None] = 'list'

parametrized: ClassVar[bool] = True

classmethod preserves_domain(*, value_type: str | None = None) → bool[source]

Return True if the output stays in the source value domain.

The rule is element-wise: the output, or each element of a list output, is a value of the source domain. list and most_common return original values, so they preserve it.

class PositionScoreAggregationQuery(score: str, aggregator: str | None = None, none_value_replacement: ScoreValue | None = None)[source]

Bases: object

The same request over a position score’s expansion (gain#727).

Asks ScoreAggregationQuery’s two questions, plus the one part of a position score’s request that no other kind can ask. none_value_replacement substitutes for every null the per-position expansion holds – uncovered and covered-but-NA alike – before the aggregator sees it; unset, nulls stay inert for every aggregator, all of which already skip None.

A position score answers with a value at every position of the queried region, so a position no record covers is still a position, and a caller may need it to count as a zero rather than go missing. A kind whose records are either in the result or not – a fragment, an allele – has no such position to speak for, and only the covered-but-NA half of the field would ever apply to it. That is why it lives here and not on the neutral query.

A sibling of ScoreAggregationQuery, not a subclass – see that class for why (gain#1302). The two shared fields are repeated in its order, so the three land as the flat dataclass had them (score, aggregator, none_value_replacement) and every positional call site keeps its meaning.

aggregator: str | None = None

none_value_replacement: ScoreValue | None = None

score: str

class ProductAggregator[source]

Bases: Aggregator

Aggregator that multiplies the values it is given.

The output keeps the input’s type – an int score multiplies to an int – and it is None when no non-None value was added.

Past the float range. A product past about 1.8e308 saturates to inf or -inf by the sign of the exact product, an exact int one included: an int stays exact only while a float can hold it. A weighted add(value, count) gives what count calls of add(value) give, in bounded time and memory. A product that holds a 0 is 0, in any order and after a saturation too.

The running product decides the result, as in repeated multiplication: once it passes the float range it stays saturated, and once it underflows it stays 0. The result therefore depends on the input order: (3.0, 1000) then (0.5, 2000) gives inf, the reverse order gives 0.0, and the exact product is about 1e-125.

get_final() → Any[source]: Return the aggregated result.

product: Any

class ScoreAggregationQuery(score: str, aggregator: str | None = None)[source]

Bases: object

One score’s reduction request, in the terms every kind shares.

Names a score and how to reduce it, and nothing else; aggregator of None resolves to the score’s own default from its definition. Kind-neutral because nothing about “reduce this score with this aggregator” depends on how a kind lays its records out, so a position score, a fragment score and an allele score all ask the same thing here.

It deliberately carries no none_value_replacement. That field speaks for a locus NO record covers, which only a kind that reads a value at every position of a region even has – see PositionScoreAggregationQuery, which carries it. Keeping it off the neutral query is what makes the neutral query kind-neutral at all.

PositionScoreAggregationQuery is a SIBLING of this class, not a subclass, though it asks these same two questions and repeats these two fields (gain#1302). A subclass would substitute for this one everywhere – every Sequence[ScoreAggregationQuery] would accept a position query – and the surfaces typed that way answer (score_id, aggregator) PAIRS, which have nowhere to put a replacement: the field would drop, invisibly, because the pair that comes back is exactly the one a caller who never asked for a replacement would get. gain#1121 made it a subclass for what the hierarchy states, and gain#1158 then had to close that door at runtime. As a sibling, mypy closes it: a position query in a list[ScoreAggregationQuery] is a type error, and so is handing a list[PositionScoreAggregationQuery] to a neutral resolver.

aggregator: str | None = None

score: str

class SumAggregator[source]

Bases: Aggregator

Aggregator that sums the values it is given.

The output keeps the input’s type – an int score sums to an int – and it is None when no non-None value was added.

get_final() → Any[source]: Return the aggregated result.

total: Any

aggregator_name(aggregator: AggregatorDefinition | str | dict[str, Any]) → str[source]

The canonical string spelling of an aggregator, whatever its form.

An annotation pipeline may write an attribute’s aggregator as a name, as a {aggregator_type, parameters} mapping, or as an already parsed AggregatorDefinition; a resource may only write the name. Everything downstream of the config – a ScoreDef’s field, a ScoreAggregationQuery’s – holds the name alone, so the spellings collapse here, on the way in.

A name is returned as it stands rather than parsed and printed again. The round trip is exact for every registered aggregator and for the parametrized forms (pinned by test_the_three_aggregator_spellings_collapse_to_one_name), so this is not about the answer differing – it is that a caller holding a malformed name should meet the complaint where its aggregator is BUILT, naming the score, rather than here while a config is being serialised.

get_aggregator_class(aggregator: str) → type[Aggregator][source]: Return the aggregator class for the given aggregator name.

to_float64(value: Any) → float[source]

Convert a numeric aggregation result to a float, saturating.

The one saturation rule for every float store of a sum or a product: an exact int past the float range (about 1.8e308) has no float and becomes inf or -inf by its sign, as a float result past the range already is. The sign comes from a comparison, not from math.copysign: that converts the int and overflows again.

validate_aggregator(aggregator: AggregatorDefinition | str | dict[str, Any], value_type: str | None = None) → None[source]: Raise ValueError for invalid aggregator or value type combinations.

gain.genomic_resources.allele_classification module

What an allele-score row’s ref/alt pair is, as one of five classes.

The classes and the rule that assigns them are vocabulary, defined once in GLOSSARY.md and decided in ADR 0020; classify_allele() states the rule and nothing else here restates it. The classification is what the allele-score statistics count, and it is deliberately independent of them: no accumulator, no storage, no scan – just the rule.

ALLELE_BASES = 'ACGT': The bases an allele may be written with. Anything else – N, a symbolic allele such as <DEL>, the missing-allele * – means the pair does not parse as an allele and is counted as other.

class AlleleClass(*values)[source]

Bases: Enum

The class of an allele-score row’s ref/alt pair (ADR 0020).

COMPLEX = 'complex'

DELETION = 'deletion'

INSERTION = 'insertion'

OTHER = 'other'

SUBSTITUTION = 'substitution'

class AlleleClassification(allele_class: AlleleClass, ref_length: int | None, alt_length: int | None)[source]

Bases: object

A classified ref/alt pair.

allele_class: AlleleClass

alt_length: int | None

property length_change: int | None: Bases the alternative adds over the reference.

ref_length: int | None: Lengths of the two alleles, absent for AlleleClass.OTHER – those strings are not alleles, so they have no allele length.

classify_allele(ref: str | None, alt: str | None) → AlleleClassification[source]

Classify a ref/alt pair as written, VCF-anchored (ADR 0020).

Both alleles are upper-cased first, so a soft-masked lowercase base classifies as the base it masks rather than missing the anchor and inflating complex. What is still not an allele afterwards – N, a symbolic allele, an empty string – is other, and the remaining rules apply in the order the ADR states them:

substitution – strictly one base to one base, the identity pair included;
insertion – anchored: a single reference base that the alternative starts with, adding length_change bases;
deletion – the mirror image, removing -length_change bases;
complex – everything else, MNVs and unanchored indels alike, carrying both lengths.

Total over rows, not merely over strings: a table need not configure a ref or an alt column, and a VCF ALT of . yields a record with no alternative at all, so either allele may arrive as None. A row missing an allele is other – it is still a row, and the class counts of a resource’s rows always sum to its row count. Never raises.

gain.genomic_resources.ann_data_10x module

gain’s readers for the two 10x Genomics formats.

The Matrix Market triple and the 10x-Genomics HDF5, both built on anndata + pandas + h5py + scipy, which gain already depends on. See docs/adr/0014-gain-owns-the-10x-readers.md in the repository for why the work is here rather than delegated to scanpy, and for the parameter surface these readers define.

class TenXH5Parameters(gex_only: bool = False, genome: str | None = None)[source]

Bases: object

The knobs a 10x_h5 resource may set, already validated.

Fewer than the triple’s, because the h5 answers for itself what the triple needs telling: it names its own features, so there is no var_names choice to make and nothing to make unique.

genome: str | None = None

gex_only: bool = False

class TenXMtxParameters(var_names: str = 'gene_symbols', make_unique: bool = True, gex_only: bool = False)[source]

Bases: object

The knobs a 10x_mtx resource may set, already validated.

gex_only: bool = False

make_unique: bool = True

var_names: str = 'gene_symbols'

parse_10x_h5_parameters(parameters: Mapping[str, Any], resource_id: str) → TenXH5Parameters[source]: Validate a 10x_h5 resource’s parameters: block.

parse_10x_mtx_parameters(parameters: Mapping[str, Any], resource_id: str) → TenXMtxParameters[source]

Validate a resource’s parameters: block into the knobs gain has.

An unrecognised key raises rather than being forwarded, so a typo is reported instead of silently doing nothing – and so is a key that used to reach scanpy and now has no meaning here.

read_10x_h5(file_path: str, *, resource_id: str, parameters: TenXH5Parameters | None = None, matrix_free: bool = False) → AnnData[source]

Read a 10x-Genomics HDF5 into an AnnData.

resource_id names the resource in diagnostics only.

With matrix_free, X is an all-zero matrix of the declared shape and IS NOT THE RESOURCE’S DATA. Everything else – both axis tables, the shape, the genome and feature-type filters – is built by the same code as an ordinary read, so anything derived from those is identical. It exists for the statistics build, which reads neither X nor anything computed from it.

read_10x_mtx(matrix_path: str, barcodes_path: str, features_path: str, *, resource_id: str, legacy: bool = False, parameters: TenXMtxParameters | None = None, matrix_free: bool = False) → AnnData[source]

Read a 10x matrix-market triple into an AnnData.

The three members are named outright rather than assembled from a directory and a prefix: which names they carry is a question about the resource’s layout, and the resource is what answers it.

10x writes features as rows and barcodes as columns, so the matrix is transposed into the cells x genes an AnnData carries. legacy marks the CellRanger v2 feature table, which has no feature-type column – and therefore nothing for gex_only to filter on.

resource_id names the resource in diagnostics only.

With matrix_free, X is an all-zero matrix of the declared shape and IS NOT THE RESOURCE’S DATA. Everything else – both axis tables, the shape, the feature-type filter – is built by the same code as an ordinary read, so anything derived from those is identical. It exists for the statistics build, which reads neither X nor anything computed from it.

gain.genomic_resources.ann_data_resource module

Loading helpers for ann_data genomic resources.

class TenXMtxLayout(barcodes: str, features: str, legacy: bool)[source]

Bases: object

The three members of a 10x matrix-market triple, by resource name.

Which names the triple carries is a question about the resource, and resolving it once here is what keeps the read, the statistics inputs and the cache prefetch from each answering it differently.

barcodes: str

features: str

legacy: bool

property sidecars: set[str]: Return the two non-matrix members.

is_10x_matrix_name(file_name: str) → bool[source]: Return whether file_name names a 10x matrix-market member.

load_ann_data_from_resource(resource: GenomicResource | None, *, matrix_free: bool = False) → AnnData[source]

Load an AnnData from an ann_data genomic resource.

The format: key of the resource selects the reader: h5ad, 10x_mtx or 10x_h5. When the key is missing, the suffix of the file: key decides, and h5ad is the default. The data files must be reachable as local files.

The caller owns the file handle. An h5ad file is opened read-only and backed by the file, so the AnnData keeps the file open. Call ann_data.file.close() when ann_data.isbacked is true. The two 10x formats are read into memory and hold no handle.

Parameters:

resource (GenomicResource or None) – The resource to load. Its type must be ann_data. None is an error, not an empty result.
matrix_free (bool, default False) – Skip the data matrix. It applies to the 10x_mtx and 10x_h5 formats only, and h5ad ignores it. The X of the result is a matrix of zeros of the right shape, not the data of the resource. obs and var are the same as in an ordinary read. Do not use this option to read values.

Returns:

The annotated matrix. The rows of obs are the cells and the rows of var are the genes.

Return type:

anndata.AnnData

Raises:

ValueError – If resource is None, is not of type ann_data, has no file: key, names an unknown format:, cannot reach a data file as a local file:// url, or has a parameters: block that makes the reader return something other than an AnnData. Each message except the None one names the resource.

load_ann_data_from_resource_id(resource_id: str, grr: GenomicResourceRepo | None = None) → AnnData[source]

Load an AnnData from an ann_data resource id.

The caller owns the file handle, as for load_ann_data_from_resource().

Parameters:

resource_id (str) – The id of the resource in the repository.
grr (GenomicResourceRepo or None, default None) – The repository to read from. None builds the default repository with build_genomic_resource_repository().

Returns:

The annotated matrix, as load_ann_data_from_resource() returns it.

Return type:

anndata.AnnData

Raises:

ValueError – As load_ann_data_from_resource() raises it.

resolve_10x_layout(manifest: Manifest, file_name: str) → TenXMtxLayout[source]

Return the layout of the 10x matrix file_name, per the manifest.

Resolution is manifest-driven for the same reason the tabix index is: the manifest is already loaded and is protocol-agnostic, whereas probing costs a network round trip per candidate on http and s3.

This and resolve_10x_layout_for_read() are the pair, and they differ only in where the answer comes from – exactly as resolve_tabix_index_filename() and its _for_read twin do. A caller that only wants the two names takes .sidecars.

resolve_10x_layout_for_read(resource: GenomicResource, file_name: str) → TenXMtxLayout[source]

Return the layout to read file_name with, never building.

Shaped like resolve_tabix_index_filename_for_read(), and for the same reason: a read must stay a pure read, and GenomicResource.get_manifest() would build – md5-scanning the whole resource and writing state files – for a resource that carries no .MANIFEST (gain#430). With no manifest at hand, falls back to probing the resource itself for the same two markers.

resolve_ann_data_format(config: Mapping[str, Any]) → str[source]

Return the format an ann_data config is read as.

A declared format: wins; otherwise the file: suffix decides, and a name matching none of them falls back to h5ad. A config with no file: gets the fallback too – the missing key is the loader’s error to report, and this is also reached from the statistics hash, which degrades rather than raising.

The loader and the statistics hash both resolve through here so that the format a resource is read as and the format its hash records cannot disagree. They did: the hash used to state the h5ad fallback outright, so an explicit format: h5ad added to a 10x config changed the read without changing the hash, and the statistics never rebuilt.

gain.genomic_resources.bigwig_scores module

Reading a bigWig’s single value column as a genomic score.

Everything the score layer knows about bigWig, in one module: how a bigWig’s score definitions are finished off, what a bigWig resource is allowed to configure, and how the one value is read off a record. Symmetric to vcf_scores, and for the same reason – these are all statements about one backend’s peculiarities, and they belong together rather than scattered through genomic_scores.

Reject what corrupts values; warn about what is merely inert. That is the principle that decides which of the two halves below a piece of misconfiguration falls into, and it is worth stating because bigWig attracts a lot of config that does nothing. A bigWig is a binary format with a fixed layout: one numeric value per interval, no columns, no header, no text. So a resource that declares a second score, or a score type: of int, or a column index: other than the deprecated 3, is asking for a value the file cannot give – it would be truncated, or read from a column that does not exist – and validate_bigwig_scoredefs() refuses to open it, naming the resource and the score. A resource that declares chrom:/pos_begin:/pos_end: column blocks, or a header:, is merely describing a tabular file that this one is not; nothing reads those keys, no value changes because of them, and genomic_position_table.utils warns and ignores them.

The bigWig table itself is not here and does not belong here. genomic_position_table.table_bigwig produces records: it owns the payload’s shape – which, since this module exists, is the bare value. This module says what that value means as a score. That is the same seam vcf_scores draws, and it is why the table layer still imports nothing from the score layer.

build_bigwig_scoredefs(config: dict[str, Any], config_scoredefs: dict[str, GenomicScoreDef]) → dict[str, GenomicScoreDef][source]

Finish a bigWig resource’s score definitions.

Currently a pass-through, and kept as the seam rather than deleted: it is where GenomicScore._build_scoredefs routes a bigWig, matching the VCF branch beside it, so bigWig-specific definition work has an obvious home.

It used to empty the default ``na_values``, and that is worth recording because the reasoning was right and the decision still wrong. A float score defaults to the sentinels ("", "nan", ".", "NA") – four TEXT tokens, which exist because a tabular backend hands the score layer strings. A bigWig hands it a float, which can never equal any of them, so on this backend the set is dead config.

Emptying it changed nothing at runtime but everything to GenomicScoreImplementation.calc_statistics_hash, which folds na_values in verbatim: every one of the 150 deployed bigWig resources would have gone stale and been rescanned – ~74.7 G records, hours of compute – to arrive at byte-identical statistics. Verified against a deployed resource’s stored stats_hash: na_values was the ONLY field that differed.

The runtime saving it was reaching for is kept, and taken from the data instead: value_extraction.select_value_extractor binds the identity read unless the NA set holds a sentinel a float could actually match. A text-only set – which is every unconfigured bigWig score – takes the identity path either way.

extract_bigwig_value(record: tuple[Any, ...], score_def: GenomicScoreDef) → str | int | float | bool | None[source]

Read the one score off a bigWig record: the payload IS the value.

A true identity function, and that is the whole point. A bigWig stores a float per interval and pyBigWig hands it up as a Python float; a bigWig score is declared type: float (anything else is refused at open); and the NA default for such a score is empty (see build_bigwig_scoredefs()). So every step GenomicScoreDef.parse_value() would perform on this value – the NA membership test, the float(value) reparse – is provably a no-op, and this reads the value out of the record and returns it.

score_def is unused and stays in the signature because ValueExtractor is what GenomicScore.open routes to; a bigWig-shaped extractor with a different arity would need a call-site branch per record to invoke, which is the cost this removes.

The variant that does consult the definition is extract_bigwig_value_na(), and open() picks between the two once, from whether the score configures any NA sentinels at all.

extract_bigwig_value_na(record: tuple[Any, ...], score_def: GenomicScoreDef) → str | int | float | bool | None[source]

extract_bigwig_value(), plus the configured NA-sentinel check.

Bound instead of the identity for a score whose na_values is non-empty – which, since the default is empty for bigWig, means a resource that explicitly configured sentinels (na_values: "-1" is the deployed shape). The sentinel set carries both the text and the parsed form of each sentinel (see normalize_na_values), so a numeric payload matches by value.

Still no parse: a sentinel match yields the null score, and anything else is the payload unchanged. The choice between this and the identity is made once per open, never per record.

validate_bigwig_scoredefs(resource_id: str, score_defs: dict[str, GenomicScoreDef]) → None[source]

Refuse a bigWig score config that cannot mean what it says.

Called from GenomicScore.open before the table is opened, in the same slot as the extractor routing: a refusal that costs no file handle cannot leak one, and both of this function’s inputs are known at construction, so nothing here needs the handle.

Deliberately NOT called from __init__. GenomicScoreImplementation.__init__ builds its score eagerly, so a constructor that refused would make a misconfigured bigWig resource impossible to list or describe – and listing and describing it is precisely what grr_manage has to do in order to report it. Refusing at open leaves the resource inspectable and stops only the read.

What is refused, and why each one corrupts values rather than merely sitting there:

more than one score – a bigWig file carries one value per interval. A second score has no second value to read, so both scores would read the same number under different names.
a ``type:`` other than ``float`` – pyBigWig hands up a Python float, and the value read is an identity (see extract_bigwig_value()). int would silently truncate every value; str/bool would let a float through wearing the wrong declared type, and the declared type is what the aggregators, the histograms and the bulk read path all branch on.
a column address other than the deprecated 3 – there are no columns to address. Under the old four-element payload, 0/1/2 read the contig and the two coordinates and >= 4 was out of range; either way a resource that names one is asking for something that is not the score.
a column NAME – a bigWig has no header, so there is nothing to resolve the name against.

index: 3 is the one exception, accepted with a deprecation notice at DEBUG – all 150 deployed bigWig resources carry it, so anything louder fires for every one of them on every open: see DEPRECATED_VALUE_INDEX.

A score that declares no type: at all is let through. That is not a declaration of a non-float type – it is the same absent-type config every backend accepts, and it corrupts nothing: the value still arrives as the float the file holds.

gain.genomic_resources.cached_repository module

Provides caching genomic resources.

class CacheResource(resource: GenomicResource, protocol: CachingProtocol)[source]

Bases: GenomicResource

Represents resources stored in cache.

class CachingProtocol(remote_protocol: ReadOnlyRepositoryProtocol, local_protocol: FsspecReadWriteProtocol, public_url: str | None = None)[source]

Bases: ReadOnlyRepositoryProtocol

Defines caching GRR repository protocol.

classify_cached_resource_file(resource: GenomicResource, filename: str) → FileCacheVerdict[source]

Classify a resource file without taking any lock or downloading.

The lock-free decision half of refresh_cached_resource_file(): it resolves the remote resource and delegates to the local protocol’s classify_resource_file(). See gain#78.

download_cached_resource_file(resource: GenomicResource, filename: str, *, on_bytes: Callable[[int], None] | None = None) → tuple[str, str][source]

Download a resource file into cache unconditionally.

Takes the per-file lock and copies the file regardless of its local state – the decision was already made by classify_cached_resource_file(). See gain#78.

file_exists(resource: GenomicResource, filename: str) → bool[source]: Check if given file exist in give resource.

get_all_resources() → Generator[GenomicResource, None, None][source]: Return generator for all resources in the repository.

get_all_resources_dict() → dict[str, GenomicResource][source]: Return dictionary for all resources in the repository.

get_public_url() → str[source]

Return the public base URL of the repository.

Returns:: URL or path string pointing to a public repository root

get_resource_file_url(resource: GenomicResource, filename: str) → str[source]: Return url of a file in the resource.

get_resource_url(resource: GenomicResource) → str[source]: Return url of the specified resources.

get_url() → str[source]

Return the base URL of the repository.

Returns:: URL or path string pointing to repository root

invalidate() → None[source]: Invalidate internal cache of repository protocol.

load_manifest(resource: GenomicResource) → Manifest[source]: Load resource manifest.

open_bigwig_file(resource: GenomicResource, filename: str) → Any[source]

Open a bigwig file in a resource and return it.

Not all repositories support this method. Repositories that do no support this method raise and exception.

open_fasta_file(resource: GenomicResource, filename: str, index_filename: str | None = None, compressed_index_filename: str | None = None) → FastaFile[source]

Open a bgzipped fasta file in a resource and return a FastaFile.

Not all repositories support this method. Repositories that do not support this method raise an exception.

open_raw_file(resource: GenomicResource, filename: str, mode: str = 'rt', **kwargs: str | bool | None) → IO[source]: Open file in a resource and returns a file-like object.

open_repository_metadata() → Connection[source]: Open the db file for repo metadata and return the connection.

open_tabix_file(resource: GenomicResource, filename: str, index_filename: str | None = None) → TabixFile[source]

Open a tabix file in a resource and return a pysam tabix file.

Not all repositories support this method. Repositories that do no support this method raise and exception.

open_vcf_file(resource: GenomicResource, filename: str, index_filename: str | None = None) → VariantFile[source]

Open a vcf file in a resource and return a pysam VariantFile.

Not all repositories support this method. Repositories that do no support this method raise and exception.

refresh_cached_resource(resource: GenomicResource) → tuple[str, None][source]: Refresh all resource files in cache if neccessary.

refresh_cached_resource_file(resource: GenomicResource, filename: str) → tuple[str, str][source]: Refresh a resource file in cache if neccessary.

class GenomicResourceCachedRepo(child: GenomicResourceRepo, cache_url: str, **kwargs: str | None)[source]

Bases: GenomicResourceRepo

Defines caching genomic resources repository.

Carries the id of the repository it wraps, unchanged: a cache decides how a repository’s resources are served, not what the repository is called. The caching layer treats its other identities the same way – CachingProtocol.get_url() and get_public_url() both report the remote’s.

The id used to be f"{child.repo_id}.caching_repo", which renamed a repository an operator had already named, and renamed it after the definition was validated: the ids check_child_ids_are_unique reasoned about were then not the ids the repositories were built with, so a definition pairing {"id": "a", "cache_dir": ...} with {"id": "a.caching_repo"} passed validation and built two repositories sharing one id. See #447.

find_resource(resource_id: str, version_constraint: str | None = None, repository_id: str | None = None) → GenomicResource | None[source]

Return requested resource or None if not found.

Mirrors get_resource: the child resolves the resource (and owns repository_id semantics), then the hit is wrapped so the returned resource is cache-backed.

Forwarding repository_id unchanged is what makes this repository answer to its own id: it carries the child’s id (#447), so a filter naming this repository names the child too, and the child self-names – a leaf protocol repo by comparing the filter against its own id, a group by dropping the filter it matches itself. There is no separate self-match to keep in step here.

This used to enumerate every resource and pick the highest version across protocols, which had two defects. It filtered repository_id against the cache protocol’s id – registered as f"{proto_id}.cached" – so the filter never matched anything. And because get_resource already delegated, the two methods could return different versions of the same id when a group repository’s children overlapped. Group child order is a priority list; first child wins. See #429.

get_all_resources() → Generator[GenomicResource, None, None][source]

Return a generator over all resources of the repository.

Read the generator lazily. A repository can be large.

Yields:: GenomicResource – A resource of the repository. A repository without resources yields nothing.

get_resource(resource_id: str, version_constraint: str | None = None, repository_id: str | None = None) → GenomicResource[source]

Return the resource with the given id.

Use this method when the id is a constant and its absence is a bug. Use find_resource() when absence is an ordinary result.

Parameters:

resource_id (str) – The id of the resource, without the version.
version_constraint (str or None, default None) – A constraint such as ">=2.0" or "=2.1". None selects the latest version.
repository_id (str or None, default None) – Restricts the lookup to the repository with this id, anywhere in the tree of this repository. An empty string or None means no restriction.

Returns:

The matching resource. The method never returns None.

Return type:

GenomicResource

Raises:

FileNotFoundError – If no resource matches and the repository is backed by a protocol (directory, http, s3, in-memory).
ValueError – If no resource matches and the repository is a group repository.

get_resource_cached_files(resource_id: str) → set[str][source]: Get a set of filenames of cached files for a given resource.

invalidate() → None[source]

Drop the cached state of the repository.

The next access loads the resource list, the metadata and the file contents again.

search_resources(search_term: str | None = None, resource_type: str | None = None, resource_query: str | None = None) → Generator[GenomicResource, None, list[tuple[str, str]] | None][source]

Search the resources by a full-text term, a type or a query.

The method returns a generator, so it reads the repository lazily. A resource must match every filter that you supply.

Parameters:

search_term (str or None, default None) – A full-text term. None means no full-text filter.
resource_type (str or None, default None) – A resource type, for example position_score. None means no type filter.
resource_query (str or None, default None) – A wildcard query over the ids and the labels of the resources. None means no query filter.

Yields:

GenomicResource – A matching resource. With no filter, every resource matches.

Returns:

The generator returns, when it ends, a (repository id, reason) pair for each child that a group repository skipped while it still answered. None and an empty list both mean that nothing was skipped. A for loop discards this value.

Return type:

list of tuple of (str, str) or None

Raises:

SearchTermError – If the search term cannot be parsed.
SearchIndexUnavailableError – If the repository has no usable search index.

cache_resources(repository: GenomicResourceRepo, resource_ids: Iterable[str] | None, workers: int | None = None, *, progress: bool = True) → None[source]: Cache resources from a list of remote resource IDs.

gain.genomic_resources.cli module

Provides CLI for management of genomic resources repositories.

class CommandResult(needs_update: int = 0, failed: frozenset[str] = frozenset({}), repo_failed: bool = False, wrote: bool = False)[source]

Bases: object

What a repository-management command found and what it could not do.

Three outcomes, deliberately kept apart (gain#364):

needs_update – how many resources are OUT OF DATE. Only a --dry-run reports this; a real run repairs them instead of counting them. This is the meaning the plain int these commands used to return carried.
failed – the ids of the resources that are BROKEN: whatever GAIn was asked to do to them raised, or silently did not happen. A run collects these rather than aborting on the first one, so a single broken resource cannot stop the healthy ones from being repaired.
repo_failed – something failed that no single resource can be blamed for: the repository’s own configuration, or a statistics task graph whose failure the per-resource check could not pin on any resource. Inventing a resource id for it would be a lie, but it still has to make the run exit non-zero.

An int could express only the first, which is why non-dry-run repair was structurally incapable of reporting failure.

wrote is not a fourth outcome but a fact about the run: whether it changed anything on disk. The resource-scoped commands read it to note that the repository-global artifacts are now behind the resources they describe (gain#760). It is set by the command layer, not reported by the protocol, so a new write site on a resource-scoped path must set it, or the note silently stops firing for that write.

failed: frozenset[str] = frozenset({})

property has_failures: bool: Whether anything failed at all, attributable or not.

needs_update: int = 0

repo_failed: bool = False

wrote: bool = False

class ManifestOutcome(updates_needed: dict[str, bool], failed: frozenset[str], wrote: bool)[source]

Bases: NamedTuple

What a manifest pass over a set of resources found.

updates_needed is keyed by the resources the pass got through, and valued by whether that resource’s manifest is stale. failed names the resources whose manifest could not be built at all - today, only a resource whose content drifted from its .dvc sidecars under --without-dvc; it has NO entry in updates_needed (#373). wrote is whether the pass saved any manifest.

failed: frozenset[str]: Alias for field number 1

updates_needed: dict[str, bool]: Alias for field number 0

wrote: bool: Alias for field number 2

cli_browse(cli_args: list[str] | None = None) → None[source]: Provide CLI for repository browsing.

cli_manage(cli_args: list[str] | None = None) → None[source]: Provide CLI for repository management.

gain.genomic_resources.cli_cache_repo module

CLI for caching genomic resources referenced by an annotation pipeline.

The tool resolves a GenomicResourceRepo and an AnnotationPipeline from a combination of command-line flags and the registered genomic context providers, then caches the resources the pipeline depends on.

The annotation pipeline can come from:

the positional pipeline argument — a file path or a GRR resource id of type annotation_pipeline. This is supplied by the standard CLIAnnotationContextProvider (the same mechanism annotate_tabular / annotate_vcf use).
-i / --instance — when the GPFInstanceContextProvider plugin is installed, this resolves to the pipeline of the configured GPF instance.

When both are supplied, the positional pipeline wins (it is supplied by a higher-priority context provider); the tool logs which source it used. When neither is supplied (positional omitted / left at its "context" sentinel and no instance pipeline available), the tool logs a warning and exits cleanly without caching anything.

cli_cache_repo(argv: list[str] | None = None) → None[source]: Cache genomic resources used by an annotation pipeline.

gain.genomic_resources.cli_dvc module

What grr_manage refuses about a .dvc sidecar, before it starts.

Its own module because the refusal has two gates that must say the same thing: this pre-flight, which runs before a command touches the repository, and the manifest builder’s own check in repository.collect_dvc_entries, which is what makes a manifest impossible to build from a sidecar GAIn cannot verify (#255, #284). The error they raise and the message they say it with live below both, in dvc and repository, so the builder’s gate does not depend on the CLI layer (#721); this module keeps the pre-flight.

refuse_dvc_directory_outputs(proto: ReadWriteRepositoryProtocol, resources: Sequence[GenomicResource]) → None[source]

Refuse a dvc add <dir> output before the command writes anything.

The gate in repository.collect_dvc_entries fires where ONE resource’s manifest is built, which is far too late for a command that spans a repository: by then the resources ordered before the offender have their .MANIFEST and .grr state written, and *-stats / *-info have run a whole task graph – so a run that refuses the repository could still leave statistics and info pages behind for the resources it happened to reach first (#284). A refusal must be side-effect-free, so the sidecars are read up front, here, and the command fails before it touches anything.

Scoped to the resources the command SELECTED, which for the repo-* subcommands is the whole repository: a resource GAIn refuses does not make a resource-* command on some OTHER resource illegal, in keeping with one broken resource never stopping work on the healthy ones (gain#503).

A sidecar this pass cannot read or cannot parse is not its business: it asks one question – does any sidecar describe a directory? – and repository.collect_dvc_entries remains the one place that reports an unusable sidecar, so a run does not warn about it twice. Neither is a resource whose files cannot even be LISTED – an unreadable directory, a DVC cache this run may not traverse, a remote store that fails to describe a key it just listed. It is skipped for the same reason: the command’s own per-resource handler is what reports it and fails that resource alone, and a pre-flight that raised instead would take the whole run down over one broken resource – the very failure mode gain#503 removed.

It costs one extra listing of the selected resources, since the listing is what says which sidecars exist and the answer is needed before the first write. Only the sidecars are then read, and a .dvc file is a few hundred bytes of YAML – nothing next to the hashing and statistics the pass protects.

Raises:: UnsupportedDvcDirectoryOutputError – some selected resource has a dvc add <dir> output.

gain.genomic_resources.cli_errors module

How grr_manage reports a failure that belongs to ONE resource.

Its own module because more than one command needs it and they must agree: a repository-wide command that dies on the first bad resource reports half a repository and hides the rest (gain#364, gain#503).

It is imported from the statistics package (statistics.region_fold), so it may import nothing of GAIn beyond the logging shim and the leaves it names an exception from – anything more can reach histogram, which imports that package’s base class, and close a cycle. The architecture suite pins the allowlist and says what the cycle is (gain#1293).

report_resource_failure(err: Exception, action: str, resource_id: str) → None[source]

Report a failed operation on one resource, at the right tier.

action names what could not be done – never the phase the failure happened in. A handler that wraps several operations cannot know which one raised, and naming the wrong one sends the reader looking in the wrong place (gain#364); the cause, which is always carried, says it.

gain.genomic_resources.cli_list module

The grr_manage list command.

Split out of cli because listing is the one read-only command in that module: everything else there manages a repository – rebuilds manifests, statistics and info pages – while this only describes what is already there. Keeping it apart also keeps the reporting policy it needs (a bad resource is named and skipped, never fatal) from being read as the management commands’ policy, which is to fail the run.

run_list_command(proto: ReadOnlyRepositoryProtocol | GenomicResourceRepo, args: Namespace) → None[source]: List the resources of a repository.

gain.genomic_resources.data_frame_resource module

Loading helpers for data_frame genomic resources.

load_data_frame_from_resource(resource: GenomicResource | None) → DataFrame[source]

Load a pandas DataFrame from a data_frame genomic resource.

The format: key of the resource selects the reader: csv (the default), tsv or excel. The parameters: block passes keyword arguments to the pandas reader. The data are read at once, and the result holds no open file.

Parameters:: resource (GenomicResource or None) – The resource to load. Its type must be data_frame. None is an error, not an empty result.
Returns:: The table of the resource. A file with a header and no rows gives a DataFrame with columns and no rows. A file with no content at all raises the error of the pandas reader.
Return type:: pandas.DataFrame
Raises:: ValueError – If resource is None, is not of type data_frame, has no file: key, names an unknown format:, or has a parameters: block that makes pandas return something other than a DataFrame, such as chunksize or sheet_name: null. Each message except the None one names the resource.

load_data_frame_from_resource_id(resource_id: str, grr: GenomicResourceRepo | None = None) → DataFrame[source]

Load a pandas DataFrame from a data_frame resource id.

Parameters:

resource_id (str) – The id of the resource in the repository.
grr (GenomicResourceRepo or None, default None) – The repository to read from. None builds the default repository with build_genomic_resource_repository().

Returns:

The table of the resource, as load_data_frame_from_resource() returns it.

Return type:

pandas.DataFrame

Raises:

ValueError – As load_data_frame_from_resource() raises it.

gain.genomic_resources.draw_score_histograms module

exception ScorelessResourceError[source]

Bases: TypeError

A resource whose type carries no scores at all.

Distinguished from the errors that mean a resource is broken: there is nothing wrong with a genome, it simply has no histograms to draw. A TypeError because that is what selecting such a resource by id has always raised, and callers still get exactly that.

main(argv: list[str] | None = None) → None[source]: Liftover dae variants tool main function.

parse_cli_arguments() → ArgumentParser[source]: Create CLI parser.

gain.genomic_resources.dvc module

The .dvc sidecar vocabulary.

dvc add <file> drops a <file>.dvc sidecar next to the data file it stores and gitignores the file itself. GAIn reads those sidecars in three places – the repository scan, grr_manage’s entry collection and the manifest builder – and this module is the only place that interprets one, so the three can never classify the same sidecar differently.

Recognising a sidecar by name is part of that vocabulary: naming it here keeps the info pages, which hide sidecars, and the manifest builder, which reads them, from ever disagreeing about which files are sidecars.

class DvcContentDrift(name: str, content_md5: str, dvc_md5: str)[source]

Bases: object

A materialised file whose content disagrees with its .dvc.

Produced only by the verifier (grr_manage --without-dvc), which is the one mode that reads a DVC-managed file’s bytes (#373).

content_md5: str

dvc_md5: str

name: str

exception DvcContentDriftError(resource_id: str, drifts: Sequence[DvcContentDrift])[source]

Bases: ValueError

Every file of ONE resource whose content drifted from its sidecar.

Collected rather than raised on the first offender, so that a single grr_manage --without-dvc run reports all of them (#373). It is a ValueError because it is a fault of the RESOURCE, and cli_errors.report_resource_failure reports those as one line carrying the cause, with the traceback demoted to DEBUG (gain#364).

exception UnsupportedDvcDirectoryOutputError[source]

Bases: Exception

A resource declares a dvc add <dir> output, which GAIn refuses.

Raised by cli_dvc.refuse_dvc_directory_outputs and by repository.collect_dvc_entries, and turned into a non-zero exit by cli.cli_manage (#255). Defined here, below both, so that the manifest builder’s gate does not depend on the CLI layer (#721).

dvc_sidecar_target(name: str) → str[source]: Return the path the sidecar name describes.

is_dvc_directory_out(out: dict[str, Any]) → bool[source]

Return True if a .dvc output describes a dvc add <dir> output.

DVC writes two signals for a directory output, and either one on its own is enough to recognise it:

its md5 is the hash of a DVC cache object - a listing of the directory’s files - and carries a .dir suffix to say so;
it declares nfiles, the number of files in the directory.

Neither is checked in isolation: an out that lost its nfiles is still a directory, and so is one whose md5 sum lost its suffix. GAIn does not support directory outputs - it cannot verify a .dir md5 sum against anything it can read - so grr_manage refuses a resource that has one (#255).

is_dvc_sidecar(name: str) → bool[source]: Return True if name names a .dvc sidecar.

parse_dvc_pointer_out(content: str | bytes, basename: str) → dict[str, Any] | None[source]

Parse a .dvc sidecar; return the output entry describing basename.

A well-formed .dvc pointer is a mapping with an outs list of mappings; the output that describes basename is the one whose path equals it exactly. Anything else - a mapping without outs, an outs that is not a list of mappings, an output for some other path, YAML that does not parse, non-UTF-8 bytes - is not a pointer for basename and yields None.

Parsing NEVER raises. This is the single place a .dvc file is interpreted, so that the repository scan (_is_dvc_managed_leaf, which must never abort on stray content) and repository.collect_dvc_entries cannot classify the same sidecar differently (#251).

Parameters:

content – raw content of the .dvc file; bytes are safe to pass - yaml decodes them itself, so a binary file cannot raise a UnicodeDecodeError past this function.
basename – the base name of the data file the pointer must describe.

Returns:

The matching outs entry, or None if the content is not a pointer for basename. The entry is NOT validated beyond its path: callers that need an md5 sum and a size must check for them.

gain.genomic_resources.fsspec_protocol module

Provides GRR protocols based on fsspec library.

exception ChecksumMismatchError[source]

Bases: RetryableCopyError

A completed download whose md5 disagrees with the manifest.

Almost always a truncated or corrupted transfer.

exception CorruptedPublishError[source]

Bases: RetryableCopyError

A published object that differs in size from the verified download.

The download’s byte-count and md5 checks both run before the move, so they describe the temp file, not the object the move landed. Since the digest is carried across rather than recomputed from the store (gain#865), a move that published something else – a partial object-store copy, a rename that lost the tail, one that left no object at all – would be recorded under the digest of bytes that are no longer there: size and timestamp come from the published object and agree with it, the md5 comes from the download and agrees with the manifest, so every later cache verdict passes and the file is served corrupt indefinitely (gain#880).

Only size-changing corruption is caught. Catching the rest needs a second full read of every downloaded file – for a remote store, a second transfer – which is exactly the cost gain#865 removed.

Raised on a second, digest-free path since gain#933: the repository’s own artifacts publish through _publish_file(), which compares the published object against the bytes it staged rather than against a manifest. The reasoning above is about the download, but the guard is the same one – it lives on the move, so everything that moves gets it. The retryability inherited from RetryableCopyError does not reach that second path, though: its callers publish once and propagate, so a corrupting publish of a repository artifact fails rather than retrying.

class FileCacheVerdict(needs_download: bool, size: int)[source]

Bases: NamedTuple

The lock-free classification of a single resource file.

needs_download is True when the local copy is missing or has drifted from the remote manifest and must be (re)downloaded; size is the manifest-recorded byte size of that pending download (0 when nothing needs downloading). See gain#78.

needs_download: bool: Alias for field number 0

size: int: Alias for field number 1

class FsspecReadOnlyProtocol(*args: Any, **kwargs: Any)[source]

Bases: ReadOnlyRepositoryProtocol

Provides fsspec genomic resources repository protocol.

(proto_id, url) names ONE protocol instance for the life of the process: __new__ memoizes it in _FSSPEC_PROTOCOLS, keyed on the url’s canonical scheme://netloc/path form so its spelling cannot split one repository in two, and never evicts. A second build over that pair therefore reaches the object every earlier caller is already holding, and Python re-runs __init__ on it.

That makes a rebuild a refresh – it drops the resource memo, which is how a caller that has just changed a repository reads it back. It is not a reconfiguration: a rebuild asking for a different mode, public url or credentials is refused rather than applied to the incumbent. See docs/adr/0005-fsspec-protocol-memo-rebuild.md (#514).

Construction is one atomic step, and a protocol is reachable through the memo only once the whole of it has returned (#527). __new__ records an in-flight construction instead of publishing, so a thread that arrives while another is building that key waits and is answered with the same, configured instance – rather than either building a second protocol over a key that names one, or reading one whose filesystem, url, public_url and kwargs are not bound yet.

close() → None[source]: Close the genomic resource.

file_exists(resource: GenomicResource, filename: str) → bool[source]: Check if given file exist in give resource.

get_all_resources() → Generator[GenomicResource, None, None][source]: Return generator over all resources in the repository.

get_all_resources_dict() → dict[str, GenomicResource][source]

Return the repository’s resources, keyed by full id.

The whole memo protocol – the lock, the check-then-populate, the keying, the ordering and the return – lives here and only here, for every fsspec protocol. A subclass enumerates the repository by overriding _enumerate_resources and inherits the rest (#515).

FsspecReadWriteProtocol used to carry a second copy of all of it and differ only in the enumeration, which is how the release-then-read defect of #458 came to be in two places while the report named one.

get_file_content(resource: GenomicResource, filename: str, *, uncompress: bool = True, mode: str = 't') → Any[source]

Return content of a file in given resource.

Overrides the base, which opens through open_raw_file and reads on the handle it returns: that open is redacted and the read is not, so on an authed GRR a failure mid-read surfaces the credential-bearing fetch url verbatim (gain#1058). _read_fetch_file covers both.

This sits IN FRONT of the fasta-index copy gain#1017 fixed – ReferenceGenome.open reads the .fai through here first – and also backs load_manifest and load_yaml.

get_loaded_manifest reads a missing manifest’s FileNotFoundError as “no manifest”, and the rebuild keeps that type: it reconstructs via type(exc)(message), which FileNotFoundError supports.

An error that CANNOT be reconstructed from a message alone loses its type – an aiohttp.ClientResponseError (an HTTP 5xx, which fsspec does not translate) needs request_info and history. It comes back an OSError, or a RetryableCopyError where the original was transient, which _rebuild_error_without_url_credentials preserves so redaction cannot change a retry decision. Safe here for the reason _copy_resource_file_to_local sets out: no retry or control-flow decision on this path keys off the type.

uncompress names nothing on this path and never has, so compression=None preserves the as-stored read exactly – see _copy_resource_file_to_local, which found the same. Repairing the dead parameter is separate work.

get_public_url() → str[source]

Return the public base URL of the repository.

Returns:: URL or path string pointing to a public repository root

get_resource_url(resource: GenomicResource) → str[source]

Return url of the specified resources.

The resource id is the other operand of this join and is no less untrusted than a file name – on the remote path it is read verbatim out of the repository’s .CONTENTS.json.gz – so it is contained here, at the join, exactly as get_resource_file_url contains the name (gain#467).

get_url() → str[source]

Return the base URL of the repository.

Returns:: URL or path string pointing to repository root

invalidate() → None[source]

Drop the memoized resources, leaving handed-out ones alone.

Clears only this protocol’s own cache. The resources in the memo are handed out by reference, so unbinding their proto on the way out – as this used to do – breaks the objects live callers are already holding, and they raise AttributeError on None at first use (#513). Their lifetime is the caller’s business; dropping the memo is enough to make the next read reload, and enough to let a resource no one else holds be collected, since the memo held the only reference to it. Not the protocol, though – _FSSPEC_PROTOCOLS memoizes every protocol for the life of the process and never evicts, so no amount of unbinding here ever released one.

kwargs: dict[str, Any]

load_contents() → list[dict[str, Any]][source]: Load the content JSON of the repository.

load_manifest(resource: GenomicResource) → Manifest[source]: Load resource manifest.

md5_contents() → str[source]: Calculate md5 hash of the repository content.

open_bigwig_file(resource: GenomicResource, filename: str) → Any[source]

Open filename of resource with pyBigWig.

A file GRR is opened by path. An s3, http or https GRR is opened by url – presigned for s3 – which needs a pyBigWig built with libcurl (pyBigWig.remote == 1); the PyPI wheel is not, and a remote open on it is refused with an OSError naming the remedies. That refusal never carries the url, which may hold a credential.

open_fasta_file(resource: GenomicResource, filename: str, index_filename: str | None = None, compressed_index_filename: str | None = None) → FastaFile[source]

Open a bgzipped fasta file in a resource and return a FastaFile.

Not all repositories support this method. Repositories that do not support this method raise an exception.

open_raw_file(resource: GenomicResource, filename: str, mode: str = 'rt', **kwargs: str | bool | None) → IO[source]: Open file in a resource and returns a file-like object.

open_repository_metadata() → Connection[source]: Open the db file for repo metadata and return the connection.

open_tabix_file(resource: GenomicResource, filename: str, index_filename: str | None = None) → TabixFile[source]

Open a tabix file in a resource and return a pysam tabix file.

Not all repositories support this method. Repositories that do no support this method raise and exception.

open_vcf_file(resource: GenomicResource, filename: str, index_filename: str | None = None) → VariantFile[source]

Open a vcf file in a resource and return a pysam VariantFile.

Not all repositories support this method. Repositories that do no support this method raise and exception.

class FsspecReadWriteProtocol(*args: Any, **kwargs: Any)[source]

Bases: FsspecReadOnlyProtocol, ReadWriteRepositoryProtocol

Provides fsspec genomic resources repository protocol.

build_content_file(failed: frozenset[str] = frozenset({})) → list[dict[str, Any]][source]

Build the content of the repository (i.e ‘.CONTENTS.json.gz’).

failed names resources this run could not verify; each is published from the manifest it already had, or left out if it never had one, so a failed run never rebuilds a manifest from scratch and poisons the contents with it (#373).

Only the gzipped index is written. An uncompressed .CONTENTS.json left by an older release is reported rather than deleted (#758).

build_index_info(repository_template: str = 'grr_index.jinja', about_template: str | None = 'grr_about.jinja', failed: frozenset[str] = frozenset({})) → dict[source]

Build info dict for the repository.

failed names resources this run could not verify; each is described from the manifest it already had, or left off the index page if it never had one, so the page never triggers a build-from-scratch of a failed resource’s manifest (#373).

classify_resource_file(remote_resource: GenomicResource, dest_resource: GenomicResource, filename: str) → FileCacheVerdict[source]

Decide whether a resource file needs (re)downloading.

This is the lock-free decision half of update_resource_file(): it performs the same checks and the same state-refresh side effect (rebuild + save the .state on a missing state or one that no longer describes the stored file, and delete a file no longer in the remote manifest), but it never copies/downloads. The verdict’s size is the manifest byte size for files that will download (0 otherwise). See gain#78.

Only an absent file reads as “not cached”. A stat that fails for any other reason – an unreadable cache directory, say – reaches the caller rather than being answered with a download that would fail on the same directory.

collect_all_resources() → Generator[GenomicResource, None, None][source]: Return generator over all resources managed by this protocol.

copy_resource_file(remote_resource: GenomicResource, dest_resource: GenomicResource, filename: str, on_bytes: Callable[[int], None] | None = None) → ResourceFileState | None[source]

Copy a resource file into repository.

A transient stall or drop mid-download (common when fetching a large resource over a slow HTTP GRR link) is retried from scratch with exponential backoff rather than aborting the file. See gain#43.

on_bytes, when given, is called with the number of bytes written for each chunk during the download (see gain#77). Because a retried attempt re-downloads the whole file from scratch, the bytes credited by a failed attempt are rolled back with a single compensating negative call before the retry, so a caller-side byte counter never double-counts.

delete_resource_file(resource: GenomicResource, filename: str) → None[source]: Delete a resource file and it’s internal state.

get_resource_file_change_token(resource: GenomicResource, filename: str) → str | None[source]

Return the store’s change token for a resource file, if any.

A change token is whatever the store itself offers as “this is the version of the object you are looking at”: it changes on every write and holds still for as long as the object is not written. Stores that offer none answer None, and for them the modification time remains the only change hint there is.

The value is opaque. It is never parsed, never compared against an md5 sum and never assumed to be one, even where a particular store happens to derive it from one.

See ADR 0022 for why a state is judged by this rather than by the modification time.

get_resource_file_size(resource: GenomicResource, filename: str) → int[source]: Return the size of a resource file.

get_resource_file_timestamp(resource: GenomicResource, filename: str) → float[source]: Return the timestamp (ISO formatted) of a resource file.

load_resource_file_state(resource: GenomicResource, filename: str) → ResourceFileState | None[source]

Load resource file state from internal GRR state.

If the specified resource file has no internal state returns None.

obtain_resource_file_lock(resource: GenomicResource, filename: str, timeout: float = -1) → AbstractContextManager[source]

Lock a resource’s file.

The lock is a lockfile, which only provides mutual exclusion on a local filesystem. Off file this used to return a no-op context manager – every caller “acquired” it instantly, so the caching protocol serialised nothing and concurrent readers saw partially written files. Refuse rather than hand out a lock that does not lock; a GRR cache must be local. See #473.

publish_raw_file(resource: GenomicResource, filename: str, mode: str = 'wt') → AbstractContextManager[IO][source]

Open a resource file for a write that replaces it, or does not.

The publishing counterpart of open_raw_file(): what the caller writes reaches filename only once the handle has closed cleanly. A write that fails part-way – and an interrupt – leaves whatever was published before exactly as it was, rather than truncating it in place with nothing to roll back to (gain#933).

A separate name rather than a flag on open_raw_file(): the two differ in what they guarantee, not in a parameter, and the callers that need the guarantee are not the ones that need a plain handle.

publish_repository_file(filename: str, mode: str = 'wb') → AbstractContextManager[IO][source]

Publish one of the repository’s own artifacts, by file name.

The counterpart of publish_raw_file() for the artifacts that belong to no resource. The artifacts this protocol builds itself reach _publish_file() directly; this is the way in for one that is built somewhere else and handed over as bytes – the FTS search index, which the CLI assembles from every resource’s implementation and cannot build from in here without importing the layer above it (gain#948).

Same seam, so the same guarantee: the artifact already published is replaced by a completed one in a single move, or not at all.

Binary by default, unlike publish_raw_file(): a repository artifact is a built object – gzipped bytes, a rendered page – and not the text a resource file usually is.

filename names an artifact of the repository itself and is joined to its url unvalidated, which is safe only while callers pass a constant. There is no containment rule to check it against: the resource-file joins each re-run validate_resource_file_name() (gain#467), but that rule is about staying inside a resource, and these artifacts sit beside the resources rather than in one. A caller that ever wants to publish a name it did not author needs such a rule written first.

save_resource_file_state(resource: GenomicResource, state: ResourceFileState) → None[source]: Save resource file state into internal GRR state.

scan_resource_entries(resource: GenomicResource) → ResourceScan[source]: Scan the resource and return what was found.

update_resource_file(remote_resource: GenomicResource, dest_resource: GenomicResource, filename: str) → ResourceFileState | None[source]

Update a resource file into repository if needed.

Returns the state recorded for the file afterwards: the one the download wrote, the one the classification kept, or None for a file deleted because the remote manifest no longer lists it.

GRR_INTERNAL_DIR = '.grr': Directory inside a resource holding the protocol’s own bookkeeping – per-file .state documents, lockfiles, and partial downloads. Not part of the resource: everything that enumerates resource files skips it.

exception RetryableCopyError[source]

Bases: OSError

A copy failure the download loop retries from scratch.

Retryability is a property of the class, not of a list kept beside it: copy_resource_file catches this base, so a new transient failure shape becomes retryable by subclassing it and nothing else (gain#934).

What the retry buys is a fresh remote handle and a fresh temp file, so only faults a second full attempt could plausibly clear belong here – a stalled link, a short read, corrupt bytes. A fault that would repeat identically is a plain OSError and surfaces on the first attempt.

The classification is the download path’s, and it reaches everything that path does – the move included, so a corrupting publish inside a download is retried like any other transient fault.

One shape reaches here from outside a download: redacting a credential-bearing failure whose own type cannot be reconstructed from a message rebuilds it as this class, so that redaction cannot silently reclassify a transient failure as permanent (gain#1078, ADR 0023). Such an error can therefore surface from a plain read – get_file_content, a tabix header – where nothing will retry it. It is still true of it that a second attempt could plausibly have cleared it; only the opportunity is absent.

exception TruncatedDownloadError[source]

Bases: RetryableCopyError

A download that ended short of the manifest’s recorded byte size.

A silent short read in the fsspec range-reassembly layer (gain#292, H1) makes infile.read() return EOF before the whole file has been streamed; the copy loop stops on that empty read and writes a truncated file. Caught explicitly by byte count – before the md5 check – so the failure is reported as the truncation it is, with both the received and the expected size, rather than as an opaque checksum mismatch.

build_fsspec_protocol(proto_id: str, root_url: str, **kwargs: str | bool | None) → FsspecReadOnlyProtocol | FsspecReadWriteProtocol[source]

Create fsspec GRR protocol based on the root url.

read_only is the one boolean among the keyword arguments – hence the widened value type; every other keyword is a url or a credential. It is absent by default rather than False so that asking for a read-write protocol can be told apart from not asking at all: the two mean different things on an http(s) url, where only one of them is serviceable (#528).

build_inmemory_protocol(proto_id: str, root_path: str, content: dict[str, Any]) → FsspecReadWriteProtocol[source]: Build and return an embedded fsspec protocol for testing.

build_local_resource(dirname: str, config: dict[str, Any]) → GenomicResource[source]: Build a resource from a local filesystem directory.

canonical_public_url(public_url: str) → str[source]

Return a public url in the one spelling two builds can be compared in.

Only for comparison – the value a protocol reports through get_public_url stays exactly as its caller wrote it.

gain.genomic_resources.genomic_context module

Genomic context provides a way to collect various genomic resources from various sources and make them available through a single interface.

The module follows a registry-based approach. Providers register themselves and are later consulted (in priority order) to build individual GenomicContext instances. Every created context is combined into a PriorityGenomicContext, offering a single access point for resources such as genomic resource repositories, reference genomes, gene models, annotation pipelines, etc. Providers can be registered programmatically via register_context_provider() or discovered automatically through entry points.

Example usage of genomic context in a tool with command line interface:

import argparse
import sys

from gain.genomic_resources.genomic_context import (
    context_providers_add_argparser_arguments,
    context_providers_init,
    get_genomic_context,
)


parser = argparse.ArgumentParser()
context_providers_add_argparser_arguments(parser)

args = parser.parse_args(sys.argv[1:])
context_providers_init(**vars(args))
genomic_context = get_genomic_context()

If you don’t need command line arguments you can do:

context_providers_init()
genomic_context = get_genomic_context()

When you need a CLI with all defaults and without modifying the argument parser you can do:

context_providers_init_with_argparser("GenomicTool")
genomic_context = get_genomic_context()

class DefaultRepositoryContextProvider[source]

Bases: GenomicContextProvider

Provide access to the default genomic resources repository.

The default repository is resolved via build_genomic_resource_repository() using the environment configuration. The resulting context exposes a single key, "genomic_resources_repository", which can be consumed by other code participating in the context chain.

add_argparser_arguments(parser: ArgumentParser, **kwargs: Any) → None[source]

Declare command line arguments for this provider.

The default repository provider is fully configuration driven and has nothing to expose on the CLI, so the method intentionally leaves the parser untouched. The override exists to make the behaviour explicit in the generated documentation.

init(**kwargs: Any) → GenomicContext[source]

Instantiate a context backed by the default GRR.

Parameters:: **kwargs – Accepted for interface compatibility; the provider ignores runtime keyword arguments because everything is derived from the global configuration.
Returns:: A context exposing a single genomic_resources_repository entry pointing at the default repository instance.
Return type:: GenomicContext

build_cli_genomic_context(cli_args: dict[str, Any]) → GenomicContext[source]

Initialise the context providers from parsed CLI arguments and merge.

The one call a CLI tool makes between parsing its arguments and asking for a GRR: every registered provider sees cli_args (see context_providers_init()), and the result is the merged PriorityGenomicContext of get_genomic_context().

clear_registered_contexts() → None[source]

Forget all contexts created by context_providers_init().

This function exists primarily for testing scenarios where the global registry should be reset between test cases.

context_providers_add_argparser_arguments(parser: ArgumentParser, **kwargs: Any) → None[source]

Delegate command line argument registration to each provider.

Parameters:: parser – The parser that should receive additional arguments from every registered provider.

context_providers_init(**kwargs: Any) → None[source]

Materialize contexts from every registered provider.

The function walks all registered providers in priority order and asks each of them to initialise a GenomicContext. The resulting contexts are stored for later retrieval via get_genomic_context().

Notes

Providers are invoked at most once per process. Subsequent calls are ignored until clear_registered_contexts() is executed, which is especially helpful in unit tests.

Parameters:: **kwargs – Keyword arguments forwarded to every provider’s init method.

context_providers_init_with_argparser(toolname: str = 'GenomicTool') → None[source]

Initialise providers using arguments parsed from sys.argv.

Parameters:: toolname – The program name presented to argparse.ArgumentParser.

Notes

This helper is useful for simple tools that do not customise their argument parser but still want to expose the command line options defined by registered context providers.

get_genomic_context() → GenomicContext[source]

Return a genomic context that merges every registered context.

When two contexts expose the same key, the context that was registered last wins.

Returns:: A PriorityGenomicContext over a snapshot of the registered contexts. A context registered later is not part of it.
Return type:: GenomicContext

get_grr_from_context(context: GenomicContext) → GenomicResourceRepo[source]

Return the repository that a genomic context exposes.

Parameters:: context (GenomicContext) – The context to ask, for example the result of get_genomic_context().
Returns:: The repository of the context. The function never returns None.
Return type:: GenomicResourceRepo
Raises:: ValueError – If the context exposes no repository.

register_context(context: GenomicContext) → None[source]

Record context so it participates in future lookups.

Parameters:: context – The context instance to be considered when get_genomic_context() is invoked.

register_context_provider(context_provider: GenomicContextProvider) → None[source]

Register context_provider so it participates in initialization.

Parameters:: context_provider – The provider implementation that should be considered when contexts are assembled. Providers are stored in registration order and later sorted by their priority before initialization.

gain.genomic_resources.genomic_context_base module

Base classes and interfaces for genomic context management.

This module defines the foundational abstractions for organizing and accessing genomic resources through a unified context system. The central concept is GenomicContext, which acts as a key-value store exposing resources like genomic repositories, reference genomes, gene models, and annotation pipelines. Providers implementing GenomicContextProvider are responsible for building concrete context instances, often by consulting configuration files or command-line arguments.

The module also provides two concrete context implementations: SimpleGenomicContext for straightforward dictionary-backed contexts and PriorityGenomicContext for merging multiple contexts with fallback semantics.

Key Constants

GC_GRR_KEYstr: Standard key for the genomic resources repository object.
GC_REFERENCE_GENOME_KEYstr: Standard key for the reference genome object.
GC_GENE_MODELS_KEYstr: Standard key for the gene models object.
GC_ANNOTATION_PIPELINE_KEYstr: Standard key for the annotation pipeline object.

gain.genomic_resources.genomic_context_cli module

Command-line helpers for configuring genomic resource contexts.

This module exposes CLIGenomicContextProvider, a concrete implementation of GenomicContextProvider that resolves genomic resources based on command-line arguments. Tools can register the provider to let their users supply a genomic resources repository, reference genome, and gene models at runtime.

class CLIGenomicContextProvider[source]

Bases: GenomicContextProvider

Resolve genomic resources from command-line arguments.

The provider allows CLI tools to override the default genomic resources repository, reference genome, and gene models. When invoked without any overrides, it falls back to the previously initialised genomic context so that defaults from gpf_instance or other providers remain available.

add_argparser_arguments(parser: ArgumentParser, **kwargs: Any) → None[source]

Expose CLI options that control genomic resource resolution.

Parameters:

parser – The argument parser that should receive the provider specific options.
**kwargs – skip_cli_reference_genome and skip_cli_gene_models leave out -R and -G for a tool that resolves neither from the command line. The GRR options are always added.

init(**kwargs: Any) → GenomicContext | None[source]

Create a SimpleGenomicContext based on CLI arguments.

Parameters:: **kwargs – Arguments produced from the command-line parser. The provider recognises grr_filename, grr_directory, reference_genome_resource_id, and gene_models_resource_id.
Returns:: A context containing the resolved objects, or None if the genomic resources repository could not be determined.
Return type:: GenomicContext | None

gain.genomic_resources.group_repository module

Provides group genomic resources repository.

class GenomicResourceGroupRepo(children: list[GenomicResourceRepo], repo_id: str | None = None)[source]

Bases: GenomicResourceRepo

Defines group genomic resources repository.

find_resource(resource_id: str, version_constraint: str | None = None, repository_id: str | None = None) → GenomicResource | None[source]

Return the resource with the given id, or None.

Use this method when the id comes from outside and absence is an ordinary result. Use get_resource() when absence is a bug.

Parameters:

resource_id (str) – The id of the resource, without the version.
version_constraint (str or None, default None) – A constraint such as ">=2.0". None selects the latest version.
repository_id (str or None, default None) – Restricts the lookup to the repository with this id. The rule is the same as for get_resource().

Returns:

The matching resource. None means that no resource matches the id, the constraint and the repository id.

Return type:

GenomicResource or None

get_all_resources() → Generator[GenomicResource, None, None][source]

Return a generator over all resources of the repository.

Read the generator lazily. A repository can be large.

Yields:: GenomicResource – A resource of the repository. A repository without resources yields nothing.

get_resource(resource_id: str, version_constraint: str | None = None, repository_id: str | None = None) → GenomicResource[source]

Return the resource with the given id.

Use this method when the id is a constant and its absence is a bug. Use find_resource() when absence is an ordinary result.

Parameters:

resource_id (str) – The id of the resource, without the version.
version_constraint (str or None, default None) – A constraint such as ">=2.0" or "=2.1". None selects the latest version.
repository_id (str or None, default None) – Restricts the lookup to the repository with this id, anywhere in the tree of this repository. An empty string or None means no restriction.

Returns:

The matching resource. The method never returns None.

Return type:

GenomicResource

Raises:

FileNotFoundError – If no resource matches and the repository is backed by a protocol (directory, http, s3, in-memory).
ValueError – If no resource matches and the repository is a group repository.

invalidate() → None[source]

Drop the cached state of the repository.

The next access loads the resource list, the metadata and the file contents again.

search_resources(search_term: str | None = None, resource_type: str | None = None, resource_query: str | None = None) → Generator[GenomicResource, None, list[tuple[str, str]]][source]

Search the resources by a full-text term, a type or a query.

The method returns a generator, so it reads the repository lazily. A resource must match every filter that you supply.

Parameters:

search_term (str or None, default None) – A full-text term. None means no full-text filter.
resource_type (str or None, default None) – A resource type, for example position_score. None means no type filter.
resource_query (str or None, default None) – A wildcard query over the ids and the labels of the resources. None means no query filter.

Yields:

GenomicResource – A matching resource. With no filter, every resource matches.

Returns:

The generator returns, when it ends, a (repository id, reason) pair for each child that a group repository skipped while it still answered. None and an empty list both mean that nothing was skipped. A for loop discards this value.

Return type:

list of tuple of (str, str) or None

Raises:

SearchTermError – If the search term cannot be parsed.
SearchIndexUnavailableError – If the repository has no usable search index.

search_resources_by_child(search_term: str | None = None, resource_type: str | None = None, resource_query: str | None = None) → Generator[tuple[GenomicResourceRepo, GenomicResource], None, list[tuple[str, str]]][source]

Search, pairing each hit with the child repository serving it.

The pair names the repository that actually holds the resource: a nested group projects its own pairs upward rather than naming itself, so a caller never has to take a group apart to label a row.

This is where a child that cannot answer the filter is skipped, and search_resources() is its projection – the two cannot drift, because there is only one loop. The skips of a search that still answered are the generator’s return value (gain#686); a for loop discards them, which is exactly right for the callers that already hear about them from the log.

gain.genomic_resources.histogram module

Handling of genomic scores statistics.

Currently we support only genomic scores histograms.

class CategoricalHistogram(config: CategoricalHistogramConfig, counter: dict[str | int, int] | None = None, *, truncated: bool = False, unique_values: int | None = None, total_count: int | None = None)[source]

Bases: Statistic

A histogram that counts each distinct value of a score.

Parameters:

config (CategoricalHistogramConfig) – The configuration.
counter (dict[str | int, int] | None) – The counts of the values. None means an empty histogram.
truncated (bool) – True when counter holds only the values of a truncated file.
unique_values (int | None) – The distinct values of the full histogram. None means the length of counter.
total_count (int | None) – The sum of the counts of the full histogram. None means the sum of counter.
Attributes –

type (str): The type name that a stored file carries,
categorical_histogram.

UNIQUE_VALUES_LIMIT (int): The most distinct values that a
histogram accepts when its config does not set enforce_type. The value is 100.

config (CategoricalHistogramConfig): The configuration.

enforce_type (bool): The enforce_type flag of the config.

truncated (bool): True when this histogram was loaded from a
truncated file. Its counts are then only the kept values.

y_log_scale (bool): The y_log_scale flag of the config.

UNIQUE_VALUES_LIMIT = 100

add_batch(values: ndarray, weights: ndarray) → None[source]

Add a batch of (value, weight) pairs, vectorized.

The result is the same as calling add_value for each pair in order.

Parameters:

values (np.ndarray) – The values. None cells are skipped.
weights (np.ndarray) – The weight of each value, of the same length as values.

Raises:

TypeError – If a value is neither text nor an integer.
HistogramError – If the config does not set enforce_type and the batch takes the histogram past UNIQUE_VALUES_LIMIT.

add_value(value: str | int | None, count: int = 1) → None[source]

Add a value to the categorical histogram.

Parameters:

value (str | int | None) – The value. None is skipped and nothing is counted.
count (int) – The weight of the value.

Raises:

TypeError – If the value is neither text nor an integer.
HistogramError – If the config does not set enforce_type and the histogram now holds more than UNIQUE_VALUES_LIMIT distinct values.

static deserialize(content: str) → CategoricalHistogram[source]

Rebuild a categorical histogram from serialize output.

Parameters:: content (str) – The JSON text.
Returns:: The histogram.
Return type:: CategoricalHistogram

property display_values: dict[str | int, int]

The counts that the summary page shows, in display order.

The config selects and orders the values. The values that the selection leaves out are summed under the key Other.

Raises:: ValueError – If the config sets value_order and the order leaves out a counted value.

static from_dict(data: dict[str, Any]) → CategoricalHistogram[source]

Build a categorical histogram from a dict.

Parameters:: data (dict[str, Any]) – A mapping like the result of to_dict, or the truncated form. A full histogram has no truncated, unique_values or total_count key. They default to False and to the values computed from the counts.
Returns:: The histogram.
Return type:: CategoricalHistogram
Raises:: KeyError – If the key config is missing.

merge(other: CategoricalHistogram) → None[source]

Add the counts of another histogram into this one.

Parameters:

other (CategoricalHistogram) – A full histogram with the same config.

Raises:

HistogramError – If either histogram is truncated, or if the config does not set enforce_type and the sum has too many distinct values.
AssertionError – If the two configs differ.

moments_summary() → None[source]

Return None, because a categorical value has no mean.

Returns:: Always.
Return type:: None

plot(outfile: IO, score_id: str, *, y_axis_label: str | None = None, small_values_description: str | None = None, large_values_description: str | None = None) → None[source]

Plot the displayed values and write the image into outfile.

Parameters:

outfile (IO) – A binary file object that receives the image.
score_id (str) – The score name. It is the x axis label.
y_axis_label (str | None) – The y axis label. None means count.
small_values_description (str | None) – The text under the first value. The plot shows both descriptions or neither.
large_values_description (str | None) – The text under the last value.

property raw_values: dict[str | int, int]

Every counted value with its count, unordered.

A truncated histogram returns only the values that its file kept.

serialize() → str[source]

Render the full histogram as the JSON stored in the resource.

Returns:: The JSON text.
Return type:: str

serialize_gzipped() → bytes[source]

Render the serialize JSON as gzip bytes.

Returns:: The same histogram gives the same bytes on every call.
Return type:: bytes

serialize_truncated() → str[source]

Render the small truncated form of this histogram as JSON.

Returns:: The JSON text. It holds the config, the values that the config selects for display, unique_values and total_count of the full histogram, and the flag truncated. A value in value_order that has no count gets the count 0.
Return type:: str

to_dict() → dict[str, Any][source]

Render this histogram as the mapping from_dict reads back.

Returns:: A mapping with the keys config and values. It holds every counted value, not only the displayed ones.
Return type:: dict[str, Any]

property total_count: int

The sum of all counts of the full histogram.

A truncated histogram returns the sum of the full data, not the sum of the values it kept.

type = 'categorical_histogram'

property unique_values: int

The number of distinct values of the full histogram.

A truncated histogram returns the count of the full data, not the count of the values it kept.

values_domain() → str[source]

The displayed values, as text for the summary page.

Returns:: The displayed values, separated by commas. A truncated histogram adds the text (top N of M values).
Return type:: str

class CategoricalHistogramConfig(displayed_values_count: int | None = 20, displayed_values_percent: float | None = None, value_order: list[str | int] | None = None, y_log_scale: bool = False, label_rotation: int = 0, plot_function: str | None = None, enforce_type: bool = True, natural_order: bool = False, allow_only_whole_values_y: bool = False)[source]

Bases: object

Configuration of a categorical histogram.

Variables:

displayed_values_count (int | None) – The number of values that the summary page shows. The default is 20. None means no limit, when displayed_values_percent is set.
displayed_values_percent (float | None) – Show the most common values until they cover this percent of the counts. None means the count rule applies.
value_order (list[str | int] | None) – The values to show, in this order. None or an empty list means the order is by count.
y_log_scale (bool) – Whether the plot draws the counts on a log scale.
label_rotation (int) – The rotation of the x labels, in degrees.
plot_function (str | None) – The file:function name of a custom plot function. None means the built-in plot.
enforce_type (bool) – False means the histogram refuses more than CategoricalHistogram.UNIQUE_VALUES_LIMIT distinct values. True means there is no limit. The flag name is the reverse of its effect.
natural_order (bool) – Order the displayed values by value, numbers before text, and not by count.
allow_only_whole_values_y (bool) – Whether the plot uses whole numbers for the y ticks.

allow_only_whole_values_y: bool = False

static default_config() → CategoricalHistogramConfig[source]

Build the config of a score that declares no histogram.

Returns:: A config that enforces CategoricalHistogram.UNIQUE_VALUES_LIMIT.
Return type:: CategoricalHistogramConfig

displayed_values_count: int | None = 20

displayed_values_percent: float | None = None

enforce_type: bool = True

static from_dict(parsed: dict[str, Any]) → CategoricalHistogramConfig[source]

Create categorical histogram config from a configuration dict.

Parameters:

parsed (dict[str, Any]) – The histogram: mapping of a score. Its type must be categorical. When it sets neither displayed_values_count nor displayed_values_percent, the count is 20.

Returns:

The config. It does not enforce the unique values limit.

Return type:

CategoricalHistogramConfig

Raises:

TypeError – If type is not categorical.
ValueError – If both displayed_values_count and displayed_values_percent are set.

label_rotation: int = 0

natural_order: bool = False

plot_function: str | None = None

to_dict() → dict[str, Any][source]

Transform categorical histogram config to dict.

Returns:: The mapping that from_dict reads back. The keys displayed_values_count, displayed_values_percent and plot_function are present only when they differ from the default.
Return type:: dict[str, Any]

value_order: list[str | int] | None = None

y_log_scale: bool = False

GZIPPED_HISTOGRAM_COMPRESSLEVEL = 6: The gzip level and suffix of a full histogram stored gzipped (ADR 0032).

class HistogramStatisticMixin[source]

Bases: object

Mixin for creating statistics classes with histograms.

static get_histogram_file(score_id: str) → str[source]

static get_histogram_image_file(score_id: str) → str[source]

NUMBER_HISTOGRAM_VALUE_TYPES = ('float', 'int', 'bool')

Which score value types a NUMBER histogram can accumulate, one value at a time. bool is in: numpy folds it as 0/1, and a two-bin histogram over a flag is meaningful. str is not, and a resource pairing the two aborted its entire statistics build in np.isnan (gain#1285); since gain#1336 that pairing is refused when the score is CONSTRUCTED, by refuse_unfoldable_histograms(), which is the only thing that reads this set.

It lives here, with the config whose acceptance it describes, rather than in the statistics scan that used to own it – the refusal moved to the score layer, and the score layer must not import the scan to ask what a histogram can fold.

Deliberately WIDER than the scan’s _BULK_HISTOGRAM_VALUE_TYPES, and the two are different questions rather than one rule stated twice: this asks what a histogram can fold value by value, that asks what it can fold a whole column of (a bulk read yields a number histogram’s column as float64, which a bool score’s column is not). Merging them would widen the vectorized path to a type it cannot read.

class NullHistogram(config: NullHistogramConfig | None)[source]

Bases: Statistic

A histogram that has no counts and records why.

Parameters:

config (NullHistogramConfig | None) – The configuration. None gives the reason Unspecified reason.
Attributes –

type (str): The type name that a stored file carries,
null_histogram.

reason (str): Why there is no histogram.

add_value(value: Any, count: int = 1) → None[source]

Discard the value.

Parameters:

value (Any) – Ignored.
count (int) – Ignored.

static deserialize(content: str) → NullHistogram[source]

Rebuild a null histogram from serialize output.

Parameters:: content (str) – The JSON text.
Returns:: The histogram.
Return type:: NullHistogram

static from_dict(data: dict[str, Any]) → NullHistogram[source]

Build a null histogram from a dict.

Parameters:: data (dict[str, Any]) – A mapping like the result of to_dict. A missing reason gives an empty text.
Returns:: The histogram.
Return type:: NullHistogram
Raises:: TypeError – If the type in config is not null.

merge(other: NullHistogram | CategoricalHistogram | NumberHistogram) → None[source]

Do nothing, because there are no counts to add.

Parameters:: other (Histogram) – Ignored.

moments_summary() → None[source]

Return None, because a null histogram has no moments.

Returns:: Always.
Return type:: None

plot(_outfile: IO, _score_id: str) → None[source]

Draw nothing and leave outfile untouched.

Parameters:

_outfile (IO) – Ignored.
_score_id (str) – Ignored.

serialize() → str[source]

Render this histogram as the JSON stored in the resource.

Returns:: The JSON text.
Return type:: str

to_dict() → dict[str, Any][source]

Render this histogram as the mapping from_dict reads back.

Returns:: A mapping with the key config. It holds the reason, which is all the content of a null histogram.
Return type:: dict[str, Any]

type = 'null_histogram'

values_domain() → str[source]

Report that there is no domain.

Returns:: The text NO DOMAIN.
Return type:: str

class NullHistogramConfig(reason: str)[source]

Bases: object

Configuration of a null histogram.

Variables:: reason (str) – Why the score has no histogram. The reason is required, so a summary page can always explain a null histogram.

static default_config() → NullHistogramConfig[source]

Build a null config for a caller with no reason to give.

Returns:: A config with the reason Unspecified reason.
Return type:: NullHistogramConfig

static from_dict(parsed: dict[str, Any]) → NullHistogramConfig[source]

Create a null histogram config from a configuration dict.

Parameters:: parsed (dict[str, Any]) – The histogram: mapping of a score. Its type must be null.
Returns:: The config. A missing reason gives Unspecified reason.
Return type:: NullHistogramConfig
Raises:: TypeError – If type is not null.

reason: str

to_dict() → dict[str, Any][source]

Render this config as the mapping from_dict reads back.

Returns:: The mapping with the keys type and reason.
Return type:: dict[str, Any]

class NumberHistogram(config: NumberHistogramConfig, bins: ndarray | None = None, bars: ndarray | None = None)[source]

Bases: Statistic

A histogram of numeric values in bins.

Parameters:

config (NumberHistogramConfig) – The configuration. Its view range must have both edges set.
bins (np.ndarray | None) – The bin edges, one more than the bars. Pass both bins and bars or neither.
bars (np.ndarray | None) – The count of each bin.

Raises:

ValueError – If the view range has an unset or NaN edge, if a log scale has no x_min_log, or if only one of bins and bars is given.
Attributes: – type (str): The type name that a stored file carries, number_histogram. config (NumberHistogramConfig): The configuration. bins (np.ndarray): The bin edges. There is one more edge than there are bars. bars (np.ndarray): The count of each bin. Bar i counts the values from bins[i] up to bins[i + 1]. out_of_range_bins (list[int]): The counts of the values below and of the values above the view range. min_value (float): The smallest value that was added. NaN when there is none. max_value (float): The largest value that was added. NaN when there is none. view_range (tuple[float, float]): The low and the high edge of the binned range.

add_batch(values: ndarray, weights: ndarray) → None[source]

Add a batch of (value, weight) pairs, vectorized.

The result is the same as calling add_value for each pair in order, the sums behind mean and std included.

Parameters:

values (np.ndarray) – The values. NaN values are skipped. The array is converted to float64.
weights (np.ndarray) – The weight of each value, of the same length as values.

add_value(value: float | generic | None, count: int = 1) → None[source]

Add a value to the histogram.

Parameters:

value (float | np.generic | None) – The value. None and NaN are skipped and nothing is counted.
count (int) – The weight of the value.

Raises:

TypeError – If the value is not a real number, for example text.

property count: int | None: The weighted count of the added values, or None when unknown.

static deserialize(content: str) → NumberHistogram[source]

Rebuild a number histogram from serialize output.

Parameters:: content (str) – The JSON text.
Returns:: The histogram.
Return type:: NumberHistogram

static from_dict(data: dict[str, Any]) → NumberHistogram[source]

Build a number histogram from a dict.

Parameters:: data (dict[str, Any]) – A mapping like the result of to_dict. A missing key min_value or max_value gives NaN. A missing out_of_range_bins gives two zeros. A mapping without the stored count, sum and sum of squares gives a histogram with unknown count, mean and std.
Returns:: The histogram.
Return type:: NumberHistogram
Raises:: KeyError – If the key config is missing.

max_value: float

property mean: float | None: The weighted mean, or None when unknown or nothing was added.

merge(other: NumberHistogram) → None[source]

Add the counts of another histogram into this one.

Parameters:: other (NumberHistogram) – A histogram with the same bin edges. This method changes only self.
Raises:: AssertionError – If the bin edges of the two histograms differ.

min_value: float

property moments: Moments | None: The count, sum and sum of squares, or None when unknown.

moments_summary() → tuple[tuple[str, str], ...] | None[source]

The count, mean and standard deviation, as text for the page.

Returns:: Three (label, text) pairs: n, mean and sd. None when the numbers are unknown or nothing was added.
Return type:: MomentsSummary | None

out_of_range_bins: list[int]

out_of_range_values: list[float]

plot(outfile: IO, score_id: str, y_axis_label: str | None = None, small_values_description: str | None = None, large_values_description: str | None = None) → None[source]

Plot the histogram and write the image into outfile.

Parameters:

outfile (IO) – A binary file object that receives the image.
score_id (str) – The score name. It is the x axis label.
y_axis_label (str | None) – The y axis label. None means count.
small_values_description (str | None) – The text under the low end of the x axis. The plot shows both descriptions or neither.
large_values_description (str | None) – The text under the high end of the x axis.

serialize() → str[source]

Render this histogram as the JSON stored in the resource.

Returns:: The JSON text.
Return type:: str

property std: float | None: The population standard deviation, or None when mean is.

to_dict() → dict[str, Any][source]

Render this histogram as the mapping from_dict reads back.

Returns:: A mapping that JSON can store as it is. The bin edges and the counts are lists.
Return type:: dict[str, Any]

type = 'number_histogram'

values_domain() → str[source]

The observed value range, as text for the summary page.

Returns:: The smallest and the largest added value, as [min, max]. It is not the view range.
Return type:: str

view_max() → float[source]

The high edge of the last bin.

Returns:: The high edge of the view range. Values above it are counted in out_of_range_bins and not binned.
Return type:: float

view_min() → float[source]

The low edge of the first bin.

Returns:: The low edge of the view range. Values below it are counted in out_of_range_bins and not binned. It is not the smallest value that was added.
Return type:: float

view_range: tuple[float, float]

class NumberHistogramConfig(view_range: tuple[float | None, float | None], number_of_bins: int = 100, x_log_scale: bool = False, y_log_scale: bool = False, x_min_log: float | None = None, plot_function: str | None = None)[source]

Bases: object

Configuration of a number histogram.

Variables:

view_range (tuple[float | None, float | None]) – The low and the high edge of the binned range. None at either end means the edge is not pinned yet. The histogram needs both edges.
number_of_bins (int) – The number of bars. The default is 100.
x_log_scale (bool) – Whether the bins are spaced on a log scale. A log scale requires x_min_log.
y_log_scale (bool) – Whether the plot draws the bar heights on a log scale. The bin counts are the same.
x_min_log (float | None) – The high edge of the first bin on a log scale. None when the scale is linear.
plot_function (str | None) – The file:function name of a custom plot function. None means the built-in plot.

static default_config(min_max: MinMaxValue | None) → NumberHistogramConfig[source]

Build the config of a score that declares no histogram.

Parameters:: min_max (MinMaxValue | None) – The observed minimum and maximum of the score. None means they are not known yet.
Returns:: A config with 100 linear bins. The view range is the observed range. It is (None, None) when min_max is None. When the minimum equals the maximum, the range is the minimum to the minimum plus one.
Return type:: NumberHistogramConfig

static from_dict(parsed: dict[str, Any]) → NumberHistogramConfig[source]

Build a number histogram config from a parsed yaml file.

Parameters:: parsed (dict[str, Any]) – The histogram: mapping of a score. Its type must be number. A missing key takes the default of the field.
Returns:: The config. A missing view_range gives (None, None).
Return type:: NumberHistogramConfig
Raises:: TypeError – If type is not number.

has_view_range() → bool[source]

Whether both ends of the view range are pinned.

Returns:: True when neither edge of view_range is None. False means the config still needs the minimum and the maximum of the score before a histogram can be built from it.
Return type:: bool

number_of_bins: int = 100

plot_function: str | None = None

to_dict() → dict[str, Any][source]

Transform number histogram config to dict.

Returns:: The mapping that from_dict reads back. The key plot_function is present only when the config sets one.
Return type:: dict[str, Any]

view_range: tuple[float | None, float | None]

x_log_scale: bool = False

x_min_log: float | None = None

y_log_scale: bool = False

build_default_histogram_conf(value_type: str) → NumberHistogramConfig | CategoricalHistogramConfig | NullHistogramConfig[source]: Build default histogram config for given value type.

build_empty_histogram(config: NullHistogramConfig | CategoricalHistogramConfig | NumberHistogramConfig) → NumberHistogram | CategoricalHistogram | NullHistogram[source]: Create an empty histogram from a deserialize histogram dictionary.

build_histogram_config(config: dict[str, Any] | None) → NullHistogramConfig | CategoricalHistogramConfig | NumberHistogramConfig | None[source]: Create histogram config form configuration dict.

drop_stale_histogram_file(resource: GenomicResource, filename: str) → None[source]

Delete a histogram file an earlier build left in resource.

A DVC-tracked file – one with a .dvc pointer beside it – is left in place with a warning, whether or not its blob is pulled: deleting the blob alone leaves a pointer the manifest build re-adds an entry from, for a file that is gone, and a pointer-only checkout still carries that stale entry. A file that is not there at all is not an error, since a score that never had statistics has nothing to drop.

load_histogram(resource: GenomicResource, filename: str) → NullHistogram | CategoricalHistogram | NumberHistogram[source]

Load a stored histogram from a resource.

Parameters:

resource (GenomicResource) – The resource that holds the file.
filename (str) – The file name in the resource. The suffix selects the format: .yaml, .json or .json.gz (gzipped JSON).

Returns:

The histogram. A missing file, an unknown suffix, an unknown histogram type and a body that fails to deserialize each give a NullHistogram with the reason.

Return type:

Histogram

Raises:

KeyError – If the parsed file has no config key or no type in it.
yaml.YAMLError – If a .yaml file is not valid YAML.
json.JSONDecodeError – If a .json or .json.gz file is not valid JSON.

plot_histogram(res: GenomicResource, image_filename: str, hist: NullHistogram | CategoricalHistogram | NumberHistogram, score_id: str, small_values_desc: str | None = None, large_values_desc: str | None = None) → None[source]: Plot histogram and save it into the resource.

save_histogram(resource: GenomicResource, filename: str, histogram: NullHistogram | CategoricalHistogram | NumberHistogram) → None[source]: Save histogram into a resource.

truncated_histogram_filename(histogram_filename: str) → str[source]

Return the truncated-sidecar filename for a histogram filename.

statistics/histogram_cell.json maps to statistics/truncated/histogram_cell.json: a directory rather than a name suffix, because the score-id part of the histogram filename is arbitrary (a score may itself be named foo_truncated) while no score id can contain /.

gain.genomic_resources.liftover_chain module

Provides LiftOver chain resource.

class LiftoverChain(resource: GenomicResource)[source]

Bases: ResourceConfigValidationMixin

Convert coordinates between assemblies with a UCSC chain file.

The class wraps a pyliftover object. Build it with build_liftover_chain_from_resource_id and call open before the first conversion.

Parameters:

resource (GenomicResource) – A resource of type liftover_chain.

Variables:

resource (GenomicResource) – The resource that holds the chain file.
config (dict) – The validated and normalized resource configuration.
source_genome_id (str or None) – The id from the source_genome label, or None when the label is absent or unusable.
target_genome_id (str or None) – The id from the target_genome label, or None when the label is absent or unusable.

Raises:

ValueError – If the type of the resource is not liftover_chain.

close() → None[source]: Do nothing: a chain holds no resource to release.

convert_coordinate(chrom: str, pos: int) → tuple[str, int, str, int] | None[source]

Lift over a genomic coordinate.

The variant_coordinates rename applies before the lift and the target_coordinates rename applies after it.

Parameters:

chrom (str) – The contig name in the naming of the caller.
pos (int) – The 1-based position.

Returns:

The target contig, the target 1-based position, the strand of the target and the chain score. None when the position is in no aligned block of the chain. When the chain maps the position to more than one target, the first one is returned.

Return type:

tuple of (str, int, str, int) or None

property files: set[str]: Return the names of the files of the resource that the chain uses.

static get_schema() → dict[str, Any][source]

Return the cerberus schema of the resource configuration.

Returns:: The schema. It has the filename key and the optional chrom_prefix block with variant_coordinates and target_coordinates.
Return type:: dict

is_open() → bool[source]: Return True when the chain file is read.

static map_chromosome(chrom: str, mapping: dict[str, str] | None) → str[source]

Rename a chromosome (contig) name.

Parameters:

chrom (str) – The contig name.
mapping (dict, optional) – A chrom_prefix entry. Its del_prefix is removed first, then its add_prefix is prepended.

Returns:

The renamed contig. The name is unchanged when mapping is None or empty.

Return type:

str

open() → LiftoverChain[source]

Read the chain file.

Returns:: This chain, so the call chains. A call on an open chain does nothing.
Return type:: LiftoverChain

build_liftover_chain_from_resource(resource: GenomicResource) → LiftoverChain[source]

Build the liftover chain of a resource.

The chain is memoized for each resource. A second call for the same resource returns the same chain, which may already be open.

Parameters:: resource (GenomicResource) – A resource of type liftover_chain.
Returns:: The chain. It is not open until the caller calls open.
Return type:: LiftoverChain
Raises:: ValueError – If resource is None or its type is not liftover_chain.

build_liftover_chain_from_resource_id(resource_id: str, grr: GenomicResourceRepo | None = None) → LiftoverChain[source]

Build the liftover chain of a resource, found by its id.

Parameters:

resource_id (str) – The id of a resource of type liftover_chain.
grr (GenomicResourceRepo, optional) – The repository to search. The default repository is built when this is None.

Returns:

The memoized chain. It may already be open.

Return type:

LiftoverChain

Raises:

FileNotFoundError – If the repository has no resource with this id.
ValueError – If the type of the resource is not liftover_chain.

gain.genomic_resources.memo module

Process-wide memos of objects built from a resource or a local file.

Gene models, gene scores, gene set collections and liftover chains are expensive to build and are shared by every caller in a process. Each factory keeps one memo; a resource-built object is keyed on GenomicResource.get_memo_key(), a file-built one on the file name and config_memo_key() of the synthetic config it builds. The rule, and the two resource-keyed caches that deliberately do not follow it, are recorded in ADR 0036 (docs/adr/).

class Memo[source]

Bases: Generic

A thread-safe memo from a key to a built object.

get_or_build returns the object stored under a key, building and storing it first on a miss. The build runs under the memo’s lock, so two callers racing on one key build it once and both receive the very same object; a builder must therefore not call back into the same memo. The memo never evicts on its own: objects are held by strong reference until clear drops them all, or discard drops the ones whose key a predicate accepts.

clear() → None[source]: Drop every memoised object.

discard(doomed: Callable[[K], bool]) → int[source]

Drop every memoised object whose key doomed accepts.

Returns:: The number of objects dropped.
Return type:: int

get_or_build(key: K, build: Callable[[], V]) → V[source]

Return the object memoised under key, building it if absent.

A builder that raises leaves nothing stored under key.

config_memo_key(config: Any) → str[source]

Return the memo-key spelling of a resource config.

Two configs differing only in the order their keys were written get the same key. Any value a parsed yaml can hold is accepted, including a bare date and a number used as a key beside string ones.

gain.genomic_resources.reference_genome module

class ReferenceGenome(resource: GenomicResource)[source]

Bases: ResourceConfigValidationMixin

Query the chromosomes and the nucleotides of a reference genome.

The genome wraps a resource of type genome. The builder returns it closed. Call open() before fetch() or get_sequence(). The chromosome accessors load the index on demand, so they answer without open().

Variables:

resource (GenomicResource) – The genome resource that the object wraps.
config (dict) – The validated configuration of the resource.
pars (dict) – The pseudoautosomal regions from the PARS key of the configuration, as a mapping from the chromosome name to a list of Region. The mapping is empty when the configuration has no PARS key.

property chrom_prefix: str

"chr" or "".

The value is "chr" when the first chromosome name starts with it, and "" otherwise. A chrom_prefix key in the configuration does not change it.

Type:: The prefix of the chromosome names

property chromosomes: list[str]: The chromosome names, in the order of the .fai index.

close() → None[source]

Close the sequence file and drop the loaded index.

Closing a genome that is not open does nothing.

fetch(chrom: str, start: int, stop: int | None, buffer_size: int = 512) → Generator[str, None, None][source]

Yield the nucleotides of a region, one character at a time.

The coordinates are 1-based and both ends are included. The genome must be open.

Parameters:

chrom (str) – The name of the chromosome.
start (int) – The first position of the region.
stop (int | None) – The last position of the region. A value past the end of the chromosome is cut to the end. When it is None, the region runs to the end of the chromosome.
buffer_size (int) – The number of characters that the reader takes from the file in one step.

Returns:

The nucleotides in upper case. The generator is empty when the genome has no chromosome with this name. In that case a warning is logged.

Return type:

Generator[str, None, None]

get_all_chrom_lengths() → dict[str, int][source]

Return the lengths of all chromosomes.

Returns:: A mapping from the chromosome name to its length, in the order of the .fai index. The mapping is the internal one, not a copy. Do not change it.
Return type:: dict[str, int]

get_chrom_length(chrom: str) → int[source]

Return the length of a chromosome.

Parameters:: chrom (str) – The name of the chromosome.
Returns:: The length of the chromosome in nucleotides.
Return type:: int
Raises:: ValueError – If the genome has no chromosome with this name.

static get_schema() → dict[str, Any][source]

Return the schema that the config of a genome is checked against.

Returns:: The base resource schema, with the keys filename, index_file, PARS and chrom_prefix.
Return type:: dict[str, Any]

get_sequence(chrom: str, start: int, stop: int) → str[source]

Return the nucleotides of a region as one string.

The coordinates are 1-based and both ends are included. The genome must be open.

Parameters:

chrom (str) – The name of the chromosome.
start (int) – The first position of the region.
stop (int) – The last position of the region. A value past the end of the chromosome is cut to the end.

Returns:

The nucleotides in upper case. The string is empty when the genome has no chromosome with this name.

Return type:

str

is_open() → bool[source]

Whether open() has run and close() has not since.

Returns:: True when the sequence file is open.
Return type:: bool

is_pseudoautosomal(chrom: str, pos: int) → bool[source]

Tell whether a position is in a pseudoautosomal region.

Parameters:

chrom (str) – The name of the chromosome.
pos (int) – The 1-based position.

Returns:

True when the position is in a region of pars. The value is False for a chromosome without such regions, and for every position when the configuration has no PARS key.

Return type:

bool

open() → ReferenceGenome[source]

Open the sequence file of the genome.

Returns:: The genome itself, so that the call chains onto a builder. A genome that is already open is returned unchanged.
Return type:: ReferenceGenome
Raises:: FileNotFoundError – If the index file or the sequence file is missing.

property resource_id: str: The id of the genome resource this object wraps.

split_into_regions(region_size: int, chromosome: str | None = None) → Generator[Region, None, None][source]

Split the genome into regions of a given size.

Parameters:

region_size (int) – The size of each region in nucleotides. The value 0 yields one region for each chromosome.
chromosome (str | None) – The chromosome to split. When it is None, the method splits all chromosomes.

Returns:

The regions, chromosome by chromosome. The last region of each chromosome has stop set to None, and it can be shorter than region_size.

Return type:

Generator[Region, None, None]

Raises:

ValueError – If region_size is negative. The call raises. If chromosome is not a chromosome of the genome, the generator raises on the first step.

build_reference_genome_from_file(filename: str) → ReferenceGenome[source]

Build a reference genome from a FASTA file.

The function wraps the file in a local resource. The genome is closed.

Parameters:: filename (str) – The path of the FASTA file. The index file <filename>.fai must be next to it.
Returns:: The closed genome. Call open() before any sequence query.
Return type:: ReferenceGenome

build_reference_genome_from_resource(resource: GenomicResource) → ReferenceGenome[source]

Build the reference genome of a genomic resource.

Parameters:: resource (GenomicResource) – A resource of type genome.
Returns:: The closed genome. Call open() before any sequence query.
Return type:: ReferenceGenome
Raises:: ValueError – If the resource is not of type genome.

build_reference_genome_from_resource_id(resource_id: str, grr: GenomicResourceRepo | None = None) → ReferenceGenome[source]

Build the reference genome of the resource with a given id.

Parameters:

resource_id (str) – The id of a resource of type genome in the repository.
grr (GenomicResourceRepo | None) – The repository that resolves resource_id. When it is None, the function builds the default repository with build_genomic_resource_repository().

Returns:

The closed genome. Call open() before any sequence query.

Return type:

ReferenceGenome

Raises:

FileNotFoundError – If the repository has no resource with this id.
ValueError – If the resource is not of type genome.

genome_index_file(config: dict[str, Any]) → str[source]

Return the FASTA .fai index name for a genome config.

The optional index_file key overrides the default <filename>.fai. The three consumers that read a genome resource from its config – reference_genome_files(), the index parsing and the sequence backend – resolve the name through here, so that a resource is listed, hashed, parsed and opened against one and the same index.

Resolution is by name only: the file is not checked for existence and is not looked up in the resource manifest.

reference_genome_files(config: dict[str, Any]) → set[str][source]

Return all files a reference-genome resource consists of.

The set always contains the genome FASTA and its .fai index (honoring the optional index_file config key). For a bgzipped genome (filename ending in .gz/.bgz, in any case) htslib random access also needs the .gzi BGZF index, so it is included as well.

gain.genomic_resources.repository module

Provides basic classes for genomic resources and repositories.

This module defines the core architecture for managing genomic resources through a flexible repository system. It supports different storage backends (local files, HTTP, S3) and provides both read-only and read-write access.

Class Hierarchy:

      +---------------------+                    +-----------------+
+-----| GenomicResourceRepo |--------------------| GenomicResource |
|     +---------------------+                    +-----------------+
|        ^               ^                                    |
|        |               |                                    |
|        |  +-----------------------------+     +----------------------------+
|        |  | GenomicResourceProtocolRepo | ----| ReadOnlyRepositoryProtocol |
|        |  +-----------------------------+     +----------------------------+
|        |                                                    ^
|        |                                                    |
|    +--------------------------+            +-----------------------------+
+----| GenomicResourceGroupRepo |            | ReadWriteRepositoryProtocol |
     +--------------------------+            +-----------------------------+

Key Concepts:

GenomicResource: Represents a single genomic resource (e.g., a reference genome, score set, or gene model) with metadata and file access methods.
GenomicResourceRepo: Abstract base for repositories that manage collections of genomic resources.
RepositoryProtocol: Defines the storage backend interface (file system, HTTP, S3, etc.) for accessing resource files.
Manifest: Tracks files and their checksums within a resource to ensure data integrity and enable caching.

Resource Identifiers:

Resources are identified by an ID and optional version suffix:

Simple: “hg19/gene_models/refseq”
Versioned: “hg19/gene_models/refseq(1.2.3)”

Configuration Files:

Each resource contains a genomic_resource.yaml configuration file with metadata including type, description, and resource-specific settings.

GR_GENERATED_INFO_PAGES = frozenset({'index.html', 'statistics/index.html'}): The pages grr_manage resource-info writes into a resource. They are regenerated on every run, so they are build artefacts rather than resource data and are never manifested – whether or not DVC manages them (#373).

GR_INDEX_NON_LABEL_COLUMNS = frozenset({'description', 'full_id', 'id', 'score_descriptions', 'score_ids', 'summary', 'type'})

Index columns that describe the resource rather than one of its meta.labels entries. A label query names a label, so a clause on one of these must not be answered out of the column that shares its name – and no resource can carry them as labels anyway, because the index build refuses a label key repeating any of these names, whatever fields the resource’s own implementation contributes (gain#542).

An implementation that contributes a further field of its own belongs here too; the index cannot tell on its own which of its columns came from a label. Registering it here is what both refuses it as a label key and keeps a clause naming it off that column.

GR_INDEX_RESOURCE_FIELDS = ('full_id', 'id', 'type', 'description', 'summary'): The columns every resource contributes to the FTS index, in the order ResourceImplementation.collect_index_info emits them.

GR_INDEX_SCORE_FIELDS = ('score_ids', 'score_descriptions'): The columns a score implementation contributes on top of those.

GR_STATISTICS_INDEX_FILE_NAME = 'statistics/index.html': The path grr_manage resource-info writes the statistics page to; named here so the writer and the exclusion below cannot drift (#373).

class GenomicResource(resource_id: str, version: tuple[int, ...], protocol: ReadOnlyRepositoryProtocol | ReadWriteRepositoryProtocol, config: dict[str, Any] | None = None, manifest: Manifest | None = None)[source]

Bases: object

Represents a single genomic resource with metadata and file access.

A genomic resource is a versioned collection of data files with a configuration file (genomic_resource.yaml) that defines its type, description, and resource-specific settings.

Common resource types include:

genome: Reference genome sequences
gene_models: Gene annotations and transcript models
position_score: Position-based genomic scores
allele_score: Variant effect scores
gene_score: Gene-level scores

Variables:

resource_id – Unique identifier like “hg19/gene_models/refseq”
version (tuple[int, ...]) – Version tuple like (1, 2, 3)
config – Configuration dictionary from genomic_resource.yaml
proto – Repository protocol for accessing resource files

file_exists(filename: str) → bool[source]

Check whether a file exists in the resource.

Parameters:: filename (str) – The name of the file, relative to the resource directory.
Returns:: True if the file exists, else False.
Return type:: bool

get_config() → dict[str, Any][source]

Return the configuration of the resource.

Returns:: The content of genomic_resource.yaml. The method never returns None.
Return type:: dict
Raises:: ValueError – If the resource has no config.

get_description() → str[source]

Return the description of the resource.

Returns:: The description field of the meta block. The string is empty when the resource has no description.
Return type:: str

get_file_content(filename: str, *, uncompress: bool = True, mode: str = 't') → Any[source]

Return the content of a file of the resource.

Parameters:

filename (str) – The name of the file, relative to the resource directory.
uncompress (bool, default True) – If True, decompress a compressed file before the method returns the content.
mode (str, default "t") – "t" returns the content as str. "b" returns bytes.

Returns:

The content of the file. The value is empty when the file is empty.

Return type:

str or bytes

Raises:

FileNotFoundError – If the file does not exist.

get_file_url(filename: str) → str[source]

Return the address of a file of the resource.

The method does not check that the file exists.

Parameters:: filename (str) – The name of the file, relative to the resource directory.
Returns:: A filesystem path for a directory repository. Otherwise, an http(s):// or s3:// url.
Return type:: str
Raises:: ValueError – If the name is not a valid file name for a resource.

get_full_id() → str[source]

Return the id of the resource together with its version.

Returns:: aa/bb/cc(3.2) for the id aa/bb/cc and the version 3.2. The string is aa/bb/cc when the version is 0.
Return type:: str

get_id() → str[source]

Return the id of the resource, without the version.

Returns:: The id, for example hg38/gene_models/MANE/1.3.
Return type:: str

get_labels() → dict[str, Any][source]

Return the labels of the resource.

Returns:: The labels mapping of the meta block, with the values as the curator wrote them. The dictionary is empty when the resource has no labels or when labels is not a mapping. The second case is also reported in the log.
Return type:: dict

get_loaded_manifest() → Manifest | None[source]

Return the stored manifest of the resource, and never build one.

Returns:: The stored manifest. None means that the resource has no stored manifest.
Return type:: Manifest or None

get_manifest() → Manifest[source]

Return the manifest of the resource, and build it if it is missing.

The build reads every file of the resource. It needs a read-write protocol. Use get_loaded_manifest() to avoid the build.

Returns:: The stored manifest, or a new manifest when the resource has none. The method never returns None.
Return type:: Manifest
Raises:: Exception – If the resource has no stored manifest and the protocol is read-only, for example a read-only mount.

get_meta() → dict[str, Any][source]

Return the meta block of the resource.

Returns:: The meta block of the config. The dictionary is empty when the block is absent, is null or is not a mapping. A block that is not a mapping is also reported in the log. The method does not validate the fields inside the block.
Return type:: dict

get_public_url() → str[source]

Return the address of the resource on the public mirror.

Returns:: The public url of the repository joined with the full id of the resource.
Return type:: str

get_repo_public_url() → str[source]

Return the public url of the repository that serves the resource.

Returns:: The public address of the repository, as set in its definition.
Return type:: str

get_repo_url() → str[source]

Return the url of the repository that serves the resource.

Returns:: A filesystem path for a directory repository, or an http(s):// or s3:// url.
Return type:: str

get_summary() → str | None[source]

Return the summary of the resource.

Returns:: The summary field of the meta block. When the resource has no summary, the method returns the description. None means that the resource has neither.
Return type:: str or None

get_type() → str[source]

Return the type of the resource.

Returns:: The type field of genomic_resource.yaml, for example position_score. A resource without the field has the type basic.
Return type:: str
Raises:: ValueError – If the resource has no config.

get_url() → str[source]

Return the address of the resource on its repository.

Returns:: The url of the repository joined with the full id of the resource.
Return type:: str

get_version_str() → str[source]

Return the version of the resource as a string.

Returns:: The version in the form 3.1.
Return type:: str

open_bigwig_file(filename: str) → Any[source]

Open a bigWig file of the resource.

Parameters:: filename (str) – The name of the bigWig file.
Returns:: The open file. Close it after use.
Return type:: pyBigWig.pyBigWig
Raises:: FileNotFoundError – If the file does not exist.

open_fasta_file(filename: str, index_filename: str | None = None, compressed_index_filename: str | None = None) → FastaFile[source]

Open a bgzipped FASTA file of the resource.

Parameters:

filename (str) – The name of the bgzipped FASTA file.
index_filename (str or None, default None) – The name of the .fai index. None means the name of the data file with the suffix .fai.
compressed_index_filename (str or None, default None) – The name of the .gzi index of the compressed file. None means the name of the data file with the suffix .gzi.

Returns:

The open file. Close it after use.

Return type:

pysam.FastaFile

Raises:

ValueError – If the .gzi index of the compressed file does not exist.
FileNotFoundError – If the data file or the .fai index does not exist.

open_raw_file(filename: str, mode: str = 'rt', **kwargs: str | bool | None) → IO[source]

Open a file of the resource.

The same call works for a local directory, an HTTP mirror and an S3 bucket. Close the returned object, or use it in a with block.

Parameters:

filename (str) – The name of the file, relative to the resource directory.
mode (str, default "rt") – The open mode. Text and binary read modes work for every repository. Write modes work for a read-write repository.
**kwargs – Options that the protocol passes to the open call.

Returns:

A file-like object.

Return type:

IO

Raises:

FileNotFoundError – If the file does not exist and the mode reads.

open_tabix_file(filename: str, index_filename: str | None = None) → TabixFile[source]

Open a tabix file of the resource.

Parameters:

filename (str) – The name of the bgzipped data file.
index_filename (str or None, default None) – The name of the index file. None means that the method looks for the .tbi index and then for the .csi index next to the data file.

Returns:

The open file. Close it after use.

Return type:

pysam.TabixFile

Raises:

FileNotFoundError – If the data file or the index file does not exist.

open_vcf_file(filename: str, index_filename: str | None = None) → VariantFile[source]

Open a VCF file of the resource.

Parameters:

filename (str) – The name of the VCF file.
index_filename (str or None, default None) – The name of the index file. None means that the method looks for the index next to the data file.

Returns:

The open file. Close it after use.

Return type:

pysam.VariantFile

Raises:

FileNotFoundError – If the data file or the index file does not exist.

version: tuple[int, ...]

class GenomicResourceProtocolRepo(proto: ReadOnlyRepositoryProtocol | ReadWriteRepositoryProtocol)[source]

Bases: GenomicResourceRepo

Base class for real genomic resources repositories.

close() → None[source]

Release the resources that the repository holds.

After the call, definition is None.

find_resource(resource_id: str, version_constraint: str | None = None, repository_id: str | None = None) → GenomicResource | None[source]

Return the resource with the given id, or None.

Use this method when the id comes from outside and absence is an ordinary result. Use get_resource() when absence is a bug.

Parameters:

resource_id (str) – The id of the resource, without the version.
version_constraint (str or None, default None) – A constraint such as ">=2.0". None selects the latest version.
repository_id (str or None, default None) – Restricts the lookup to the repository with this id. The rule is the same as for get_resource().

Returns:

The matching resource. None means that no resource matches the id, the constraint and the repository id.

Return type:

GenomicResource or None

get_all_resources() → Generator[GenomicResource, None, None][source]

Return a generator over all resources of the repository.

Read the generator lazily. A repository can be large.

Yields:: GenomicResource – A resource of the repository. A repository without resources yields nothing.

get_resource(resource_id: str, version_constraint: str | None = None, repository_id: str | None = None) → GenomicResource[source]

Return the resource with the given id.

Use this method when the id is a constant and its absence is a bug. Use find_resource() when absence is an ordinary result.

Parameters:

resource_id (str) – The id of the resource, without the version.
version_constraint (str or None, default None) – A constraint such as ">=2.0" or "=2.1". None selects the latest version.
repository_id (str or None, default None) – Restricts the lookup to the repository with this id, anywhere in the tree of this repository. An empty string or None means no restriction.

Returns:

The matching resource. The method never returns None.

Return type:

GenomicResource

Raises:

FileNotFoundError – If no resource matches and the repository is backed by a protocol (directory, http, s3, in-memory).
ValueError – If no resource matches and the repository is a group repository.

invalidate() → None[source]

Drop the cached state of the repository.

The next access loads the resource list, the metadata and the file contents again.

search_resources(search_term: str | None = None, resource_type: str | None = None, resource_query: str | None = None) → Generator[GenomicResource, None, None][source]

Search the resources by a full-text term, a type or a query.

The method returns a generator, so it reads the repository lazily. A resource must match every filter that you supply.

Parameters:

search_term (str or None, default None) – A full-text term. None means no full-text filter.
resource_type (str or None, default None) – A resource type, for example position_score. None means no type filter.
resource_query (str or None, default None) – A wildcard query over the ids and the labels of the resources. None means no query filter.

Yields:

GenomicResource – A matching resource. With no filter, every resource matches.

Returns:

The generator returns, when it ends, a (repository id, reason) pair for each child that a group repository skipped while it still answered. None and an empty list both mean that nothing was skipped. A for loop discards this value.

Return type:

list of tuple of (str, str) or None

Raises:

SearchTermError – If the search term cannot be parsed.
SearchIndexUnavailableError – If the repository has no usable search index.

class GenomicResourceRepo(repo_id: str)[source]

Bases: ABC

Abstract base class for genomic resource repositories.

A repository manages a collection of genomic resources, providing methods to discover, retrieve, and (for writable repos) create resources.

Repositories can be:

Protocol-based: Direct access to a single storage backend
Group: Aggregates multiple child repositories
Cached: Wraps another repository with local caching

All repositories support resource lookup with optional version constraints:

repo.get_resource(“hg19/genome”) # Latest version repo.get_resource(“hg19/genome”, “>=2.0”) # Version 2.0 or higher repo.get_resource(“hg19/genome”, “=2.1”) # Exact version 2.1

Build a repository with grr.build_genomic_resource_repository. The properties repo_id and definition describe the repository.

close() → None[source]

Release the resources that the repository holds.

After the call, definition is None.

property definition: dict[str, Any] | None

The definition that built the repository, as a copy.

None means that nobody set a definition, or that the repository is closed.

abstractmethod find_resource(resource_id: str, version_constraint: str | None = None, repository_id: str | None = None) → GenomicResource | None[source]

Return the resource with the given id, or None.

Use this method when the id comes from outside and absence is an ordinary result. Use get_resource() when absence is a bug.

Parameters:

resource_id (str) – The id of the resource, without the version.
version_constraint (str or None, default None) – A constraint such as ">=2.0". None selects the latest version.
repository_id (str or None, default None) – Restricts the lookup to the repository with this id. The rule is the same as for get_resource().

Returns:

The matching resource. None means that no resource matches the id, the constraint and the repository id.

Return type:

GenomicResource or None

abstractmethod get_all_resources() → Generator[GenomicResource, None, None][source]

Return a generator over all resources of the repository.

Read the generator lazily. A repository can be large.

Yields:: GenomicResource – A resource of the repository. A repository without resources yields nothing.

abstractmethod get_resource(resource_id: str, version_constraint: str | None = None, repository_id: str | None = None) → GenomicResource[source]

Return the resource with the given id.

Use this method when the id is a constant and its absence is a bug. Use find_resource() when absence is an ordinary result.

Parameters:

resource_id (str) – The id of the resource, without the version.
version_constraint (str or None, default None) – A constraint such as ">=2.0" or "=2.1". None selects the latest version.
repository_id (str or None, default None) – Restricts the lookup to the repository with this id, anywhere in the tree of this repository. An empty string or None means no restriction.

Returns:

The matching resource. The method never returns None.

Return type:

GenomicResource

Raises:

FileNotFoundError – If no resource matches and the repository is backed by a protocol (directory, http, s3, in-memory).
ValueError – If no resource matches and the repository is a group repository.

abstractmethod invalidate() → None[source]

Drop the cached state of the repository.

The next access loads the resource list, the metadata and the file contents again.

property repo_id: str: The identifier of the repository.

abstractmethod search_resources(search_term: str | None = None, resource_type: str | None = None, resource_query: str | None = None) → Generator[GenomicResource, None, list[tuple[str, str]] | None][source]

Search the resources by a full-text term, a type or a query.

The method returns a generator, so it reads the repository lazily. A resource must match every filter that you supply.

Parameters:

search_term (str or None, default None) – A full-text term. None means no full-text filter.
resource_type (str or None, default None) – A resource type, for example position_score. None means no type filter.
resource_query (str or None, default None) – A wildcard query over the ids and the labels of the resources. None means no query filter.

Yields:

GenomicResource – A matching resource. With no filter, every resource matches.

Returns:

The generator returns, when it ends, a (repository id, reason) pair for each child that a group repository skipped while it still answered. None and an empty list both mean that nothing was skipped. A for loop discards this value.

Return type:

list of tuple of (str, str) or None

Raises:

SearchTermError – If the search term cannot be parsed.
SearchIndexUnavailableError – If the repository has no usable search index.

search_resources_by_child(search_term: str | None = None, resource_type: str | None = None, resource_query: str | None = None) → Generator[tuple[GenomicResourceRepo, GenomicResource], None, list[tuple[str, str]] | None][source]

Search the resources, and pair each hit with its repository.

The filters mean the same as for search_resources().

Parameters:

search_term (str or None, default None) – A full-text term. None means no full-text filter.
resource_type (str or None, default None) – A resource type. None means no type filter.
resource_query (str or None, default None) – A wildcard query over the ids and the labels. None means no query filter.

Yields:

tuple of (GenomicResourceRepo, GenomicResource) – A matching resource and the repository that serves it. For a group, this is the child that holds the resource.

Returns:

The skipped children, as for search_resources().

Return type:

list of tuple of (str, str) or None

INDEX_COLUMN_PATTERN = '[A-Za-z_][A-Za-z0-9_]*': What an FTS index column may be named. Every name the index build creates is vetted against this before it becomes a column, because a column name cannot be bound as a parameter and so has to be spliced into SQL (gain#464).

class Manifest[source]

Bases: object

Manages file listings and checksums for a genomic resource.

A manifest maintains a catalog of all files in a resource with their sizes and MD5 checksums. This enables data integrity verification, efficient caching, and incremental updates.

The manifest is typically stored in a .MANIFEST file within the resource directory and is automatically loaded when accessing the resource.

entries: dict[str, ManifestEntry]

static from_file_content(file_content: str) → Manifest[source]

Create a manifest from the YAML content of a manifest file.

Parameters:

file_content (str) – YAML text that holds a list of entries with the keys name, size and md5. Empty YAML text gives an empty manifest.

Returns:

The manifest with one entry for each item of the list.

Return type:

Manifest

Raises:

KeyError – If an item of the list has no name, size or md5 key.
yaml.YAMLError – If the text is not valid YAML.

get_files() → list[tuple[str, int]][source]

Return the name and size of each file in the manifest.

Returns:: A (name, size) pair for each file. The list is empty when the manifest has no entries.
Return type:: list of tuple of (str, int)

names() → set[str][source]

Return the names of the files in the manifest.

Returns:: The file names. The set is empty when the manifest has no entries.
Return type:: set of str

class ManifestEntry(name: str, size: int, md5: str | None)[source]

Bases: object

Represents a file entry in a genomic resource manifest.

A manifest tracks all files within a resource with their sizes and checksums to ensure data integrity and enable efficient caching.

Variables:

name (str) – Relative path to the file within the resource
size (int) – File size in bytes
md5 (str | None) – MD5 checksum of file content, or None if not computed

md5: str | None

name: str

size: int

class ManifestUpdate(manifest: Manifest, entries_to_delete: set[str], entries_to_update: set[str])[source]

Bases: object

Represents a set of changes to apply to a manifest.

Used during resource synchronization to track which files need to be deleted or updated.

Variables:

manifest (gain.genomic_resources.repository.Manifest) – The updated manifest with all changes applied
entries_to_delete (set[str]) – Set of filenames to remove
entries_to_update (set[str]) – Set of filenames that need updating

entries_to_delete: set[str]

entries_to_update: set[str]

manifest: Manifest

class Mode(*values)[source]

Bases: Enum

Enumeration of repository protocol access modes.

Variables:

READONLY – Protocol supports only read operations
READWRITE – Protocol supports both read and write operations

READONLY = 1

READWRITE = 2

RESOURCE_ID_CHARACTER_CLASS = 'a-zA-Z0-9/._-': Every character a resource id may be spelled with, as the body of a regex character class. One definition, composed into the pattern that accepts an id, the one that names what a malformed one carries and the one that accepts a single segment, so the three cannot drift apart – they used to be written out separately, and a rule whose single source of truth is a test is a rule waiting to disagree with itself (gain#1352). - stays last: anywhere else it would read as a range.

class ReadOnlyRepositoryProtocol(proto_id: str, url: str)[source]

Bases: ABC

Abstract base class for read-only repository storage protocols.

A protocol defines how to access genomic resources from a specific storage backend (local filesystem, HTTP server, S3 bucket, etc.). Read-only protocols can retrieve resources but cannot modify them.

Subclasses must implement methods for:

Listing available resources
Reading configuration files
Opening resource files
Loading manifests

Variables:

proto_id – Unique identifier for this protocol instance
url – Base URL or path to the repository root
CHUNK_SIZE – Default read-buffer size for chunked file operations (1 MiB). This is the application-level read size for the download and md5 loops, not the network transfer unit – fsspec does its own block-level fetching/buffering underneath. Larger chunks cut Python-loop and progress-callback overhead on multi-GB resources.

CHUNK_SIZE = 1048576

build_genomic_resource(resource_id: str, version: tuple[int, ...], config: dict | None = None, manifest: Manifest | None = None) → GenomicResource[source]

Build a genomic resource instance using this protocol.

Parameters:

resource_id – Resource identifier like “hg19/gene_models/refseq”
version – Version tuple like (1, 2, 3)
config – Optional pre-loaded configuration dict. If None, will load from genomic_resource.yaml
manifest – Optional pre-loaded manifest. If None, will load when first accessed

Returns:

GenomicResource instance configured with this protocol

compute_md5_sum(resource: GenomicResource, filename: str) → str[source]: Compute a md5 hash for a file in the resource.

abstractmethod file_exists(resource: GenomicResource, filename: str) → bool[source]: Check if given file exist in give resource.

find_resource(resource_id: str, version_constraint: str | None = None) → GenomicResource | None[source]: Return requested resource or None if not found.

abstractmethod get_all_resources() → Generator[GenomicResource, None, None][source]: Return generator for all resources in the repository.

abstractmethod get_all_resources_dict() → dict[str, GenomicResource][source]: Return dictionary for all resources in the repository.

get_file_content(resource: GenomicResource, filename: str, *, uncompress: bool = True, mode: str = 't') → Any[source]: Return content of a file in given resource.

get_id() → str[source]

Return the repository protocol identifier.

Returns:: Protocol ID string

get_manifest(resource: GenomicResource) → Manifest[source]: Load and returns a resource manifest.

abstractmethod get_public_url() → str[source]

Return the public base URL of the repository.

Returns:: URL or path string pointing to a public repository root

get_resource(resource_id: str, version_constraint: str | None = None) → GenomicResource[source]

Return requested resource or raises exception if not found.

In case resource is not found a FileNotFoundError exception is raised.

get_resource_file_url(resource: GenomicResource, filename: str) → str[source]: Return url of a file in the resource.

get_resource_url(resource: GenomicResource) → str[source]

Return url of the specified resources.

The resource id is the other operand of this join and is no less untrusted than a file name – on the remote path it is read verbatim out of the repository’s .CONTENTS.json.gz – so it is contained here, at the join, exactly as get_resource_file_url contains the name (gain#467).

abstractmethod get_url() → str[source]

Return the base URL of the repository.

Returns:: URL or path string pointing to repository root

abstractmethod invalidate() → None[source]: Invalidate internal cache of repository protocol.

abstractmethod load_manifest(resource: GenomicResource) → Manifest[source]: Load resource manifest.

load_yaml(resource: GenomicResource, filename: str) → Any[source]: Return parsed YAML file.

mode() → Mode[source]

Return repository protocol mode.

Returns:: Mode.READONLY for this base class

abstractmethod open_bigwig_file(resource: GenomicResource, filename: str) → Any[source]

Open a bigwig file in a resource and return it.

Not all repositories support this method. Repositories that do no support this method raise and exception.

open_fasta_file(resource: GenomicResource, filename: str, index_filename: str | None = None, compressed_index_filename: str | None = None) → FastaFile[source]

Open a bgzipped fasta file in a resource and return a FastaFile.

Not all repositories support this method. Repositories that do not support this method raise an exception.

abstractmethod open_raw_file(resource: GenomicResource, filename: str, mode: str = 'rt', **kwargs: str | bool | None) → IO[source]: Open file in a resource and returns a file-like object.

abstractmethod open_repository_metadata() → Connection[source]: Open the db file for repo metadata and return the connection.

abstractmethod open_tabix_file(resource: GenomicResource, filename: str, index_filename: str | None = None) → TabixFile[source]

Open a tabix file in a resource and return a pysam tabix file.

Not all repositories support this method. Repositories that do no support this method raise and exception.

abstractmethod open_vcf_file(resource: GenomicResource, filename: str, index_filename: str | None = None) → VariantFile[source]

Open a vcf file in a resource and return a pysam VariantFile.

Not all repositories support this method. Repositories that do no support this method raise and exception.

search_resources(search_term: str | None = None, resource_type: str | None = None, resource_query: str | None = None) → Generator[GenomicResource, None, None][source]

Search for resources using SQLite full-text search.

The three filters conjoin: a resource must satisfy every one that is supplied. A search_term is matched against the FTS index, and when one is supplied the resource_type and the id glob of the resource_query – the annotator wildcard language, an id glob plus an optional label query – join it in the same statement, so the index narrows once rather than handing rows to a filter.

Without a search_term the index is never opened: the type and the query are matched in Python, over every resource. That is what makes them work on a repository with no .CONTENTS.sqlite3.gz at all, where opening the metadata db raises. Only the term genuinely needs the index – FTS5 tokenisation is not reproducible in Python, while a type is one token every resource carries (gain#1212).

Both routes evaluate the same parsed query, and every label clause is answered against the resource’s own meta.labels rather than out of the value the index recorded (gain#646) – so the two agree on every resource the index knows about, whatever a published index that has fallen behind its resources says about their labels. What an index too old to name a resource at all cannot do is return it, and that much is inherent: only the index can answer a search_term. Rebuilding it with grr_manage is what makes a newly added resource findable.

An empty resource_query is an unset one: it is what a shell substitutes for a variable that was never set, and the useful reading of -q "$SELECTOR" with no selector is the one that behaves like omitting the flag. A blank search_term is unset for the same reason (gain#633), and normalising it here – ahead of the branch below that decides whether to open the index at all – is what keeps -s "" from demanding an index it has no filter to apply: "" is not None, so it used to reach MATCH and be rejected by FTS5, or, on a repository with no index, be reported as a search that cannot be run.

Whitespace counts as blank: -s "$VAR " is the same accident as -s "$VAR". What FTS5 would otherwise make of it depends on which space was typed, and neither answer is worth keeping – a run of ASCII spaces is the empty expression it rejects, while a non-breaking space is a term it accepts and nothing contains, so the one accident was an error or a silently empty result. A term that has content is passed on untouched, spaces and all – ref_genome : hg38 is one term.

resource_type is normalised the same way, for the same reason (gain#653): blank, it selected nothing where it was meant to select everything, and asked an index-less repository for an index to apply a filter nobody set.

Raises ResourceQueryParseError for a malformed resource_query – eagerly, when called, rather than on the first iteration, so a caller can still report it against the argument that caused it.

class ReadWriteRepositoryProtocol(proto_id: str, url: str)[source]

Bases: ReadOnlyRepositoryProtocol

Abstract base class for read-write repository storage protocols.

Extends ReadOnlyRepositoryProtocol with write capabilities including:

Creating and updating resources
Managing manifests
File upload and deletion
Resource versioning

This protocol type is used for local repositories and writable remote storage backends where resources can be modified or created.

abstractmethod build_content_file() → list[dict[str, Any]][source]: Build the content of the repository (i.e ‘.CONTENTS.json.gz’).

build_manifest(resource: GenomicResource, prebuild_entries: dict[str, ManifestEntry] | None = None, *, verify_content: bool = False) → Manifest[source]: Build full manifest for the resource.

Build resource file state.

Each of md5, timestamp, size and change_token is read off the stored file when it is not supplied. Supplying a digest that is already in hand – the download path verifies one against the manifest before publishing the file – saves reading the whole file back out of the store, which for a remote store is a second transfer of it. Supplying the other three saves a stat each, and the download path has all of them from the stat it makes to verify its own move (gain#936).

change_token defaults to Unread.UNREAD rather than to None because None is a value a token legitimately takes: it is what a store offering no tokens reports, and it is the value a caller holding one such answer needs to be able to pass.

The parameters are keyword-only and named explicitly so that a misspelled one is a TypeError here rather than a silently ignored value – which for md5 costs a full re-read of the file, and for the others a stat. See gain#865.

A file that is not there is still refused with a ValueError naming the resource and the filename – but it is noticed by the reads rather than ahead of them. It is worth being plain about what the check that used to run first was: not a safeguard, but a message. Every read below raises FileNotFoundError of its own accord on a file that is not there; all the check added was that the caller hears the ValueError instead. A message can be phrased where the failure happens, and asking first cost a probe of a key the caller may have just been told about – the cache verdict opens by stating the very file whose state it then rebuilds, and paid for that twice (gain#1039). A caller that supplies every field reads nothing, so it raises nothing here, exactly as before.

check_update_manifest(resource: GenomicResource, prebuild_entries: dict[str, ManifestEntry] | None = None, *, verify_content: bool = False, save_state: bool = True) → ManifestUpdate[source]

Check if the resource manifest needs update.

With save_state=False nothing it derives is recorded (#257).

abstractmethod collect_all_resources() → Generator[GenomicResource, None, None][source]

Scan repository and yield all resources.

Returns:: Generator yielding GenomicResource instances for each resource found in the repository

collect_resource_entries(resource: GenomicResource) → Manifest[source]

Scan resource directory and build manifest from files found.

The entries-only view of scan_resource_entries(), for callers that just want the names and sizes on disk. A caller that WRITES a manifest must use the scan itself, so that a file it could not describe fails the resource instead of vanishing from the manifest (gain#503).

Parameters:: resource – Resource to scan
Returns:: Manifest containing entries for all files in the resource

copy_resource(remote_resource: GenomicResource) → GenomicResource[source]: Copy a remote resource into repository.

abstractmethod copy_resource_file(remote_resource: GenomicResource, dest_resource: GenomicResource, filename: str) → ResourceFileState | None[source]: Copy a remote resource file into local repository.

abstractmethod delete_resource_file(resource: GenomicResource, filename: str) → None[source]: Delete a resource file and it’s internal state.

get_manifest(resource: GenomicResource) → Manifest[source]: Load or build a resource manifest.

get_or_create_resource(resource_id: str, version: tuple[int, ...]) → GenomicResource[source]

Return a resource with specified ID and version.

If the resource is not found create an empty resource.

abstractmethod get_resource_file_change_token(resource: GenomicResource, filename: str) → str | None[source]

Return the store’s change token for a resource file, if any.

A change token is whatever the store itself offers as “this is the version of the object you are looking at”: it changes on every write and holds still for as long as the object is not written. Stores that offer none answer None, and for them the modification time remains the only change hint there is.

The value is opaque. It is never parsed, never compared against an md5 sum and never assumed to be one, even where a particular store happens to derive it from one.

See ADR 0022 for why a state is judged by this rather than by the modification time.

abstractmethod get_resource_file_size(resource: GenomicResource, filename: str) → int[source]: Return the size of a resource file.

abstractmethod get_resource_file_timestamp(resource: GenomicResource, filename: str) → float[source]: Return the timestamp (ISO formatted) of a resource file.

abstractmethod load_resource_file_state(resource: GenomicResource, filename: str) → ResourceFileState | None[source]

Load resource file state from internal GRR state.

If the specified resource file has no internal state returns None.

mode() → Mode[source]

Return repository protocol mode.

Returns:: Mode.READWRITE for this protocol type

abstractmethod publish_raw_file(resource: GenomicResource, filename: str, mode: str = 'wt') → AbstractContextManager[IO][source]

Open a resource file for a write that replaces it, or does not.

The publishing counterpart of open_raw_file(): what the caller writes reaches filename only once the handle has closed cleanly. A write that fails part-way – and an interrupt – leaves whatever was published before exactly as it was, rather than truncating it in place with nothing to roll back to (gain#933).

A separate name rather than a flag on open_raw_file(): the two differ in what they guarantee, not in a parameter, and the callers that need the guarantee are not the ones that need a plain handle.

save_index(resource: GenomicResource, contents: str) → None[source]: Save an index HTML file into the genomic resource’s directory.

save_manifest(resource: GenomicResource, manifest: Manifest) → None[source]: Save manifest into genomic resource’s directory.

abstractmethod save_resource_file_state(resource: GenomicResource, state: ResourceFileState) → None[source]: Save resource file state into internal GRR state.

abstractmethod scan_resource_entries(resource: GenomicResource) → ResourceScan[source]

Scan resource directory for its files.

Parameters:: resource – Resource to scan
Returns:: A ResourceScan holding the entries that could be described and the names of the files that could not.

update_manifest(resource: GenomicResource, prebuild_entries: dict[str, ManifestEntry] | None = None, *, verify_content: bool = False) → Manifest[source]: Update or create full manifest for the resource.

update_resource(remote_resource: GenomicResource, files_to_copy: set[str] | None = None) → GenomicResource[source]

Copy a remote resource into repository.

Allows copying of a subset of files from the resource via files_to_copy. If files_to_copy is None, copies all files.

abstractmethod update_resource_file(remote_resource: GenomicResource, dest_resource: GenomicResource, filename: str) → ResourceFileState | None[source]: Update a resource file into repository if needed.

class ResourceFileState(filename: str, size: int, timestamp: float, md5: str, change_token: str | None = None)[source]

Bases: object

Tracks the state of a resource file in internal repository storage.

Used for caching and synchronization to determine if files need to be refreshed or re-downloaded.

Variables:

filename (str) – Relative path to the file within the resource
size (int) – File size in bytes
timestamp (float) – Last modification time as Unix timestamp
md5 (str) – MD5 checksum of file content
change_token (str | None) – Opaque token the store supplies for the stored object, or None where the store has none. It changes whenever the object changes, and nothing else about it is defined – it is never parsed, and never treated as a checksum, even when a particular store happens to derive it from one.

change_token: str | None = None

filename: str

md5: str

size: int

timestamp: float

class ResourceScan(manifest: ~gain.genomic_resources.repository.Manifest, unreadable: ~collections.abc.Mapping[str, str], change_tokens: ~collections.abc.Mapping[str, str | None] = <factory>)[source]

Bases: object

What one scan of a resource’s directory found.

unreadable maps each file the scan listed but could not stat – a dangling symlink, a symlink loop, a directory it may not traverse – to why. They are NOT in manifest: there is no size to put there.

They are carried rather than raised because the scan cannot yet know whether they matter. A DVC-managed file materialised as a link into a shared cache is unreadable exactly when that cache has been garbage collected, and its .dvc sidecar still describes it perfectly – so it is manifested from the sidecar and nothing is wrong. Only a name that NOTHING can describe is a broken resource, and that is not known until the sidecars have been merged in (gain#503).

The reason travels with the name so that whoever DOES know the outcome can report it: a resource is scanned more than once per command, so the scan itself is the wrong place to say anything the user should see exactly once.

change_tokens is the store’s change token for each name the scan stat’ed, read by the same stat as its size, so a state built for that file needs neither read again. It is carried here rather than in the manifest because ManifestEntry is the serialized .MANIFEST row.

change_token(name: str) → str | Unread | None[source]

The token the scan read for name, or Unread.UNREAD.

UNREAD for a name the scan recorded no token for, never None: None is what a store without tokens reports, and passing it on would record “this store has none” for a file nobody asked about.

change_tokens: Mapping[str, str | None]

manifest: Manifest

unreadable: Mapping[str, str]

exception SearchIndexUnavailableError(repo_id: str, reason: str)[source]

Bases: ValueError

A repository that cannot apply a search filter for want of an index.

Raised when the repository publishes no .CONTENTS.sqlite3.gz at all, and when the one it publishes carries no contents table because no resource could be indexed into it. Both are repository health: the filter is unobjectionable and a grr_manage repo-repair would let it be applied.

Typed rather than left as a bare ValueError because a group repository absorbs this to skip the child and carry on (ADR 0012), and the layers it absorbs it through raise ValueError of their own that must keep propagating – the cache layer resolving a resource it cannot place, for one.

A ValueError still, so a caller that only ever distinguished bad arguments from working ones is unaffected.

exception SearchTermError(search_term: str, cause: Exception)[source]

Bases: ValueError

A search term that SQLite’s FTS5 could not parse as a match expression.

The term is bound, never interpolated, so this is not a failure to contain it – it is that FTS5 reads a bound term as an expression, with a grammar of its own: quotes, AND/OR/NOT, NEAR, column : value. A term that does not parse is the caller’s mistake, and apsw.SQLError reports it as neither – it names no term, no argument, and reads like the database broke (gain#632).

A ValueError, like ResourceQueryParseError, so the two bad arguments this search can be given are handled the same way by the endpoint and the CLI that take them.

The column filter is why the term is not simply quoted into a literal before it reaches MATCH: label keys are index columns, and ref_genome : hg38 is a supported search. Quoting would make that a search for the text.

class Unread(*values)[source]

Bases: Enum

A keyword that was not supplied, where None is a real value.

An enum rather than a bare object() so that the sentinel has a type a signature can name and a type checker can narrow: comparing a parameter against the member leaves str | None in the other branch, which is what the field actually is. Public because it appears in the signature of a public method, and a subclass that overrides it has to be able to name it.

UNREAD = 1

Not supplied.

ReadWriteRepositoryProtocol.build_resource_file_state() and _state_describes_stored_file read a field off the stored file when the caller does not supply it, and for every other field None says so unambiguously. A change token is the exception: None is the answer a store offering no tokens gives, so a caller that has asked and been told “no token” must be able to say that, and it must not read as “go and ask”.

exception UnreadableResourceFilesError(resource_id: str, names: Sequence[str])[source]

Bases: ValueError

Files of ONE resource that could not be read or described.

Collected rather than raised on the first offender, so a single run reports every one of them. A ValueError so that cli_errors.report_resource_failure renders it as one line naming the resource and carrying the cause, with the traceback demoted to DEBUG (gain#364) – and so that one broken resource fails itself instead of aborting the repository-wide command (gain#503).

collect_dvc_entries(proto: ReadWriteRepositoryProtocol, res: GenomicResource) → dict[str, ManifestEntry][source]

Collect manifest entries defined by .dvc files.

A .dvc file that cannot be read, does not parse as a pointer for the data file it sits next to, or declares no usable md5 sum and size is skipped with a warning - never propagated into the manifest, and never allowed to abort the command. .dvc sidecars are read on every grr_manage run, and the repository scan that produced this entry has already tolerated the very same content (see FsspecReadWriteProtocol._is_dvc_managed_leaf); the two classify identically because both delegate to dvc.parse_dvc_pointer_out().

A well-formed sidecar for a dvc add <dir> output is a different matter: it is not ignored, it is REFUSED. GAIn cannot verify a .dir md5 sum - it hashes a DVC cache object, not any file GAIn can read - so writing it into the manifest would be a false clean bill of health, and quietly skipping the directory would leave its data unmanifested and unverified. Either way the resource would be certified without its content ever being checked, so the command fails instead (#255). This is the gate every grr_manage subcommand that builds or checks a manifest passes through – and, since #721, the fallback build a repository walk triggers – and it applies whether or not the directory is materialised. It is kept even though cli_dvc.refuse_dvc_directory_outputs refuses such a resource before any command reaches this function: a manifest must never be built from a sidecar GAIn cannot verify, whoever asks for it (#284).

An entry is produced for every readable sidecar. Every materialised file’s entry is consulted by ReadWriteRepositoryProtocol._update_manifest_entry_and_state() - the sidecar IS the md5 sum of the file it describes - and the entries for files the scan did not yield are merged by ReadWriteRepositoryProtocol._merge_unscanned_dvc_entries() (#373).

Lives beside the manifest builder, not in the CLI, because the builder itself must reach it: a manifest built as a FALLBACK - a repository walk meeting a resource that never had a .MANIFEST - has no CLI frame above it to collect the sidecars, and building without them hashes every DVC-managed byte the sidecar already describes (#721).

Raises:: UnsupportedDvcDirectoryOutputError – the resource has a dvc add <dir> output.

drain_search(hits: Generator[HitT, None, list[tuple[str, str]] | None]) → tuple[list[HitT], list[tuple[str, str]]][source]

Exhaust a search, answering its rows and its skips.

The consumption idiom for a caller that presents totals: a for loop silently discards the skips a group reports on its generator’s return value (gain#686). None and [] both arrive as [], so the dual spelling ends here.

dvc_directory_output_message(resource_id: str, entry_name: str, filename: str) → str[source]

Say why a dvc add <dir> output is refused, and what to do.

One text for both gates – cli_dvc’s pre-flight and the manifest builder – so what the user is told cannot depend on which of them saw the sidecar first (#284).

Every name here is untrusted GRR content and this message is a refusal report, so the names are escaped for the same reason the sibling warnings in collect_dvc_entries() are (gain#642).

is_generated_info_page(name: str) → bool[source]

Return True if name is a page resource-info generates.

Membership in a resource’s manifest is decided by the file’s PATH: the two pages GAIn writes itself are excluded, everything else is resource data. The rule it replaced – “drop every name ending in html” – was a proxy for the same question and a bad one, since it silently dropped any html file a resource legitimately carries as data (#373).

is_gr_id_token(token: str) → bool[source]

Check if token can be used as a genomic resource ID.

Genomic Resource Id Token is a string with one or more letters, numbers, ‘.’, ‘_’, or ‘-’. The function checks if the parameter token is a Genomic REsource Id Token.

is_safe_repo_id(repo_id: str) → bool[source]

Check if repo_id is usable as a single filesystem path segment.

A repository id names a directory: a cached repository derives each repository’s cache directory by joining the id onto the cache url. An id that is not a single path segment therefore decides where the process writes – .. climbs out of the configured cache directory, and an absolute id makes os.path.join discard the cache url altogether. Such an id is a configuration error and is rejected, never rewritten (#460).

Safe means exactly one non-empty path segment that still names that same segment after a round trip through a url. Each check earns its keep separately: the separator check rules out sub/dir, ../../escaped and an absolute /etc/grrcache – and, because a UNC prefix (\\server\share, //server/share, \\?\C:\x) always carries separators, those too; ntpath.splitdrive() adds the one absolute-ish prefix that has no separator in it, C:cache, which os.path.join on Windows resolves against that drive’s current directory; the control-character check covers the id that changes shape when it is parsed as part of a url (see _UNSAFE_NAME_CHARACTER_RE); and . and .. are spelled out because they are ordinary segments to every one of the checks above.

An empty id is not a segment either, but it is not a traversal: a falsy id already means “unnamed” everywhere it is read (_resolve_repo_id synthesises one, find_resource treats it as “no filter”), so its callers decide what to do with it rather than this predicate.

Deliberately NOT built on is_gr_id_token(). That helper enforces a character class ([a-zA-Z0-9._-]), which is both too weak and too strong here: is_gr_id_token("..") is True – .. matches the class in full, and a single-segment .. still escapes one directory level – while ids that are perfectly safe as directory names (a space, say) would start failing for a reason that has nothing to do with path safety. This check answers only the path question.

is_version_constraint_satisfied(version_constraint: str | None, version: tuple[int, ...]) → bool[source]

Check if a version matches a version constraint.

Supports two types of constraints:

“=X.Y.Z”: Exact match required
“>=X.Y.Z” or “X.Y.Z”: Minimum version required (default)

Parameters:

version_constraint – Constraint string like “>=1.2.0” or “=1.2.3”. None or empty string matches any version.
version – Version tuple to check like (1, 2, 3)

Returns:

True if the version satisfies the constraint

Raises:

ValueError – If constraint has invalid syntax or unknown operator

malformed_resource_id_reason(resource_id: str) → str | None[source]

Return why resource_id is not a well-formed id, or None.

Containment is the other rule an id is held to, and the two are separate: uncontained_resource_id_reason() asks whether the id escapes the repository, this one whether GAIn can process it at all. Both run where a .CONTENTS is read, and an id refused by either is dropped with a warning, the rest of the .CONTENTS still served. A scan runs neither: it parses each candidate path with parse_gr_id_version_token(), which enforces the same character class by construction, and a directory the parse refuses is skipped with a warning the same way. The two enumeration paths agree on both the grammar and its consequence.

Two segments are refused here that the scan grammar does not refuse – both / and . are inside its character class, so a//b and a/./b match it – because a scan never meets them: a filesystem has no empty name to offer, and every dot-named directory is skipped on the way down. Only a hand-written .CONTENTS can carry either. The . segment is the quieter of the two: it survives the join, the filesystem resolves it away, and the cached directory is then enumerated under a different id than the one it was served as (gain#1385).

"" and "." are exempt because both name the repository root, a supported resource in its own right that is published under the empty id and addressed as "." by build_local_resource. The exemption is by whole id: "." as a segment of a longer id is the refusal above.

parse_gr_id_version_token(token: str) → tuple[str, tuple[int, ...]][source]

Parse a genomic resource id with an optional version suffix.

The suffix has the form (3.3.2); without one the version is (0,). The empty token names the repository root. Returns the (resource_id, version) tuple, and raises ValueError on a token outside the resource id grammar.

parse_resource_id_version(resource_path: str) → tuple[str, tuple[int, ...] | None][source]

Parse a resource path into an (resource_id, version) tuple.

Like parse_gr_id_version_token(), but a path without a version suffix yields None for the version rather than (0,), so a caller can tell “unversioned” from “version 0”. Raises ValueError on a path outside the resource id grammar.

report_uncontained_manifest_entries(resource_id: str, manifest: Manifest) → None[source]

Warn about – and never raise on – entries that escape the resource.

Rejecting a poisoned entry while parsing the manifest reads as defence in depth, but a manifest is parsed while ENUMERATING a repository: one bad entry then kills the generator before a single resource is yielded, and list, repo-repair and even resource-repair on an unrelated healthy resource all die with it. That is the gain#464 shape – one poisoned resource costing the whole repository – and it is not worth paying, because the load-bearing check sits at the join in get_resource_file_url() and fails the poisoned name loudly the moment anything tries to USE it.

So this only supplies the attribution the raise used to: a warning that names the resource AND the entry, which the raise could not do because a ManifestEntry does not know which resource it belongs to.

resolve_tabix_index_filename(manifest: Manifest, filename: str) → str | None[source]

Return the tabix index of filename as recorded in manifest.

Resolution is manifest-driven on purpose: the manifest is already loaded and is protocol-agnostic, whereas probing with file_exists costs a network round-trip per candidate on the http and s3 protocols.

Returns None when the manifest records neither index – the caller decides whether that is a warning (the file set of an implementation) or an error (an open that needs an index). See gain#430.

resolve_tabix_index_filename_for_read(resource: GenomicResource, filename: str) → str[source]

Return the index to read filename with, never building a manifest.

Consults the manifest only when it is already loaded or can be loaded from the resource: an open must stay a pure read, and GenomicResource.get_manifest() would build – md5-scanning the whole resource and writing state files – for a resource that carries no .MANIFEST (gain#430).

With no manifest at hand – the hand-authored GRR directory shape that collect_all_resources tolerates – falls back to probing the resource for each candidate index. The probe is deliberately confined to this branch: a manifest-backed resource must never pay the per-candidate network round-trip the probe costs on the http and s3 protocols.

With a manifest that records no index at all, or when no probe succeeds, falls back to the historical .tbi guess so that whatever pysam raises still names a concrete path.

uncontained_resource_file_name_reason(filename: str) → str | None[source]

Return why filename is not resource-contained, or None.

Resource file names arrive from GRR content – the resource’s genomic_resource.yaml and its .MANIFEST – which is fetched from remote repositories and is therefore untrusted. A name is contained when it is relative and names no .., . or empty segment; nested names such as statistics/histogram_score.json are ordinary and stay allowed.

The joined location is a URL, not an os path, so the name is checked both as written and percent-decoded: an http(s) server decodes the path before resolving it, which makes %2e%2e a traversal there. A single decoding pass is the right depth – %252e%252e decodes to the literal text %2e%2e, which no server resolves any further.

A .. that would stay inside the resource (sub/../other.txt) is rejected as well, because the three backends GAIn speaks to disagree about what it means: yarl/aiohttp normalises it away client-side before the request is even sent, an s3 server rejects the key outright (RustFS answers InvalidArgument, MinIO XMinioInvalidResourceName), and a local filesystem resolves it. One name, three outcomes – so it is refused everywhere rather than left to mean whatever the protocol of the day decides.

A degenerate name – empty, blank, ., or carrying an empty segment – is refused too: open_raw_file("") addressed the resource DIRECTORY, and no resource file is legitimately spelled that way. See gain#467.

A name carrying a control character is refused as well – not a containment failure but a reporting one, since the name is logged unescaped. See gain#642.

uncontained_resource_id_reason(resource_id: str) → str | None[source]

Return why resource_id is not repository-contained, or None.

A resource id is the other operand of the same join a file name goes through: get_resource_url joins it onto the repository url. On the remote path it is read verbatim out of the repository’s .CONTENTS.json.gz, so it is exactly as untrusted as a manifest entry name – and containing only the file name left the escape wide open through its sibling (gain#467).

"" and "." are contained: both name the repository root, which is a supported resource in its own right – proto_builder addresses it as "" and build_local_resource as ".". Only the escape itself is refused here, not the degenerate spellings a file name is also held to: an id is joined once, at the root, so a . segment in it is a no-op rather than a way to address something else. (It is still refused – by malformed_resource_id_reason(), on the different ground that no scan can ever produce it.)

An id carrying a control character is refused, which is a reporting concern rather than a containment one – the id is logged unescaped at many call sites, and a newline in it forges a log line. See gain#642.

validate_resource_file_name(resource_id: str, filename: str) → None[source]: Raise ValueError unless filename stays inside the resource.

validate_resource_id(resource_id: str) → None[source]: Raise ValueError unless resource_id stays inside the repo.

version_tuple_to_string(version: tuple[int, ...]) → str[source]

Convert version tuple to string representation.

Parameters:: version – Version tuple like (1, 2, 3)
Returns:: String representation like “1.2.3”

version_tuple_to_suffix(version: tuple[int, ...]) → str[source]

Transform version tuple into resource ID version suffix.

The suffix is used to append version information to resource IDs. Default version (0,) produces no suffix.

Parameters:: version – Version tuple like (1, 2, 3)
Returns:: Empty string for version (0,), otherwise “(1.2.3)” format

gain.genomic_resources.repository_factory module

Provides a factory for building genomic resources repostiories.

Bases: _RepoDefinitionBase

Definition for an in-memory genomic resource repository.

cache_dir: _PathOrStr | None

content: dict[str, Any] | None

model_config = {'extra': 'forbid', 'hide_input_in_errors': True}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

type: Literal['embedded', 'memory']

Bases: _RepoDefinitionBase

Definition for a local filesystem genomic resource repository.

cache_dir: _PathOrStr | None

directory: _PathOrStr

model_config = {'extra': 'forbid', 'hide_input_in_errors': True}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

read_only: bool | None

type: Literal['file', 'dir', 'directory']

Bases: _RepoDefinitionBase

Definition for a group of genomic resource repositories.

Child repository ids must be distinct across the whole subtree below a group, not merely among siblings: an id selects a repository (find_resource/get_resource take a repository_id) and, for a cached repository, names that repository’s cache directory. Two repositories sharing an id – at the same level or at different ones – leave the second unreachable. Duplicates are a configuration error, rejected here when the definition is validated.

An id is also a directory name – a cached repository derives each repository’s cache directory from it – so it must be a single path segment. One that is not (a separator, an absolute path, . or ..) would move cached data out of the configured cache_dir, and is rejected by the base definition model rather than rewritten (#460).

The uniqueness check covers children, not the definition root: the walk starts at self.children, so the root’s own id (explicit or synthesised) is not compared against any descendant’s here. It is not exempt – a repository_id naming the root selects it, like any other repository (#447), so a root sharing an id with a descendant shadows it. That pair is refused by _check_root_id_is_not_a_descendant_id in the top-level builder instead, which is the only place the root’s path through the definition tree is genuinely empty and its id can therefore be resolved as its parent would.

Spelling id on a child is optional. A child that omits it gets a deterministic id synthesised from its own identity – its url or directory, or its path from the definition root for an embedded / memory child or a nested group, which have neither. The synthesised id is never empty, and two children that would synthesise the same id (the same directory listed twice, say) are duplicates like any other.

cache_dir: _PathOrStr | None

check_child_ids_are_unique() → GroupRepoDefinition[source]

Reject a group whose descendants do not have distinct ids.

Compares the resolved ids – an explicit id where the child spells one, the synthesised id otherwise – so a pair that would end up sharing an id is refused whether or not the collision was spelled out.

The walk covers the whole subtree, not just the direct children. Repository ids share one namespace across nesting levels: a repository_id filter is matched against every repository in the tree, and a cached repository derives each child’s cache directory from its id, so two repositories at different levels sharing an id are as ambiguous as two siblings. Pydantic validates bottom-up, so a nested group has already checked its own subtree by the time this runs; repeating the walk from here is what catches the cross-level pairs a nested group cannot see.

children: list[RepoDefinition]

model_config = {'extra': 'forbid', 'hide_input_in_errors': True}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

type: Literal['group']

Bases: _RepoDefinitionBase

Definition for an HTTP/HTTPS genomic resource repository.

cache_dir: _PathOrStr | None

check_credentials_together() → HttpRepoDefinition[source]

model_config = {'extra': 'forbid', 'hide_input_in_errors': True}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

password: str | None

type: Literal['http']

url: str

user: str | None

warn_on_insecure_credentials() → HttpRepoDefinition[source]

Warn when basic-auth credentials ride a cleartext http:// URL.

Credentials are still accepted (localhost/dev GRRs legitimately use plain http), but a non-https URL to a non-local host means the base64-encoded credentials travel unencrypted, so emit a loud warning. The message never includes the password.

Bases: _RepoDefinitionBase

Definition for an S3 genomic resource repository.

cache_dir: _PathOrStr | None

endpoint_url: str | None

model_config = {'extra': 'forbid', 'hide_input_in_errors': True}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

type: Literal['s3']

url: str

class UrlRepoDefinition(*, id: str | None = None, public_url: str | None = None, type: Literal['url'], url: str, cache_dir: str | Path | None = None)[source]

Bases: _RepoDefinitionBase

Definition for a generic URL (http/https/s3) repository.

cache_dir: _PathOrStr | None

model_config = {'extra': 'forbid', 'hide_input_in_errors': True}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

type: Literal['url']

url: str

build_genomic_resource_group_repository(repo_id: str, children: list[GenomicResourceRepo]) → GenomicResourceRepo[source]

Build a repository that groups other repositories.

A lookup in the group asks its children.

Parameters:

repo_id (str) – The identifier of the group repository.
children (list of GenomicResourceRepo) – The repositories to group. They must be built already.

Returns:

The group repository. It has no definition, because the function takes repositories and not a definition.

Return type:

GenomicResourceRepo

build_genomic_resource_repository(definition: dict | None = None, file_name: str | None = None) → GenomicResourceRepo[source]

Build a repository from a definition dictionary or a YAML file.

Call the function with no arguments to build the default repository. The environment decides which repository that is: see get_default_grr_definition().

Parameters:

definition (dict or None, default None) – The definition of the repository. The key type selects the kind of repository, for example directory, http, s3 or group. The key cache_dir adds a local cache to any kind. None or an empty dictionary means that the function takes the definition from file_name or from the default definition.
file_name (str or None, default None) – The path of a YAML file that holds the definition. Do not pass it together with a non-empty definition.

Returns:

The repository. Its definition property holds the definition that built it.

Return type:

GenomicResourceRepo

Raises:

ValueError – If both definition and file_name are given, if no definition is found, or if the definition is not valid. The message does not contain passwords.

build_resource_implementation(res: GenomicResource) → GenomicResourceImplementation[source]: Build a resource implementation from a resource.

get_default_grr_definition() → dict[str, Any][source]

Return the definition of the default repository.

The function loads the file that get_default_grr_definition_path() finds. When there is no such file, it returns a copy of the built-in definition of the public IossifovLab GRR.

Returns:

A definition that build_genomic_resource_repository accepts. The caller can change the dictionary.

Return type:

dict

Raises:

OSError – If the definition file cannot be read.
yaml.YAMLError – If the definition file is not valid YAML.

get_default_grr_definition_path() → str | None[source]

Return the path of the default repository definition file.

The environment variable GRR_DEFINITION_FILE has priority. The function does not check that the file it names exists. Otherwise, the function looks for .grr_definition.yaml in the home directory.

Returns:: The path of the definition file. None means that the variable is not set and the home directory has no .grr_definition.yaml.
Return type:: str or None

load_definition_file(filename: str) → Any[source]: Load GRR definition from a YAML file.

redact_definition(definition: Any) → Any[source]

Return a deep copy of a GRR definition with credentials masked.

user/password values are replaced with "***" recursively (including inside a group repository’s children) so that a definition can be logged or embedded in an error message without leaking secrets. Credentials embedded in a URL’s userinfo (scheme://user:pass@host) are also scrubbed.

gain.genomic_resources.resource_errors module

The exceptions more than one tier must be able to name.

MalformedResourceError places the blame on the resource: its records or its configuration do not hold to what its kind can mean. It subclasses ValueError so it falls inside cli_errors.RESOURCE_ERRORS by construction, which is what makes grr_manage report it as one attributed line rather than as an unexpected internal error carrying a traceback (ADR 0008).

The module is a leaf – it imports nothing from GAIn – so the score layer, the table layer and the CLI tier can raise and catch the same exception without any of them acquiring a dependency on another.

HistogramError is here so that cli_errors.RESOURCE_ERRORS can name it without importing the histogram module; cli_errors says why it may not.

exception HistogramError[source]

Bases: Exception

A histogram-specific failure of one resource.

Raised for a categorical histogram past its cardinality limit, which the statistics scan catches and nullifies, and for a histogram file that cannot be read, which reaches grr_manage’s one-line reporting tier. A plain Exception, not a MalformedResourceError, because the second case is a fault of the resource’s state rather than of its records or configuration.

exception MalformedResourceError[source]

Bases: ValueError

A resource refused because of its own records or configuration.

Named for the state the resource is in rather than for any one rule, so that every refusal a reader could act on the same way – by fixing the resource – arrives under one name.

backwards_records_error(resource_id: str, chrom: str, pos: int, prev_pos: int, kind: str) → MalformedResourceError[source]

Refuse a score whose records move backwards along a contig.

The sibling of overlapping_records_error(), and here for the same reason. The position rule has been built from one helper since it was detected on two paths; this rule is detected on two paths for each of TWO kinds, and was written out at all four sites – six copies of one sentence, agreeing only for as long as nobody edited one of them.

kind is the possessive naming the kind whose promise was broken (“a fragment score’s”), so the message still says which validator fired. It is passed by each raise site rather than read off a shared class attribute: an attribute two validators interpret for themselves is what ADR 0008 unwound, and a message fragment is not a rule.

index_column_mismatch_error(resource_id: str, index_filename: str, mismatches: Sequence[tuple[str, int, int]], *, end_is_implied: bool = False) → MalformedResourceError[source]

Refuse a tabix table whose index was built over other columns.

Each entry of mismatches is a field, the column its index was built over, and the column its table resolves to – resolves, not states: a coordinate no configuration and no header names is read through a hardcoded fallback, and a fallback the index disagrees with splits the read from the filter exactly as a contradictory config entry does. Both columns are named because the remedy is a one-line edit and a reader cannot write it from either number alone. end_is_implied says the index records no end column at all, so its end column is its begin column by definition rather than by anyone’s choice.

Built here rather than at the raise site for the same reason overlapping_records_error() is: the rule is one rule, and it must read identically wherever it is reported.

inverted_span_error(chrom: str, pos_begin: int, pos_end: int, ref: str | None, alt: str | None) → OSError[source]

Refuse a single record whose end precedes its begin.

Returned rather than raised, so the raise stays at the site that read the slots – and so the several sites that perform this check share one message. Off the hot path by construction: a caller compares two integers per record and only calls this when the comparison fails.

Not to be confused with backwards_records_error(), despite the neighbouring vocabulary: that one refuses a resource whose records move backwards along a contig, raises MalformedResourceError, and belongs to the scan’s validation. This one is about a single record’s own two ends, and stays an OSError – the type the read path has always raised for it, and the type the tests pin.

Takes the five decoded values rather than the record they came from, for two reasons. Reading the slots here would mean importing them from genomic_position_table.record, whose package __init__ imports table_tabix, which imports this module – a cycle, and the end of this module’s leaf status (see the module docstring). And a record’s last slot is the backend’s payload, so a helper handed the whole record must be careful never to interpolate it: f"{record}" would print an entire pysam.VariantRecord – whose repr is the whole VCF line – or a TupleProxy. Named values cannot make that mistake.

ref and alt are what tell two records at one position apart, so the message names them when the record carries them, and says nothing in their place when it does not (ADR 0027).

no_only_score_message(resource_id: str, defined: Iterable[str]) → str[source]

State that a resource has no only score for a call with no score id.

A caller that omits the score id asks for the only score of the resource. A resource with several scores, or none, has no only score. The message names the resource and every score id it defines, sorted, so the caller can name the one it wants.

overlapping_records_error(resource_id: str, chrom: str, pos: int, prev_end: int) → MalformedResourceError[source]

Refuse a position score whose record claims a position already taken.

Built here rather than at either raise site because two paths detect this one rule – the per-record region read and the vectorized statistics scan – and a reader who meets the message from one of them must not have to wonder whether the other words it differently.

score_configuration_error(resource_id: str, score_id: str, detail: str) → MalformedResourceError[source]

Refuse a score whose DEFINITION says what the score cannot be.

The sibling of overlapping_records_error() for the other half of what makes a resource malformed: not a record that breaks its kind’s promise, but a definition claiming a value the score cannot hold. The rules live in three layers, and each raise site phrases its own detail: the VCF header/config merge (the _refuse_* helpers in vcf_scores, for an id, an address or a type: the header contradicts, and for a field declared Number=G), the construction convergence point (refuse_unfoldable_histograms, for a number histogram over a value type none accumulates, and refuse_unbuildable_aggregators, for an aggregator: no accumulator can be built from), and open, once a tabular table’s header is known (validate_scoredefs, for a column address the header cannot honour, and resolve_score_indices, which holds the DEFINITIONS – not the config – to the same address rules as it resolves each one to its payload column, and refuses a VCF definition with no INFO key). The Number=G rule is the one a scores: entry need not have caused: a header-only resource has none, and the claim is the header’s.

What is shared is the ADDRESS: which resource, and which score in it. That is the half a reader needs to find the file to edit, it is the half no raise site can word differently without sending someone to the wrong place, and it is why it is built here rather than at each site.

Not built here, on purpose: a refusal of a REQUEST – a score id the resource does not define, an aggregator a caller names that does not build, a bool score asked for with no aggregator named, a none_value_replacement of the wrong type. Those fire at read time against a definition that is fine, so the resource’s config is not the file to edit, and this prefix would send the reader there. They carry the other address, score '<id>' … resource '<resource>': the undefined score id is worded by undefined_scores_message(), the aggregator refusals in aggregation and the replacement refusal on the position score.

The prefix matches ResourceConfigValidationMixin so that a caller reading a config error sees one wording; the TYPE is MalformedResourceError, so a caller that already catches “this resource’s own config is bad” by type catches these too rather than matching on a string.

undefined_scores_message(resource_id: str, unknown: Iterable[str], defined: Iterable[str]) → str[source]

State that a caller named scores the resource does not define.

The one sentence for this mistake, whichever surface a caller reaches it through: an aggregation request, a read, a histogram accessor a gene score shares, a filter expression. It is a refusal of the REQUEST, so it carries the request address, not score_configuration_error()’s prefix: the resource’s config is fine, and the caller’s list is the thing to fix.

Returned as text rather than as an exception because the surfaces raise different types – ValueError from the score API, ScoreFilterError from a filter – and a caller catching either by type must keep doing so.

One unknown id and several read as one sentence and its plural. Both lists are sorted and unknown is deduplicated, so the message does not depend on the order a caller listed its ids in, and it names what would have worked, since a typo is the usual cause.

vcf_header_file_error(resource_id: str, header_filename: str, detail: str) → MalformedResourceError[source]

Refuse a VCF table whose *.header.vcf.gz sidecar is not a header.

The sidecar is read line by line (gain#1406), and two things can be wrong with it: no ## line at all, or a line pysam cannot parse – each phrased by its raise site as detail. What is shared is the ADDRESS, the resource and the file, built here for the reason score_configuration_error() gives.

gain.genomic_resources.resource_implementation module

CONFIG_VALIDATOR_CACHE = <gain.genomic_resources.resource_implementation._ConfigValidatorCache object>: The process-wide resource-config validator cache. See the class.

DEFAULT_STATISTICS_REGION_SIZE = 3000000000

one region per contig (gain#357).

A smaller size balances a single large resource across more workers; keep it at or below about genome / (4 x workers).

Type:: The region size a statistics build uses when its caller names none

class DerivedFilesState(*values)[source]

Bases: Enum

Whether a kind’s derived files describe the resource as it is now.

A derived file is one a kind computes at repair from inputs the statistics hash must not learn about, under a freshness gate of its own; a genomic score keeps its chromosome lengths this way. The gate answers one of three states and never derives anything.

CURRENT = 1: Present and derived from the resource as it is now.

PAYLOAD_ABSENT = 3: Absent or out of date, and an input is a .dvc sidecar whose payload has not been pulled: nothing can be rebuilt here, and there is nothing wrong with the resource.

STALE = 2: Absent or out of date, and every input to rebuild it is here.

class GenomicResourceImplementation(genomic_resource: GenomicResource)[source]

Bases: ABC

Base class of the implementation of a resource type.

A resource is a folder in a repository. An implementation knows how to describe the resource, how to compute its statistics and how to render its HTML pages. A subclass overrides the five abstract methods and, when it reads files, the files property.

Parameters:

genomic_resource (GenomicResource) – The resource that the implementation wraps.

Variables:

resource (GenomicResource) – The wrapped resource.
config (dict) – The configuration read from the resource when the implementation is constructed. A subclass that validates its configuration may replace it with the validated form.

abstractmethod calc_info_hash() → bytes[source]

Compute the digest of the inputs of the info page.

The page is rendered again only when the digest changes.

Returns:: The digest of everything that get_info depends on.
Return type:: bytes

abstractmethod calc_statistics_hash() → bytes[source]

Compute the digest of the inputs of the statistics build.

grr_manage compares the digest with the stored one to decide whether it must compute the statistics again.

Returns:: The digest of everything that the statistics tasks read.
Return type:: bytes

config: dict

abstractmethod create_statistics_build_tasks(*, region_size: int = 3000000000, grr: GenomicResourceRepo | None = None) → list[TaskDesc][source]

Create the task graph tasks that compute the statistics.

grr_manage calls every kind with the same two keyword-only arguments. An implementation declares both, even when it ignores them, and declares no other argument.

Parameters:

region_size (int) – The longest region that one task covers. A kind that splits its build by genomic region reads it. 0 means do not split: a genomic score then scans the resource in one pass and a reference genome uses one region for each chromosome. A kind that does not split ignores it.
grr (GenomicResourceRepo | None) – The repository that holds the resource. None means that the caller names no repository.

Returns:

The tasks of the build. An empty list means that the kind has no statistics to compute.

Return type:

list[TaskDesc]

Raises:

ValueError – If region_size is negative.

property files: set[str]: The names of the resource files that the implementation reads.

get_config() → dict[source]

Return the configuration of the resource.

Returns:: The configuration read when the implementation was constructed. A subclass that validates its configuration returns the validated form.
Return type:: dict

abstractmethod get_info(**kwargs: Any) → str[source]

Render the HTML of the info page of the resource.

Parameters:: **kwargs (Any) – Options of the renderer. The caller defines them.
Returns:: The HTML of the page.
Return type:: str

get_statistics() → ResourceStatistics | None[source]

Load the statistics of the resource.

Returns:: The statistics. None means that the implementation has none: this base class always returns None.
Return type:: ResourceStatistics | None

abstractmethod get_statistics_info(**kwargs: Any) → str[source]

Render the HTML of the statistics page of the resource.

Parameters:: **kwargs (Any) – Options of the renderer. The caller defines them.
Returns:: The HTML of the page.
Return type:: str

reload_statistics() → ResourceStatistics | None[source]

Drop the cached statistics and load them again.

Call it after the statistics were rebuilt on disk.

Returns:: What get_statistics() returns. None means that the implementation has no statistics.
Return type:: ResourceStatistics | None

property resource_id: str: The id of the resource that the implementation wraps.

class InfoImplementationMixin[source]

Bases: object

Mixin that provides generic template info page generation interface.

class FileEntry(name: str, size: str, md5: str | None)[source]

Bases: object

Provides an entry into manifest object.

md5: str | None

name: str

size: str

get_info(**kwargs: Any) → str[source]: Construct the contents of the implementation’s HTML info page.

get_statistics_info(**kwargs: Any) → str[source]: Construct the contents of the implementation’s HTML info page.

get_statistics_template_data() → dict[source]

Return a data dictionary to be used by the statistics template.

Will transform the description in the meta section using markdown.

get_template_data() → dict[source]

Return a data dictionary to be used by the template.

Will transform the description in the meta section using markdown.

resource: GenomicResource

styles_template_name: ClassVar[str] = 'base_implementation_styles.jinja'

template_name: ClassVar[str] = 'base_implementation.jinja'

ONE_REGION_PER_CONTIG = 3000000000

A statistics region size larger than any contig, so every contig is one region.

split_into_regions clamps a region size to the contig length, so any size at least as long as the longest contig builds exactly one region per contig. This is not the 0 sentinel, which means “do not split”: a genomic score then scans the whole resource in one task, with no regions.

class ResourceConfigValidationMixin[source]

Bases: object

Mixin that provides validation of resource configuration.

abstractmethod static get_schema() → dict[source]: Return schema to be used for config validation.

classmethod validate_and_normalize_schema(config: dict, resource: GenomicResource) → dict[source]

Validate the resource schema and return the normalized version.

What comes back is the caller’s own document all the way down, memo hit or not, and detached from the config it was normalized from – so an implementation may keep it and write into it, as GenomicScore.__init__ does.

Do not edit a resource’s config in place. Offering the same config object twice is answered from a memo rather than re-normalized (gain#1059), and an edit made between the two calls is not seen: the second caller gets the document as the config was the first time. Nothing in gain or gpf does this – the implementations that validate all keep the normalized copy rather than the config – and code that wants a config re-read should hand over a new dict, which is always normalized afresh.

class ResourceStatistics(resource_id: str)[source]

Bases: object

Base class for statistics.

Subclasses should be created using mixins defined for each statistic type that the resource contains.

static get_statistics_folder() → str[source]

get_base_resource_schema() → dict[str, Any][source]

get_required_filename_schema() → dict[str, Any][source]

The required filename config-schema fragment.

Spliced into a schema – or a nested block of one – wherever the loader reads filename unconditionally. Not part of get_base_resource_schema(), since not every implementation has a file. Built fresh on every call: cerberus rewrites nested rule dicts in place, on Validator construction and on every validate().

merge_index_columns(resource_id: str, columns: Sequence[str], claimed: Mapping[str, tuple[str, str]]) → dict[str, tuple[str, str]][source]

Return claimed extended with columns, keyed case-insensitively.

The index table has one set of columns for the whole repository – the union of every resource’s fields – so a field name that is fine within one resource can still be unusable next to another resource’s. SQLite compares column names case-insensitively, so assay in one resource and Assay in another are one column asked for twice under two spellings, and a CREATE VIRTUAL TABLE naming both fails – taking the whole repository’s index with it (gain#464). Two resources spelling a field the same way share the column, which is the point of the index.

Raises ValueError if columns cannot join claimed – naming the resource that already holds a spelling, or, past MAX_INDEX_COLUMNS, saying that the repository’s labels no longer fit an FTS5 table. The caller skips that one resource and keeps the rest. claimed is never modified – a rejected resource claims nothing.

validate_index_columns(resource_id: str, columns: Sequence[str]) → None[source]

Check that columns can name the columns of the FTS index.

Every column name is interpolated into the CREATE VIRTUAL TABLE and INSERT statements that build the repository index, so a name that is not a bare identifier – and not one SQL or FTS5 has already taken – is at best unbuildable and at worst an injection (gain#464). Repeats are rejected too: the index keeps one column per name. Where the repeated name is a field of the resource itself, the label silently replaces that field’s value, which then cannot be found by it; where it is a name the index reserves for a field some other implementation contributes, the label lands in a column that means something else for every resource that does contribute it (gain#542). Both are reported against the offending column, the second naming the reserved set, since the curator has no such field of their own to look at.

Raises ValueError naming the resource and the offending column. The caller is the per-resource handler of the index build, so an offending resource is skipped and reported by id instead of taking the whole repository’s index down with it.

The columns are typed as strings but come from a YAML mapping’s keys, which need not be – a column that is not a string is refused like any other bad name rather than raising out of the check.

gain.genomic_resources.resource_query module

The query language that selects resources out of a repository.

A query is an fnmatch glob over the resource id, optionally followed by a bracketed query over the resource’s meta.labels:

hg38/scores/*[phenotype="aut*" and "UCSC" in provenance]

This module owns the grammar, the matching, and what a label value is read as (label_alternatives()) – and nothing else. It answers one question – does this resource match this query – and deliberately holds no policy about what a caller does with the answer: no result cap, no error when a query selects nothing. Those are the annotation layer’s rules about building a pipeline, not the repository’s rules about listing resources.

It lives here rather than in annotation so that the pipeline config, the repositories and the CLIs cannot disagree about what * means.

class LabelClause(key: str, operator: LabelOperator, value: str)[source]

Bases: object

One condition on one label: key = value or value in key.

A clause is data, not a closure, so a caller that evaluates the query somewhere other than in Python – against the FTS index, say – can read what was asked without reimplementing what it means. Whatever engine runs the search, matches() stays the only definition of the comparison.

key: str

matches(label: str) → bool[source]: Check whether a rendered label value satisfies this clause.

matches_an_absent_label() → bool[source]

Check whether this clause holds for a label that is not there.

An absent label is matched as "", and under this grammar a clause that holds for "" holds for every string: a value must be at least one character, so in can never accept "", and the only = values fnmatch accepts "" for are globs of * alone. A caller with no other way to evaluate the clause can therefore drop it outright.

The converse says nothing. A clause that fails here still holds for whichever resources carry the key, so a false answer is not a licence to settle the clause for all of them.

matches_in(labels: Mapping[str, Any]) → bool[source]

Check whether labels satisfies this clause.

meta.labels is a free-form YAML mapping, so a label value is whatever YAML made of it – perturbed: False is a bool and year: 2019 an int, both of which the production GRRs carry in bulk. The query language only ever spells values as text, so a value is compared in its rendered form; without that both in and = raise a bare TypeError out of the predicate. A label the resource does not carry is matched as "". A list value is a set of alternatives: the clause holds if it holds for any one element, each rendered exactly as a scalar is (see label_alternatives()).

These rules live here rather than at the call sites so that a caller evaluating one clause reads a label exactly as the caller evaluating all of them does.

operator: LabelOperator

value: str

class LabelOperator(*values)[source]

Bases: Enum

The comparisons a label clause can make.

CONTAINS = 'contains'

EQUALS = 'equals'

class ResourceQuery(resource_id_pattern: str, label_clauses: tuple[LabelClause, ...])[source]

Bases: object

A parsed resource query: an id glob plus label clauses.

label_clauses: tuple[LabelClause, ...]

match(resource: GenomicResource) → bool[source]: Check whether resource matches the query.

match_id(resource_id: str) → bool[source]: Check whether resource_id matches the query’s id glob.

match_labels(labels: Mapping[str, Any]) → bool[source]

Check whether labels satisfies every one of the query’s clauses.

How one label value is read – its rendered form, a list as alternatives, absence as "" – is LabelClause.matches_in()’s to say, and is said there. The FTS index cannot represent the difference between an absent label and an empty one – it stores "" for every label column a resource does not carry – so treating absence as a distinct case would put this matcher permanently out of step with the same query evaluated in SQL.

static parse(query: str) → ResourceQuery[source]

Parse query into a matcher.

Raises ResourceQueryParseError if the query is not well-formed, or if it is longer than MAX_RESOURCE_QUERY_LENGTH.

resource_id_pattern: str

exception ResourceQueryParseError[source]

Bases: ValueError

Raised when a resource query cannot be parsed.

label_alternatives(value: Any) → tuple[str, ...][source]

Render a meta.labels value as the strings it stands for.

A label value is whatever YAML made of it. A scalar – a string, or the bool and int the production GRRs carry in bulk – stands for its str(); a nested mapping does too. A list or tuple is a set of alternatives, one rendered string per element, so a resource can be labelled with everything it is (modality: [RNA, ATAC]). An empty list stands for "", which is also what an absent label reads as.

Every reader of a label value renders it through here.

gain.genomic_resources.resource_types module

GAIn’s config vocabulary: which spellings are accepted, and which mean the same thing.

Mostly resource type: values, plus the annotator names that were retired alongside one of them.

Two different relations live here, and they are not the same. fragment_score and cnv_collection are equivalent – either resolves to the same thing, one is merely deprecated. np_score is retired: it is no longer accepted at all, and it was never equivalent to its replacement, since it carried a different default read mode. The first relation is served by equivalent_resource_types(), the second by reject_retired_resource() and, for the annotator names, retired_annotator_message().

Deliberately dependency-free and low in the import graph. The equivalence below is needed by repository (which applies the type predicate in SQL), by genomic_scores and its implementations, by annotation_config and by the web API – and genomic_scores imports repository, so a home in the score layer could not be reached from the layer that needs it most. That is not a detail: the review of gain#471 found that the SQL-side predicate had been missed precisely because the helper was out of reach.

See docs/adr/0003-fragment-score-vocabulary.md, superseded by docs/adr/0011-deprecate-cnv-collection-vocabulary.md.

ALLELE_MULTIPLICITY_ENFORCEMENT_RELEASE = '2027.1.0': The GAIn release in which an allele score holding several rows for one allele key without declaring allele_multiplicity: many stops being a warning and becomes an error (gain#1749; the allele-plane design, A11). Ask allele_multiplicity_enforced() whether it has arrived rather than comparing versions at the call site. Named in every such warning: a notice that does not say when it bites cannot be scheduled against.

FRAGMENT_SCORE_TYPES = ('fragment_score', 'cnv_collection')

The resource type: values that name a fragment score.

Two spellings. fragment_score is what a resource should declare, and what the public GRR declares since the migration; cnv_collection is what a repository that has not migrated declares. It is deprecated and stops being accepted in LEGACY_VOCABULARY_REMOVAL_RELEASE; consuming it warns, at the places that open a resource rather than here.

A tuple rather than a set: it is used for membership, but also rendered into user-facing messages and into SQL placeholders, and a set would order them arbitrarily. Preferred spelling first, so a message reads as a recommendation.

GENE_SCORE_TYPE = 'gene_score': The resource type: for a gene score. One spelling, as above.

GENE_SET_TYPES = ('gene_set_collection', 'gene_set')

The resource type: values that name a gene set collection.

An ordered pair for the same reasons as FRAGMENT_SCORE_TYPES above, shared by the collection that opens the resource and the annotator that declares what it accepts (gain#1329).

Deliberately not an equivalence group. Unlike the fragment score’s pair, this one is absent from equivalent_resource_types below, so a search or type filter for gene_set_collection does not answer a repository’s gene_set resources. That is the behaviour as it stands, not a considered position: making the pair searchable would change what the repository predicate, the resources endpoint and the editor return, which is a decision of its own rather than a consequence of naming the pair here.

LEGACY_ANNOTATOR_NAMES = {'cnv_collection': 'fragment_score', 'cnv_collection_annotator': 'fragment_score_annotator'}

Annotator names still accepted but deprecated, mapped to what to write instead.

The annotator half of the fragment-score vocabulary deprecation (ADR 0011, gain#538): both spellings are registered entry-point keys, so a pipeline naming either builds, and only the value is worth typing in a config written today. They stop being accepted in LEGACY_VOCABULARY_REMOVAL_RELEASE.

Here for the same reason RETIRED_ANNOTATOR_NAMES below is: two seams on opposite sides of an import edge need it – the fragment-score annotator warns on one, and annotation_config keeps them out of the names it advertises while parsing, and the former imports the latter. Deriving the set from a resource-type spelling instead would make a naming coincidence load-bearing: these are annotator names, and nothing obliges a deprecated one to be spelled like the type it reads.

LEGACY_FRAGMENT_SCORE_TYPE = 'cnv_collection': The deprecated resource type: for a fragment score. Still accepted, and still declared by repositories that have not migrated.

LEGACY_GENE_SET_TYPE = 'gene_set': The deprecated resource type: that also names one. Still accepted; opening one warns.

LEGACY_VOCABULARY_REMOVAL_RELEASE = '2027.1.0': The GAIn release in which every legacy fragment-score spelling stops being accepted (gain#539). Named in every deprecation warning: a notice that does not say when it bites cannot be scheduled against.

PREFERRED_ALLELE_SCORE_TYPE = 'allele_score': The resource type: for an allele score.

PREFERRED_FRAGMENT_SCORE_TYPE = 'fragment_score': The preferred resource type: for a fragment score.

PREFERRED_GENE_SET_TYPE = 'gene_set_collection': The resource type: for a gene set collection.

PREFERRED_POSITION_SCORE_TYPE = 'position_score': The resource type: for a position score. One spelling, so there is no pair below; named here because this module owns the vocabulary and its sibling kinds are named here too.

RETIRED_ALLELE_SCORE_TYPE = 'np_score'

The retired resource type: that used to name an allele score.

Deprecated since 2024-11 and removed in RETIRED_VOCABULARY_REMOVAL_RELEASE (gain#920). Unlike the fragment score’s legacy spelling above this one is no longer accepted, so it survives here only to be recognised and refused with a message that names the replacement.

RETIRED_ANNOTATOR_NAMES = {'np_score': 'allele_score', 'np_score_annotator': 'allele_score_annotator'}

Annotator names GAIn no longer accepts, mapped to what to write instead.

The annotator half of the same retirement (gain#919): these named the allele-score annotator in a pipeline’s YAML, as np_score named its resource type. Each maps to the replacement of the same shape, so a migration is a one-word edit.

Here rather than in the annotation package because two seams need it and they are on opposite sides of an import edge: annotation_factory turns the name into a factory, and annotation_config turns it into a resource set while parsing – and the former imports the latter.

RETIRED_VOCABULARY_REMOVAL_RELEASE = '2026.8.5': The GAIn release that removed np_score (gain#781, announced in gain#918). Named in the refusal so a reader who meets it in an old environment can tell which upgrade changed under them.

class RetirableResource(*args, **kwargs)[source]

Bases: Protocol

The little of a resource reject_retired_resource() needs.

A structural type rather than GenomicResource itself: this module is deliberately dependency-free and low in the import graph (see the module docstring), and repository imports it, so naming the class here would close a cycle.

get_full_id() → str[source]: Return the resource’s id, with version where it has one.

get_type() → str[source]: Return the resource’s declared type:.

allele_multiplicity_enforced(version: str | None = None) → bool[source]

Whether an undeclared duplicate allele is an error rather than a warning.

True when version – the installed gain.__version__ when omitted – is ALLELE_MULTIPLICITY_ENFORCEMENT_RELEASE or later, by PEP 440 ordering. A dev or pre-release of that release is earlier, and so is the 0.0.0.dev0 an editable checkout without a generated version reports. A version that does not parse answers False rather than raising: a malformed version string must not turn a warning into an error.

The one place that question is answered: a reader that warns or refuses by it asks here, and a test that needs either answer patches this function rather than the installed version.

deprecated_spelling_message(surface: str, legacy: str, preferred: str, *, found_in: str) → str[source]

Return the warning text for one use of one legacy spelling.

surface names the kind of configuration the spelling was written as ("resource type", "annotator name", "parameter"), found_in names where it was written – a resource id, or an annotator within a pipeline. Both are required because the stack at the point of the warning points into GAIn’s own config parsing rather than at the YAML the reader has to edit, so the message must carry the location itself.

A plain string rather than a logging call: the module that recognised the spelling logs it, so the record carries that module’s logger name.

equivalent_resource_types(resource_type: str) → tuple[str, ...][source]

Return every type: value denoting the same kind of resource.

Only a fragment score is treated as having more than one spelling here; every other type maps to itself, so a caller can filter by the result unconditionally without special-casing.

A gene set collection also has two spellings (GENE_SET_TYPES) and is deliberately NOT one of them – see that constant. So this is the set of equivalences that SEARCH honours, which is narrower than the set of types some annotator will open.

Exists because filtering resources by an exact type string went wrong the moment a second spelling appeared: asking for fragment_score matched nothing at all in a repository whose resources declare cnv_collection. An empty result is indistinguishable from “this repository has none of those”, so the failure is silent – a wrong answer rather than an error.

reject_retired_resource(resource: RetirableResource) → None[source]

Raise if resource declares a spelling GAIn has removed.

Called from each seam that turns a type: string into something: the score factory (build_score_from_resource), AlleleScore itself, the implementation builder (what grr_manage sweeps with), and the annotation pipeline’s resource resolver. Four call sites rather than one because there is no single seam they all pass through – ADR 0011 established the same for the fragment-score warning – and each is reachable without the others: the pipeline’s own type check would otherwise pre-empt this message with a generic one, and the implementation builder never constructs a score at all.

Not pushed down into GenomicResource itself, which would be the only common ancestor: that is also the enumeration and display path (grr_manage list, the web API’s resource-types endpoint, the repository’s SQL type predicate), and refusing there would abort a whole run over a repository that merely contains a retired resource. That is the failure ADR 0011 records as the reason its predecessor expired.

Raising here rather than letting the entry-point lookup fail is the whole point. Deleting the registration already makes an np_score resource fail, but it fails as unsupported resource implementation type <np_score> – which tells a holder that GAIn does not know the type, not that GAIn removed it and what to write instead.

Named by full id, matching the fragment-score notice: a repository may hold several versions of one resource id, each its own directory with its own config to migrate, and the bare id would name none of them precisely. Rendered only on the failure path – this runs on every resource open, including the statistics scan’s per-region rebuilds.

require_fragment_score_type(resource_type: str) → str[source]

Return resource_type, or raise if it names no fragment score.

Lives here rather than at its one call site (the test-data builder’s with_resource_type) because builders.py sits four lines under pylint’s 1500-line module ceiling, and because the rule it enforces is this module’s to state.

reset_deprecation_notices() → None[source]

Forget what this process has already announced.

Exists for tests: the announced-set is process-wide, and a test that asserts a warning fired must not depend on whether an earlier test in the same worker already consumed it. core/tests/conftest.py calls this before every test.

retired_annotator_message(annotator_type: str) → str[source]

Return the refusal text for one use of a retired annotator name.

Silent about allele_score_mode, unlike its resource-type sibling above: the mode was only ever derived from the resource, and an annotator name is refused before any resource is opened, so a pipeline naming both retired spellings is told about the annotator first and about the resource – with its mode advice – once that is fixed.

retired_resource_type_message(*, found_in: str) → str[source]

Return the refusal text for one use of the retired np_score.

found_in names where the type was written – a resource id – for the same reason the deprecation messages above carry it: the stack at the point of recognition runs through GAIn’s own config parsing, not through the YAML the reader has to edit.

The mode sentence is not padding. np_score is the one retired spelling that was never a pure alias: AlleleScore used to read the default mode off the resource type, so np_score meant substitutions while allele_score means alleles. A holder who swaps only the type string gets a resource that loads and reads differently, which is a silent wrong answer rather than an error – so the replacement and the mode key have to arrive together or the message causes the bug it is warning about.

warn_deprecated_spelling(logger: Logger, surface: str, legacy: str, preferred: str, *, found_in: str) → None[source]

Announce one legacy spelling once per offender, per process.

Announced once per process; see _announce_once.

warn_retired_config_key(logger: Logger, key: str, *, found_in: str, reason: str) → None[source]

Announce a config key that is accepted but ignored, once per offender.

The counterpart of warn_deprecated_spelling() for a key with no replacement: there is nothing to write instead, so the notice says why the key does nothing and when it stops being accepted. reason completes “it is ignored: …”. Announced once per process; see _announce_once.

warn_undeclared_allele_multiplicity(logger: Logger, *, found_in: str) → None[source]

Announce an undeclared duplicate allele once per resource.

found_in is the full id of an allele score holding several rows for one allele key without declaring allele_multiplicity: many. The message names the resource and the remedy, never the allele: it is the deduplication key, so naming the allele would announce every duplicate rather than every offending resource. Announced once per process; see _announce_once.

gain.genomic_resources.score_def module

What a genomic score DEFINITION is, how it reads a value, how it is built.

The bottom of the score layer: a score’s definition, the vocabulary it parses with (value types, NA sentinels, column addressing), the read that turns a record’s cell into a value, and the lifecycle that builds a resource’s definitions out of its scores: block. Nothing here knows about a GenomicScore – which is what lets vcf_scores sit above this module and genomic_scores above both, with no cycle between them.

Split out of genomic_scores so that the VCF-specific score code could be gathered into one module: vcf_scores constructs GenomicScoreDef at runtime, and genomic_scores imports what vcf_scores builds, so the two could not both live in one file without importing each other.

The lifecycle joined it in gain#1044, from private methods on GenomicScore. Only one of them was polymorphic, and only through the class attribute DEFAULT_AGGREGATORS, so each is a function parametrized by what it used to read off self. The one piece that stayed behind is the dispatch over the table’s payload_kind (GenomicScore._build_scoredefs): it calls into vcf_scores and bigwig_scores, both of which import this module, so hosting it here would close a cycle.

The other half of a definition’s story is told elsewhere, for that same reason: genomic_scores.value_extraction (gain#1114) sits above this module and holds both decisions taken at open. So what a definition IS and how it parses a cell are here; how one is ADDRESSED and which read reaches it are there.

BULK_PARSEABLE_VALUE_TYPES = ('float', 'int', 'str'): Value types GenomicScoreDef.parse_array() defines a column parse for, and so the ones a bulk column read can serve. bool is absent because no column consumer asks for it.

Bases: ScoreDef

The definition of one score of a genomic score resource.

Each entry of the scores: block of a resource becomes one definition. The definition knows the id, the value type and the description of a score, how to turn a raw cell into a value, and which histogram configuration belongs to it. Read the definitions from GenomicScore.score_definitions.

Variables:

score_id (str) – The id of the score.
value_type (str) – The type of the values: float, int, str or bool.
desc (str) – The description of the score.
small_values_desc (str | None) – The description of the small values.
large_values_desc (str | None) – The description of the large values.
hist_conf (HistogramConfig | None) – The histogram configuration.
aggregator (str | None) – The aggregator that reduces several values for one annotatable to one. It is None until the resource type fills its default.
number_mismatch_warned (bool) – Internal. True after a VCF INFO field with a wrong value count was reported once.
empty_element_warned (bool) – Internal. True after an empty element of a multi-valued VCF INFO field was reported once.

aggregator: str | None

col_index: int | None

col_name: str | None

empty_element_warned: bool = False

na_values: Any

number_mismatch_warned: bool = False

parse_array(cells: ndarray) → ndarray[source]

Turn a whole column of raw cells into values, vectorized.

The result equals [parse_value(c) for c in cells], with the non-value shown in a form that the array can hold. The result has one of two shapes, chosen by value_type. A float or int score gives a float64 array with nan for no value. A str score gives an object array with None for no value.

Parameters:: cells (np.ndarray) – The raw cells of one column.
Returns:: The parsed values, one for each cell. The array of an empty column is empty.
Return type:: np.ndarray
Raises:: TypeError – If value_type is not float, int or str. Ask GenomicScore.supports_region_value_arrays first.

Turn one raw cell into the value of this score.

Parameters:: value (str | int | float | None) – The raw cell.
Returns:: The parsed value. None if the cell is None (for example an absent VCF INFO key), if it is an NA sentinel of the score, or if it does not parse. A cell that does not parse is logged and skipped, so one bad cell does not abort a scan.
Return type:: ScoreValue

score_index: int

value_parser: Any

SCORE_TYPE_PARSERS: dict[str, Callable[[Any], Any]] = {'bool': <function parse_bool>, 'float': <class 'float'>, 'int': <class 'int'>, 'str': <class 'str'>}: How each value type turns a raw cell into a value. Annotated rather than inferred: three entries are builtin TYPES and bool’s is a function, so an inferred value type joins to object, which is not callable as far as a type checker is concerned.

build_genomic_score_schema() → dict[str, Any][source]

Declare the config every genomic score kind accepts.

The kinds’ get_schema overrides deep-copy this and splice in their aggregator, so it must build a fresh dict per call.

extract_column_value(record: tuple[Any, ...], score_def: GenomicScoreDef) → str | int | float | bool | None[source]

Read one score off a record whose PAYLOAD is a raw row.

The tabular backends and bigWig: a score is a CELL of the payload, addressed by the integer column value_extraction.resolve_score_indices resolved into score_index. Turning that cell into a value is the definition’s job (GenomicScoreDef.parse_value()), so that this read and the bulk column read cannot drift apart.

A pure function of (record, score_def) – it holds no state and needs none, which is what let the per-line score-line objects go. There is no “score_index not resolved yet” check: the attribute does not exist until open() sets it, so an unopened def raises AttributeError naming it.

finish_scoredefs(score_defs: dict[str, GenomicScoreDef], default_aggregators: dict[str, str | None]) → dict[str, GenomicScoreDef][source]

Fill in what a definition cannot decide for itself.

The value type. type: is optional, and an unstated one is recorded as float, with the float value parser alongside it – the two are resolved together, here, because they were left unresolved together by parse_scoredef_config (gain#1221). Recording the type matters rather than leaving it None: GenomicScoreDef.__post_init__ returns early on a None type, which would skip na_values normalization and leave the raw config string in place. That turns the NA check into a SUBSTRING test, so a score configured na_values: "-1" would read a real value of 1 as a null – the exact defect normalize_na_values prevents.

It is resolved HERE rather than at parse time because for a VCF score an unstated type means “the type the file’s header declares”, and defaulting before the merge would override a declared int with float. Filling it afterwards leaves that inheritance intact and still leaves no definition without a type.

The aggregator.

The default depends on how the score is reduced, which is fixed by the resource type – mean over a region of positions, max over the alleles at one, join(,) rather than list for a fragment score’s strings – so it cannot be decided from a definition alone. default_aggregators is that decision, made by the caller: each score class passes its own DEFAULT_AGGREGATORS. Resolving it here rather than in GenomicScoreDef.__post_init__ is what lets one field replace the pos_aggregator/allele_aggregator pair.

Applied at the convergence point of all three construction routes – the scores: block, a VCF header, a bigWig – because a default applied in only one of them is the same bug in a new place: a VCF-derived def would arrive with aggregator=None, and the fragment score annotator drops an attribute whose aggregator is None silently.

normalize_na_values(na_values: Any, value_type: str) → set[Any][source]

Normalize a configured na_values into a type-aware sentinel set.

The resource schema permits na_values as a bare scalar (na_values: "-1") or a list. A bare str left un-normalized turns the NA membership test in _extract_vcf_value() into a SUBSTRING test ("1" in "-1" is True) and raises TypeError when matched against a non-string raw payload (bigWig floats). This wraps a scalar into a one-element collection and returns a set that carries, for every configured sentinel, both its text form (matched against string backends) and – for numeric score types – its parsed form (matched against a float/int raw payload). So a sentinel is matched against whichever representation the incoming raw value presents, never by substring.

na_values of None selects the per-value-type default set verbatim: the defaults are non-numeric tokens ("", "nan", ".", "NA") that a numeric backend never presents as a raw value, so they are left as a pure-text set – coercing them would only add a spurious parsed nan and change the default behaviour.

A set input is treated as ALREADY normalized and returned as a copy without re-coercion, so normalization is idempotent (a fixed point). This is what the VCF scores-block merge path relies on: it rebuilds a GenomicScoreDef from an already-normalized na_values set, whose __post_init__ re-runs this function – a second coercion pass would otherwise grow the set (e.g. parsing the default "nan" text token into a float('nan')) and silently change the statistics hash. Config-supplied na_values never arrive as a set (the schema permits only None, str or list), so a set can only be a prior normalization result.

parse_bool(value: Any) → bool[source]

Read a bool score’s raw value: its TEXT, against a closed set.

A value that is ALREADY a bool is returned unchanged. That is the idempotence every parser in SCORE_TYPE_PARSERS owes its caller – float(1.5) and int(3) give it for free, and a VCF score reaches the parser with a value pysam has already decoded – so it is the shared contract rather than a special case for flags.

Anything else raises ValueError, including a bare 0/1 number: a number is not a bool, and the only way one arrives is a resource declaring type: bool over a numeric field. GenomicScoreDef.parse_value() logs the refusal and reads the cell as a non-value, so one bad cell does not abort a scan.

Why the vocabulary is closed, why bool alone declares no NA sentinels, and what that costs: docs/adr/0024-a-bool-score-reads-its-cells-text.

parse_scoredef_config(config: dict[str, Any]) → dict[str, GenomicScoreDef][source]: Parse ScoreDef configuration.

refuse_unbuildable_aggregators(score_defs: dict[str, GenomicScoreDef], resource_id: str) → dict[str, GenomicScoreDef][source]

Refuse a configured aggregator: no accumulator can be built from.

The config schema admits a parametrized aggregator by name and a parenthesis (join(...)), which is looser than the parser: a join(a)(b) passes validation and fails only when an accumulator is built from it. That is a fact about the definition, so it is refused at construction through score_configuration_error() – which says why a definition’s refusal must not wait for a read.

Called after finish_scoredefs(), so the aggregator judged is the one the score will reduce by, a class default included. The shipped defaults all build, so in practice what is refused is a config-supplied spelling; the wording assumes as much, since a default that did not build would be the class’s defect, not the yaml’s. Building once per definition is cheap: Aggregator.build remembers what a spelling resolves to.

validate_scoredefs(config: dict[str, Any], table: GenomicPositionTable, resource: GenomicResource) → None[source]

Check each configured score’s column address against the table’s header.

For a TABULAR table only: a bigWig’s scores are checked by validate_bigwig_scoredefs and a VCF’s have no column address at all (each reads the INFO field named by its id, and parse_vcf_scoredefs refuses an address that says otherwise), so GenomicScore.open() does not send either here. It runs at open because the header is only known then.

A score is refused – through score_configuration_error(), naming the resource and the score, never by assert (a resource config is data, and python -O strips an assert) – when it names a column of a table that has no header (header_mode: none is the one way a tabular table is still headerless once open), names one the header lacks, indexes past the header, or states no address at all. A headerless table’s index addresses are not checked here: there is no header to bound them by.

Also rewrites the legacy name:/index: spellings into column_name:/column_index: IN the config it is given – which is why it takes the config rather than reading one, and why the caller must hand it the same dict the score keeps.

gain.genomic_resources.score_implementation module

The implementation plane shared by gene scores and genomic scores.

ScoreImplementationBase sits one layer above ScoreResource (the catalogue plane in gain.genomic_resources.score_resource): where that base owns what a score is, this base owns what a score implementation does that means the same for both families – contributing score_ids / score_descriptions into the FTS index. Serialising-and-plotting a computed histogram into the resource is the module-level save_and_plot_histograms(), which both families’ statistics builds call.

It deliberately keeps create_statistics_build_tasks abstract: a gene score emits a single task that scans a DataFrame, whereas a genomic score emits a region-split DAG with a min/max merge stage. These are genuinely different strategies for genuinely different data shapes, so the base does not try to unify them.

The location mirrors score_resource for the same reason: gene_scores already depends on genomic_resources, so living here adds no new dependency edge, whereas a top-level gain/scores/ package would create a cycle.

class ScoreImplementationBase(genomic_resource: GenomicResource)[source]

Bases: GenomicResourceImplementation, InfoImplementationMixin

Shared implementation base for gene and genomic score resources.

A concrete subclass must set self.score (a ScoreResource) in its own __init__; from it this base reads the score definitions for the search index.

abstractmethod create_statistics_build_tasks(*, region_size: int = 3000000000, grr: GenomicResourceRepo | None = None) → list[TaskDesc][source]

Create tasks for calculating resource statistics for task graph.

Kept abstract: gene and genomic scores build statistics with genuinely different task shapes (a single DataFrame scan versus a region-split DAG), so each family provides its own.

score: ScoreResource

save_and_plot_histograms(resource: GenomicResource, score: ScoreResource, histograms: dict[str, NullHistogram | CategoricalHistogram | NumberHistogram]) → None[source]

Serialise each histogram into the resource and render its PNG.

Every score in score.score_definitions leaves with exactly the files this build produced. A file an earlier build wrote that the current histogram no longer justifies – the truncated sidecar of a histogram now within the limit, the image of a NullHistogram, all three of a score absent from histograms – is deleted rather than left to be served as current. So is every histogram file of a score id score_definitions no longer holds, the gzipped full histogram (histogram_<id>.json.gz) included.

A categorical histogram past UNIQUE_VALUES_LIMIT is written as deterministic gzipped JSON next to its plain truncated sidecar; every other histogram as plain JSON (ADR 0032). The full histogram an earlier build left in the other encoding is deleted, so neither a release that reads only the plain encoding nor a DVC-tracked sidecar that cannot be dropped serves it as current – except a legacy histogram_<id>.yaml, which the sidecar is named after and so must stay for the sidecar to be found. A DVC-tracked .json.gz beside a DVC-tracked sidecar is kept too, and still loaded, until the curator runs dvc remove. The encoding, and so which file is stale, is decided from the histogram in hand.

A written .json.gz with no .dvc pointer beside it is reported with a warning that names the dvc add to run; gain never runs DVC.

Every deletion goes through drop_stale_histogram_file(): a DVC-tracked file is reported and left in place.

gain.genomic_resources.score_resource module

The catalogue plane shared by gene scores and genomic scores.

ScoreResource is the one base both families extend. It owns only the things that mean the same for a gene score and a genomic score: the score definitions, the two ways to enumerate/look them up, and the histogram accessors. It deliberately owns nothing about a resource’s lifecycle (open/ close), its table, its fetch surface, its chromosomes or its aggregators – those belong to genomic scores alone, which are keyed by position and read over a region, whereas gene scores are keyed by gene symbol, have no open/close and nothing to aggregate.

The boundary is not incidental; it is the whole reason this module exists. See docs/2026-07-14-gain-score-abstraction.html for the design, and tests/small/genomic_resources/test_score_resource_api.py for the guard that keeps a lifecycle/table/fetch/aggregator method from being lifted into this base “because both subclasses happen to have one”.

The location is deliberate too: gene_scores already imports genomic_resources.{histogram,repository,resource_implementation}, so living here adds zero new dependency edges. A top-level gain/scores/ package would instead create a genomic_resources <-> scores import cycle.

Bases: object

Catalogue-plane fields common to a gene score and a genomic score.

Column addressing is intentionally not here: it is a loading detail that differs per family (a VCF keys a score by INFO name, a bigWig has no header, a gene score renames a pandas column), so it lives on the concrete GenomicScoreDef / GeneScoreDef subclasses instead.

desc: str

hist_conf: NullHistogramConfig | CategoricalHistogramConfig | NumberHistogramConfig | None

large_values_desc: str | None

score_id: str

small_values_desc: str | None

value_type: str

class ScoreResource(resource: GenomicResource)[source]

Bases: ResourceConfigValidationMixin, Generic

Shared catalogue base for gene and genomic score resources.

Parameterised by the concrete score-definition type so that get_score_definition returns the right kind for each family (a GenomicScoreDef for genomic scores, a GeneScoreDef for gene scores) without either side having to override it.

Constructing one keeps the resource and validates its config against the family’s get_schema(), so config is the normalized document. A concrete subclass calls super().__init__(resource) and then sets score_definitions – the score_id -> definition mapping – itself.

Everything a subclass may add on top of this (a table, an open/close lifecycle, fetch methods, aggregators) is its own concern and must NOT be lifted here – see the module docstring and the API-surface guard test.

HISTOGRAM_COUNT_UNIT: ClassVar[str]: The noun for what a histogram’s n counts – the weighting the statistics scan gives each record of this family: base pairs for a position score (record_weight), alleles, fragments, genes. Every concrete family states its own; the info pages read it.

config: dict[str, Any]

get_all_scores() → list[str][source]

get_gzipped_histogram_filename(score_id: str) → str[source]

Return the gzipped full-histogram filename for a score.

A categorical histogram past UNIQUE_VALUES_LIMIT is stored under this name, next to its plain truncated sidecar (ADR 0032).

get_histogram_config(score_id: str | None = None) → NullHistogramConfig | CategoricalHistogramConfig | NumberHistogramConfig[source]

The histogram config the score’s DEFINITION resolves to.

The stated histogram: block when there is one, else the default for the score’s value type – a NullHistogramConfig for a type with no default histogram. So a NullHistogramConfig here means the histogram is annulled by design, decided from the config alone: no statistics are read, and a checkout whose statistics were never built answers the same as the published one.

Parameters:: score_id (str or None) – The id of the score. None, the default, names the only score: the one score of a resource that defines exactly one. A resource with several scores, or none, has no only score, and the call raises ValueError with the ids it does define.

get_histogram_filename(score_id: str) → str[source]

Return the filename the full histogram of a score is read from.

The truncated sidecar decides the encoding: when the manifest lists the sidecar and the gzipped full histogram, that is the file; otherwise it is the plain one (see get_plain_histogram_filename()).

get_histogram_image_filename(score_id: str) → str[source]

get_histogram_image_public_url(score_id: str | None = None) → str | None[source]

Return the histogram image URL on the resource’s public mirror.

Unlike get_histogram_image_url(), this is built from the resource’s public URL so it is reachable from a browser even when the GRR is a local directory repository. None under exactly the conditions of the other.

Parameters:: score_id (str or None) – The id of the score. None, the default, names the only score: the one score of a resource that defines exactly one. A resource with several scores, or none, has no only score, and the call raises ValueError with the ids it does define.

get_histogram_image_url(score_id: str | None = None) → str | None[source]

Return the histogram image URL on the repository’s own url.

None when the score’s histogram is annulled by definition (see get_histogram_config(); a stale image the manifest still lists does not revive it), or when the resource’s manifest does not list the image – a configured histogram is a plan to draw, and the statistics build can decline it, or not have run. A resource with no stored manifest is addressed by its definition alone; the manifest is never built here.

Parameters:: score_id (str or None) – The id of the score. None, the default, names the only score: the one score of a resource that defines exactly one. A resource with several scores, or none, has no only score, and the call raises ValueError with the ids it does define.

get_plain_histogram_filename(score_id: str) → str[source]

Return the uncompressed full-histogram filename for a score.

statistics/histogram_<id>.json, or the legacy .yaml when the manifest lists one. This is the name every histogram within UNIQUE_VALUES_LIMIT is stored under; its truncated sidecar is derived from it.

get_score_definition(score_id: str) → ScoreDefT | None[source]

get_score_histogram(score_id: str | None = None, *, truncated: bool = False) → NullHistogram | CategoricalHistogram | NumberHistogram[source]

Return defined histogram for a score.

A score may declare a categorical (or null) histogram just as readily as a numeric one, so the honest return type is the full Histogram union. Callers that need numeric-only attributes (bars/bins/min_value/…) must narrow with isinstance(hist, NumberHistogram) first.

With truncated=True a truncated sidecar is acceptable: when one exists (categorical histograms past UNIQUE_VALUES_LIMIT have one), it is returned instead of the full histogram, so the caller never reads the full values file. Without a sidecar the full histogram is returned either way.

A histogram annulled by the score’s definition (see get_histogram_config()) is answered from the definition alone, with its configured reason: no statistics file is written for it (gain#305), and a stale one is not consulted (gain#1604).

Parameters:

score_id (str or None) – The id of the score to read. None, the default, reads the only score: the one score of a resource that defines exactly one. A resource with several scores, or none, has no only score, and the call raises ValueError with the ids it does define.
truncated (bool) – Accept the truncated sidecar instead of the full histogram.

get_score_range(score_id: str | None = None) → tuple[float, float] | None[source]

Return the value range for a numeric score.

Parameters:: score_id (str or None) – The id of the score. None, the default, names the only score: the one score of a resource that defines exactly one. A resource with several scores, or none, has no only score, and the call raises ValueError with the ids it does define.

get_truncated_histogram_filename(score_id: str) → str[source]

Return the truncated-sidecar filename for a score.

Derived from the plain histogram name whichever encoding the full histogram is stored in, so the sidecar stays where every gain release looks for it.

static histogram_schema() → dict[str, Any][source]

The histogram config-schema fragment shared by both families.

Contributed into each family’s get_schema() instead of pasted into both – the two blocks used to be byte-identical modulo line-wrapping. Built fresh on every call so a caller that mutates the returned schema (e.g. a copy.deepcopy then in-place edit) cannot affect another.

resource: GenomicResource

score_definitions: dict[str, ScoreDefT]

refuse_unfoldable_histograms(score_defs: dict[str, ScoreDefT], resource_id: str) → dict[str, ScoreDefT][source]

Refuse a configured NUMBER histogram no value of the score can feed.

A histogram: {type: number} over a score whose value type is not one a number histogram accumulates (NUMBER_HISTOGRAM_VALUE_TYPES) is a config stating something the score cannot do, so gain#1336 raises on it rather than working around it, naming the resource and the score.

It is a fact about a score DEFINITION – the value_type and hist_conf every ScoreDef carries – so it lives here, on the base both families share, and each family calls it once at its own construction point (gain#1308): the caller decides when (after the value type is final, before anything reads the column), and says why there. Raising at CONSTRUCTION rather than where the statistics build reads the configs is what makes the refusal reach every consumer: annotation never unpacks score definitions.

Two things it does not refuse. A definition with hist_conf=None is skipped – the default chosen for its type later never pairs a non-numeric type with a number histogram. And a CATEGORICAL histogram over a number is deliberately let through: it folds one value at a time and nullifies just that score, which is a fact about a value rather than about the config.

gain.genomic_resources.utils module

build_chrom_mapping(resource: GenomicResource | None, config: dict[str, Any] | None = None) → Callable[[str], str | None] | None[source]

Build chromosome mapping function from resource config.

The resource config may contain chrom_mapping section with filename, add_prefix and del_prefix keys. The filename points to a file with two columns: original chromosome names and mapped names.

These keys are mutually exclusive, only one of them may be present.

Parameters:: resource – genomic resource with config
Returns:: function that maps chromosome names or None if no mapping is defined

read_resource_id_label(resource: GenomicResource, label: str) → str | None[source]

The named label’s value, when that label names another resource.

Read through the accessor that narrows both meta levels, the way every other label reader does (gain#654, gain#1004) – and then narrowed once more, because that accessor promises a mapping and says nothing about what is in it (gain#1050). A value that cannot be a resource id reads as absent and is reported. An absent label, and the explicit YAML null the production GRRs carry, are not curator mistakes and stay silent.

Taken by label name rather than hard-wired to reference_genome, because four labels across three modules name a resource this way: reference_genome on gene models (gain#1050) and on scores, and source_genome/target_genome on a liftover chain. All four read a free-form YAML value into a str | None, and unnarrowed all four died the same way – the int in a regex, the list and the dict wherever the id was first hashed – with a TypeError that named neither the resource nor the label (gain#1053).

Lives here rather than beside any one of its callers, and beside build_chrom_mapping(), which is the same kind of thing: a read of a resource’s own configuration that several resource types share.

Whitespace is deliberately NOT stripped: only an empty value is narrowed away, so a padded id or the trailing newline a folded scalar leaves still reaches resolution and fails there naming itself. Stripping would be a normalization policy for ids rather than a narrowing, and that is a separate decision.

gain.genomic_resources.variant_utils module

maximally_extend_variant(chrom: str, pos: int, ref: str, alts: list[str], genome: ReferenceGenome) → tuple[str, int, str, list[str]][source]: Maximally extend a variant.

normalize_variant(chrom: str, pos: int, ref: str, alts: list[str], genome: ReferenceGenome) → tuple[str, int, str, list[str]][source]

Normalize a variant.

Using algorithm defined in the https://genome.sph.umich.edu/wiki/Variant_Normalization

gain.genomic_resources.vcf_scores module

Reading a VCF’s INFO fields as genomic scores.

Everything the score layer knows about VCF, in one module: how a VCF header’s INFO metadata becomes score definitions, and how one of those scores is read off a record. Both encode the same thing – INFO field semantics, and Number=1/A/R/. in particular – and they used to sit 362 lines apart in genomic_scores.

The VCF table itself is not here and does not belong here. genomic_position_table.table_vcf produces records: it owns the payload’s shape, the pysam proxies it carries and the constants that name them. This module interprets those records as scores. That is the same seam the record contract draws everywhere else – a backend yields records, the score layer says what they mean – and it is why the table layer still imports nothing from the score layer.

extract_vcf_value(record: tuple[Any, ...], score_def: GenomicScoreDef) → str | int | float | bool | None[source]

Read one score off a VCF record: an INFO field, not a column.

VCF is the awkward backend and this function is where the whole of its awkwardness lives. A VCF score is addressed by INFO name – which is col_name, the string the config gave – looked up on the variant, typed by the header metadata, and for a per-allele field selected by the record’s allele index.

The five cases a TUPLE value falls into (a scalar value – the common shape, a well-formed Number=1 or a Flag – skips them all):

Number=A – one value per ALT allele: select this record’s allele. A record whose ALT is absent (‘.’) has no allele index and so no applicable value – under the VCF spec such a record has zero ALT alleles, so a Number=A field on it carries zero values and a row that supplies one anyway is malformed. It yields None, a null score, however many values the field carries and whatever the score def’s declared type (#256). Returning the null HERE also keeps the raw tuple from escaping as a score value. The check is a crash guard too: without it the tuple is indexed with None and the read dies with TypeError.
Number=R – one value per allele including the reference, which occupies offset 0: an ALT allele reads at allele_index + 1, and a record with no ALT reads the reference value at offset 0.
Number=1 or Number=0 – a declared scalar that arrived as a tuple (SC=2.5,3.5 under Number=1; htslib does not enforce Number on read). Refused: the row reads None and is reported by _check_number_arity() (gain#1257). It is decided on the tuple as pysam hands it over, ahead of the empty-element drop, so SS=a, is an over-arity row and not a one-value row with an empty element.
Number=. and Type=String – an unbounded string field, joined on ‘|’ into a single value (a VCF-local convention).
anything else – handed to parse_value, which joins it on ‘|’ through the converter parse_vcf_scoredefs installed.

An EMPTY element contributes nothing, whatever the shape (#630). ORIGIN=1, declares a value it does not carry, and pysam decodes that element as None. Joining one was the operation whose outcome depended on nothing but the field’s declared Type – "|".join raised TypeError here and took the whole fetch, while the converter’s map(str, ...) silently annotated the text '3|None' for every other type – so the non-per-allele shapes drop their empty elements through _drop_empty_elements() before either join sees one, and a tuple left with nothing reads null rather than the '' a join of nothing produces. The drop is done HERE, once, for both joins: this is the layer that still has the record, so it is the only one that can name the row it reports, and it leaves the converter a tuple that cannot contain a None.

The per-allele shapes need no drop – an empty element they select is already the null they give an allele with no value of its own – but they REPORT it, because the row is malformed either way and the arity check cannot see it: ALT=T,G S=d1, carries one value per allele and trips nothing, while its second allele reads null for want of a value the row said it had. Which fields can say that of a None at all is _reports_empty_elements().

Neither per-allele selection trusts the tuple to be the right length (#289). Nothing rejects a row whose value count does not match its ALT column – not pysam, not resource load – and indexing one bare aborted the whole fetch: the second allele of ALT=T,G S=d11 ran off the end with IndexError, and the try in parse_value that turns a bad cell into a logged null is deliberately not around this call (it guards the PARSE, and #256’s crash-guard test depends on that placement), so the crash escaped get_score and took the scan with it. An allele past the end of the tuple therefore reads None – the same null #256 gives the ALT-less record, by the same rule: no applicable per-ALT value, no score. The mismatch itself is reported by _check_number_arity(), which also catches the mirror shape (more values than alleles, the extras unreadable), once per table.

A key the header declares but this record does not carry yields None rather than raising: info.get returns None, None is not a tuple, so the number cases are skipped and parse_value turns it into a null score. For a key the header does NOT declare, pysam’s info.get raises ValueError: Invalid header – but nothing in this tree can ask for one, since a VCF table’s score defs are built FROM the header and a configured score naming an undeclared field is rejected when the score is opened (pinned by test_vcf_check_for_missing_score_columns).

The metadata lookup stays inside the tuple branch. INFO_META.get builds a fresh pysam VariantMetadata for the key, per score, per record. A Number=1 field decodes to a scalar, never reaches that branch, and must not pay for a metadata object it will never read; that is the common shape of a score-bearing INFO field, and hoisting the lookup out of it took a 50-score read of a 3000-row VCF from 26.65 to 19.83us/line. (Pinned by test_vcf_reads_the_info_metadata_only_for_a_tuple_value.)

The two pysam proxies this reads – INFO and INFO_META – are resolved once per record by the VCF backend and carried in the payload, because pysam allocates a fresh proxy on every variant.info access. See table_vcf for that measurement and why they live there.

parse_vcf_scoredefs(vcf_header_info: dict[str, Any] | None, config_scoredefs: dict[str, GenomicScoreDef] | None, *, resource_id: str, merge: bool = False) → dict[str, GenomicScoreDef][source]

Build score definitions from a VCF header’s INFO metadata.

Every INFO field the header declares becomes a score, typed through VCF_TYPE_CONVERSION_MAP and described by the header’s own description. This is why a VCF resource needs no scores: block to be usable: the file documents its own scores.

value_parser is set to None for Number of 1, A or R, because pysam already decodes those to a scalar (or to a tuple that extract_vcf_value() indexes by allele). Every other shape keeps converter, which joins a tuple on ‘|’ – the VCF-local convention for a field whose arity the header does not fix.

converter joins with str – a Number=./Type=Integer field therefore reads as text, which is now what such a field DECLARES as well (gain#1259; it used to declare the header’s Type= and was the one place the definition and the value disagreed) – and which is what made it the SILENT half of #630: an empty element would render as the four-character string 'None'. It is not guarded here. The tuples that reach it have already had their empty elements dropped by extract_vcf_value(), the only route to it and the only layer holding the record a report has to name; this parser sees a value, not a row.

config_scoredefs is what the resource’s own scores: block declared, and overrides the header for the fields it names: description, aggregators and NA values all take the config’s value when it gives one, falling back to the header’s. The value TYPE and the value PARSER are overridable only together, and only for a field whose header declares a scalar Number (_SCALAR_VALUED_NUMBERS). An entry that leaves type: unstated takes neither, so it reads exactly what the header-only resource reads (gain#1221). A field the header declares MULTI-VALUED takes neither either, whatever type: says: it keeps the header’s converter, because that converter IS the field’s |-join and a value type cannot describe a tuple (gain#1233), and it keeps the str that join produces, because a type is not merely descriptive – it selects the histogram, and a joined field declaring int aborted its own statistics build in np.isnan (gain#1259). An entry stating such a type is REFUSED by _refuse_overridden_type() – it contradicts the header, and gain#1336 raises rather than discarding it. Column addressing is NOT overridable – a VCF score is its INFO key, so col_name/col_index always come from the header side – which is why an entry whose id the header does not declare is a contradiction too, REFUSED by _refuse_undeclared_id() before anything else is asked of it (gain#1489), and why an entry that states an address OTHER than its id – a differing column_name:, or any column_index: – is one as well, REFUSED by _refuse_overridden_address() (gain#1498); an address equal to the id is redundant and passes. The fourth refusal, _refuse_genotype_arity(), is the header’s own claim rather than the config’s (gain#1258). The order is id, arity, address, type: the arity check runs BEFORE the two config-side ones because a per-genotype field is unreadable whatever the entry states and neither of their advised edits can change that, and the address check precedes the type check because a wrong address is wrong whatever type is stated. resource_id is threaded in for those four messages alone.

merge decides what happens to header fields the config does not mention: False (the default) returns only the configured scores, so the config acts as a filter; True keeps the rest as the header defined them. It is the resource’s merge_vcf_scores setting.

Module contents

get_resource_implementation_builder(resource_type: str) → Callable[[GenomicResource], GenomicResourceImplementation] | None[source]

Return the builder of the implementation of a resource type.

The lookup checks the in-process registry first. It then falls back to the entry points of the group gain.genomic_resources.implementations and caches the builder that it loads.

Parameters:: resource_type (str) – The resource type, the type field of genomic_resource.yaml.
Returns:: A callable that takes a resource and returns its implementation. None means that nothing registered the type.
Return type:: Callable[[GenomicResource], GenomicResourceImplementation] | None

register_implementation(resource_type: str, builder: Callable[[GenomicResource], GenomicResourceImplementation]) → None[source]

Register the builder of a resource type in the process registry.

The registration lasts for the life of the process and replaces an earlier builder of the same type.

Parameters:

resource_type (str) – The resource type, the type field of genomic_resource.yaml.
builder (Callable[[GenomicResource], GenomicResourceImplementation]) – A callable that takes a resource and returns a ready to use implementation. An implementation class whose constructor takes only the resource is its own builder.

gain.genomic_resources package

Subpackages

Submodules

gain.genomic_resources.aggregators module

gain.genomic_resources.allele_classification module

gain.genomic_resources.ann_data_10x module

gain.genomic_resources.ann_data_resource module

gain.genomic_resources.bigwig_scores module

gain.genomic_resources.cached_repository module

gain.genomic_resources.cli module

gain.genomic_resources.cli_cache_repo module

gain.genomic_resources.cli_dvc module

gain.genomic_resources.cli_errors module

gain.genomic_resources.cli_list module

gain.genomic_resources.data_frame_resource module

gain.genomic_resources.draw_score_histograms module

gain.genomic_resources.dvc module

gain.genomic_resources.fsspec_protocol module

gain.genomic_resources.genomic_context module

gain.genomic_resources.genomic_context_base module

Key Constants

gain.genomic_resources.genomic_context_cli module

gain.genomic_resources.group_repository module

gain.genomic_resources.histogram module

gain.genomic_resources.liftover_chain module

gain.genomic_resources.memo module

gain.genomic_resources.reference_genome module

gain.genomic_resources.repository module

gain.genomic_resources.repository_factory module

gain.genomic_resources.resource_errors module

gain.genomic_resources.resource_implementation module

gain.genomic_resources.resource_query module

gain.genomic_resources.resource_types module

gain.genomic_resources.score_def module

gain.genomic_resources.score_filter module

gain.genomic_resources.score_implementation module

gain.genomic_resources.score_resource module

gain.genomic_resources.utils module

gain.genomic_resources.variant_utils module

gain.genomic_resources.vcf_scores module

Module contents