gain.genomic_resources package

Subpackages

Submodules

gain.genomic_resources.aggregators module

Score aggregator classes and factory utilities.

class gain.genomic_resources.aggregators.Aggregator[source]

Bases: ABC

Base class for score aggregators.

An accumulator is not shared. An aggregator is mutable state, and every caller in gain builds a fresh one per fold: the folding reads build one per query per call, and the annotators that reduce their own values build one per attribute per call. Nothing outlives the fold it was built for, and build() is cheap enough for that to be the default – a name resolves through a memo (gain#1157).

aggregate() still clears its state first, so an instance CAN be reused single-threaded. It is not thread-safe either way: two threads folding through one accumulator would interleave their values.

add(value: Any, count: int = 1) → None[source]

Add a value to the aggregator, weighted by count.

count is the number of times the value is deemed to occur – the number of base pairs a position-score record spans, for instance. GenomicScore.record_weight is where each kind states its own rule. The weight is applied in closed form: adding a value with a weight of n produces the same result as adding it n times, without doing n units of work, which is what makes folding a region proportional to its records rather than to its length in base pairs. The one exception is mean, which is more accurate weighted than replicated: it rounds once per record rather than once per base. See _add_internal().

aggregate(values: list[Any] | None) → Any[source]: Clear state, add all values, and return the final result.

static build(source: AggregatorDefinition | str | dict[str, Any]) → Aggregator[source]

Build a FRESH aggregator from a definition, string, or dict.

A string is the hot spelling: every aggregating read builds its accumulators anew per call, so the same few names arrive here millions of times per run, and parsing one was ~70% of building it (gain#1157). A name is therefore resolved through _class_and_parameters(), which remembers what a string resolves TO – a class and its parameters, nothing mutable – and never the accumulator built from it.

clear() → None[source]: Reset the aggregator to its initial state.

default_parameter: ClassVar[str | None] = None

get_final() → Any[source]: Return the aggregated result.

get_total_count() → int[source]: Return the total weight seen, None values included.

get_used_count() → int[source]

Return the total weight of the non-None values added.

A weighted total, not a number of records: it is the denominator of the mean, so a value added with a weight of n contributes n to it.

output_value_type: ClassVar[str | None] = None

parametrized: ClassVar[bool] = False

classmethod preserves_domain(*, value_type: str | None = None) → bool[source]: Return True if output stays within the source value domain.

static resolve_class(source: AggregatorDefinition | str | dict[str, Any]) → type[Aggregator][source]

The aggregator CLASS a definition, string, or dict names.

For the callers that want what an aggregator WOULD answer rather than an accumulator to answer it with: output_value_type and preserves_domain() are both class-level, so an attribute that only knows an aggregator’s name can describe its output without building one (gain#1133).

It resolves through the same _resolve() as build(), for every spelling and not just the memoised one, so the two cannot disagree about what a source names and a source refused there is refused here, in the same words.

class gain.genomic_resources.aggregators.AggregatorDefinition(aggregator_type: str, parameters: list[Any] = <factory>)[source]

Bases: object

Parsed representation of an aggregator type string.

aggregator_type: str

as_dict() → dict[str, Any][source]: Serialize to a dictionary.

classmethod coerce(source: AggregatorDefinition | str | dict[str, Any]) → AggregatorDefinition[source]

Whichever of the three spellings arrived, as a definition.

An aggregator reaches this module written three ways – a name, a {aggregator_type, parameters} mapping, or an already parsed definition – and every consumer wants the last of those. The cascade that gets there is stated once here so a fourth spelling, or a fix to the parsing rules, is one edit rather than a hunt for the copies. (Aggregator.build() takes the string arm through a memo of its own, but that memo parses with from_string() too – a parsing fix is still one edit.)

classmethod from_dict(data: dict[str, Any]) → AggregatorDefinition[source]: Construct an aggregator definition from a dictionary.

classmethod from_string(raw: str) → AggregatorDefinition[source]

Parse an aggregator definition from a string.

Format: name or name(parameter).

parameters: list[Any]

class gain.genomic_resources.aggregators.BoolAggregator[source]

Bases: Aggregator

Aggregator that returns True if any non-None value was added.

get_final() → bool[source]: Return the aggregated result.

output_value_type: ClassVar[str | None] = 'bool'

class gain.genomic_resources.aggregators.ConcatAggregator[source]

Bases: Aggregator

Aggregator that concatenates all passed values.

One of the three aggregators whose output is genuinely proportional to the aggregated weight (see also join and list). The weight is kept run-length encoded during the scan and expanded only in get_final().

get_final() → Any[source]: Return the aggregated result.

output_value_type: ClassVar[str | None] = 'str'

values: list[tuple[str, int]]

class gain.genomic_resources.aggregators.CountAggregator[source]

Bases: Aggregator

Aggregator that counts values.

get_final() → Any[source]: Return the aggregated result.

output_value_type: ClassVar[str | None] = 'int'

class gain.genomic_resources.aggregators.CounterAggregator[source]

Bases: Aggregator

Aggregator that counts values.

counter: Counter

get_final() → Any[source]: Return the aggregated result.

output_value_type: ClassVar[str | None] = 'object'

class gain.genomic_resources.aggregators.JoinAggregator(separator: str)[source]

Bases: Aggregator

Aggregator that joins all passed values using a separator.

default_parameter: ClassVar[str | None] = ','

get_final() → Any[source]: Return the aggregated result.

output_value_type: ClassVar[str | None] = 'str'

parametrized: ClassVar[bool] = True

values: list[tuple[str, int]]

class gain.genomic_resources.aggregators.ListAggregator[source]

Bases: Aggregator

Aggregator that builds a list of all passed values.

get_final() → Any[source]: Return the aggregated result.

output_value_type: ClassVar[str | None] = 'list'

values: list[tuple[Any, int]]

class gain.genomic_resources.aggregators.MaxAggregator[source]

Bases: Aggregator

Maximum value aggregator for genomic scores.

get_final() → Any[source]: Return the aggregated result.

output_value_type: ClassVar[str | None] = 'float'

classmethod preserves_domain(*, value_type: str | None = None) → bool[source]: Return True if output stays within the source value domain.

class gain.genomic_resources.aggregators.MeanAggregator[source]

Bases: Aggregator

Aggregator for genomic scores that calculates mean value.

get_final() → Any[source]: Return the aggregated result.

output_value_type: ClassVar[str | None] = 'float'

classmethod preserves_domain(*, value_type: str | None = None) → bool[source]: Return True if output stays within the source value domain.

class gain.genomic_resources.aggregators.MedianAggregator[source]

Bases: Aggregator

Aggregator for genomic scores that calculates median value.

get_final() → Any[source]: Return the aggregated result.

output_value_type: ClassVar[str | None] = 'float'

classmethod preserves_domain(*, value_type: str | None = None) → bool[source]: Return True if output stays within the source value domain.

values: list[tuple[Any, int]]

class gain.genomic_resources.aggregators.MinAggregator[source]

Bases: Aggregator

Minimum value aggregator for genomic scores.

get_final() → Any[source]: Return the aggregated result.

output_value_type: ClassVar[str | None] = 'float'

classmethod preserves_domain(*, value_type: str | None = None) → bool[source]: Return True if output stays within the source value domain.

class gain.genomic_resources.aggregators.ModeAggregator[source]

Bases: Aggregator

Aggregator for genomic scores that calculates mode value.

get_final() → Any[source]: Return the aggregated result.

classmethod preserves_domain(*, value_type: str | None = None) → bool[source]: Return True if output stays within the source value domain.

value_counts: dict[Any, int]

class gain.genomic_resources.aggregators.PositionScoreAggregationQuery(score: str, aggregator: str | None = None, none_value_replacement: ScoreValue | None = None)[source]

Bases: object

The same request over a position score’s expansion (gain#727).

Asks ScoreAggregationQuery’s two questions, plus the one part of a position score’s request that no other kind can ask. none_value_replacement substitutes for every null the per-position expansion holds – uncovered and covered-but-NA alike – before the aggregator sees it; unset, nulls stay inert for every aggregator, all of which already skip None.

A position score answers with a value at every position of the queried region, so a position no record covers is still a position, and a caller may need it to count as a zero rather than go missing. A kind whose records are either in the result or not – a fragment, an allele – has no such position to speak for, and only the covered-but-NA half of the field would ever apply to it. That is why it lives here and not on the neutral query.

A sibling of ScoreAggregationQuery, not a subclass – see that class for why (gain#1302). The two shared fields are repeated in its order, so the three land as the flat dataclass had them (score, aggregator, none_value_replacement) and every positional call site keeps its meaning.

aggregator: str | None = None

none_value_replacement: ScoreValue | None = None

score: str

class gain.genomic_resources.aggregators.ScoreAggregationQuery(score: str, aggregator: str | None = None)[source]

Bases: object

One score’s reduction request, in the terms every kind shares.

Names a score and how to reduce it, and nothing else; aggregator of None resolves to the score’s own default from its definition. Kind-neutral because nothing about “reduce this score with this aggregator” depends on how a kind lays its records out, so a position score, a fragment score and an allele score all ask the same thing here.

It deliberately carries no none_value_replacement. That field speaks for a locus NO record covers, which only a kind that reads a value at every position of a region even has – see PositionScoreAggregationQuery, which carries it. Keeping it off the neutral query is what makes the neutral query kind-neutral at all.

PositionScoreAggregationQuery is a SIBLING of this class, not a subclass, though it asks these same two questions and repeats these two fields (gain#1302). A subclass would substitute for this one everywhere – every Sequence[ScoreAggregationQuery] would accept a position query – and the surfaces typed that way answer (score_id, aggregator) PAIRS, which have nowhere to put a replacement: the field would drop, invisibly, because the pair that comes back is exactly the one a caller who never asked for a replacement would get. gain#1121 made it a subclass for what the hierarchy states, and gain#1158 then had to close that door at runtime. As a sibling, mypy closes it: a position query in a list[ScoreAggregationQuery] is a type error, and so is handing a list[PositionScoreAggregationQuery] to a neutral resolver.

aggregator: str | None = None

score: str

gain.genomic_resources.aggregators.aggregator_name(aggregator: AggregatorDefinition | str | dict[str, Any]) → str[source]

The canonical string spelling of an aggregator, whatever its form.

An annotation pipeline may write an attribute’s aggregator as a name, as a {aggregator_type, parameters} mapping, or as an already parsed AggregatorDefinition; a resource may only write the name. Everything downstream of the config – a ScoreDef’s field, a ScoreAggregationQuery’s – holds the name alone, so the spellings collapse here, on the way in.

A name is returned as it stands rather than parsed and printed again. The round trip is exact for every registered aggregator and for the parametrized forms (pinned by test_the_three_aggregator_spellings_collapse_to_one_name), so this is not about the answer differing – it is that a caller holding a malformed name should meet the complaint where its aggregator is BUILT, naming the score, rather than here while a config is being serialised.

gain.genomic_resources.aggregators.get_aggregator_class(aggregator: str) → type[Aggregator][source]: Return the aggregator class for the given aggregator name.

gain.genomic_resources.aggregators.validate_aggregator(aggregator: AggregatorDefinition | str | dict[str, Any], value_type: str | None = None) → None[source]: Raise ValueError for invalid aggregator or value type combinations.

gain.genomic_resources.allele_classification module

What an allele-score row’s ref/alt pair is, as one of five classes.

The classes and the rule that assigns them are vocabulary, defined once in CONTEXT.md and decided in ADR 0020; classify_allele() states the rule and nothing else here restates it. The classification is what the allele-score statistics count, and it is deliberately independent of them: no accumulator, no storage, no scan – just the rule.

gain.genomic_resources.allele_classification.ALLELE_BASES = 'ACGT': The bases an allele may be written with. Anything else – N, a symbolic allele such as <DEL>, the missing-allele * – means the pair does not parse as an allele and is counted as other.

class gain.genomic_resources.allele_classification.AlleleClass(*values)[source]

Bases: Enum

The class of an allele-score row’s ref/alt pair (ADR 0020).

COMPLEX = 'complex'

DELETION = 'deletion'

INSERTION = 'insertion'

OTHER = 'other'

SUBSTITUTION = 'substitution'

class gain.genomic_resources.allele_classification.AlleleClassification(allele_class: AlleleClass, ref_length: int | None, alt_length: int | None)[source]

Bases: object

A classified ref/alt pair.

allele_class: AlleleClass

alt_length: int | None

property length_change: int | None: Bases the alternative adds over the reference.

ref_length: int | None: Lengths of the two alleles, absent for AlleleClass.OTHER – those strings are not alleles, so they have no allele length.

gain.genomic_resources.allele_classification.classify_allele(ref: str | None, alt: str | None) → AlleleClassification[source]

Classify a ref/alt pair as written, VCF-anchored (ADR 0020).

Both alleles are upper-cased first, so a soft-masked lowercase base classifies as the base it masks rather than missing the anchor and inflating complex. What is still not an allele afterwards – N, a symbolic allele, an empty string – is other, and the remaining rules apply in the order the ADR states them:

substitution – strictly one base to one base, the identity pair included;
insertion – anchored: a single reference base that the alternative starts with, adding length_change bases;
deletion – the mirror image, removing -length_change bases;
complex – everything else, MNVs and unanchored indels alike, carrying both lengths.

Total over rows, not merely over strings: a table need not configure a ref or an alt column, and a VCF ALT of . yields a record with no alternative at all, so either allele may arrive as None. A row missing an allele is other – it is still a row, and the class counts of a resource’s rows always sum to its row count. Never raises.

gain.genomic_resources.ann_data_10x module

gain’s readers for the two 10x Genomics formats.

The Matrix Market triple and the 10x-Genomics HDF5, both built on anndata + pandas + h5py + scipy, which gain already depends on. See docs/adr/0014-gain-owns-the-10x-readers.md in the repository for why the work is here rather than delegated to scanpy, and for the parameter surface these readers define.

class gain.genomic_resources.ann_data_10x.TenXH5Parameters(gex_only: bool = False, genome: str | None = None)[source]

Bases: object

The knobs a 10x_h5 resource may set, already validated.

Fewer than the triple’s, because the h5 answers for itself what the triple needs telling: it names its own features, so there is no var_names choice to make and nothing to make unique.

genome: str | None = None

gex_only: bool = False

class gain.genomic_resources.ann_data_10x.TenXMtxParameters(var_names: str = 'gene_symbols', make_unique: bool = True, gex_only: bool = False)[source]

Bases: object

The knobs a 10x_mtx resource may set, already validated.

gex_only: bool = False

make_unique: bool = True

var_names: str = 'gene_symbols'

gain.genomic_resources.ann_data_10x.parse_10x_h5_parameters(parameters: Mapping[str, Any], resource_id: str) → TenXH5Parameters[source]: Validate a 10x_h5 resource’s parameters: block.

gain.genomic_resources.ann_data_10x.parse_10x_mtx_parameters(parameters: Mapping[str, Any], resource_id: str) → TenXMtxParameters[source]

Validate a resource’s parameters: block into the knobs gain has.

An unrecognised key raises rather than being forwarded, so a typo is reported instead of silently doing nothing – and so is a key that used to reach scanpy and now has no meaning here.

gain.genomic_resources.ann_data_10x.read_10x_h5(file_path: str, *, resource_id: str, parameters: TenXH5Parameters | None = None, matrix_free: bool = False) → AnnData[source]

Read a 10x-Genomics HDF5 into an AnnData.

resource_id names the resource in diagnostics only.

With matrix_free, X is an all-zero matrix of the declared shape and IS NOT THE RESOURCE’S DATA. Everything else – both axis tables, the shape, the genome and feature-type filters – is built by the same code as an ordinary read, so anything derived from those is identical. It exists for the statistics build, which reads neither X nor anything computed from it.

gain.genomic_resources.ann_data_10x.read_10x_mtx(matrix_path: str, barcodes_path: str, features_path: str, *, resource_id: str, legacy: bool = False, parameters: TenXMtxParameters | None = None, matrix_free: bool = False) → AnnData[source]

Read a 10x matrix-market triple into an AnnData.

The three members are named outright rather than assembled from a directory and a prefix: which names they carry is a question about the resource’s layout, and the resource is what answers it.

10x writes features as rows and barcodes as columns, so the matrix is transposed into the cells x genes an AnnData carries. legacy marks the CellRanger v2 feature table, which has no feature-type column – and therefore nothing for gex_only to filter on.

resource_id names the resource in diagnostics only.

With matrix_free, X is an all-zero matrix of the declared shape and IS NOT THE RESOURCE’S DATA. Everything else – both axis tables, the shape, the feature-type filter – is built by the same code as an ordinary read, so anything derived from those is identical. It exists for the statistics build, which reads neither X nor anything computed from it.

gain.genomic_resources.ann_data_resource module

Loading helpers for ann_data genomic resources.

class gain.genomic_resources.ann_data_resource.TenXMtxLayout(barcodes: str, features: str, legacy: bool)[source]

Bases: object

The three members of a 10x matrix-market triple, by resource name.

Which names the triple carries is a question about the resource, and resolving it once here is what keeps the read, the statistics inputs and the cache prefetch from each answering it differently.

barcodes: str

features: str

legacy: bool

property sidecars: set[str]: Return the two non-matrix members.

gain.genomic_resources.ann_data_resource.is_10x_matrix_name(file_name: str) → bool[source]: Return whether file_name names a 10x matrix-market member.

gain.genomic_resources.ann_data_resource.load_ann_data_from_resource(resource: GenomicResource | None, *, matrix_free: bool = False) → AnnData[source]

Load an AnnData from an ann_data genomic resource.

The caller owns the file handle. An h5ad is read backed="r" – the default, and what keeps a multi-gigabyte X out of a dask worker – which leaves an open h5py file behind for as long as the AnnData lives. A repo sweep that loads one per resource and relies on a garbage collection that may never come is gain#480’s shape, so a caller that loads in a loop closes with ann_data.file.close() when ann_data.isbacked. A 10x read is in memory and has no handle.

matrix_free asks for a read that does not materialise the data matrix. The resulting ``X`` IS NOT THE RESOURCE’S DATA – it is an all-zero matrix of the right shape. Everything else is built by the same code as an ordinary read, so both axis tables, the shape and the feature-type filter are identical, which is what lets the statistics build use it: AnnData._gen_repr skips X and describe reads obs and var. Reading the real matrix costs about 10 GB on the largest 10x resource to write 221 bytes of statistics.

It is honoured only where it means something. Both 10x formats honour it; h5ad ignores it – backed="r" already keeps X off the heap, and reproducing anndata’s repr for an arbitrary h5ad would mean reimplementing its reader for no memory benefit (ADR 0014).

gain.genomic_resources.ann_data_resource.load_ann_data_from_resource_id(resource_id: str, grr: GenomicResourceRepo | None = None) → AnnData[source]: Load an ann_data from a genomic resource id.

gain.genomic_resources.ann_data_resource.resolve_10x_layout(manifest: Manifest, file_name: str) → TenXMtxLayout[source]

Return the layout of the 10x matrix file_name, per the manifest.

Resolution is manifest-driven for the same reason the tabix index is: the manifest is already loaded and is protocol-agnostic, whereas probing costs a network round trip per candidate on http and s3.

This and resolve_10x_layout_for_read() are the pair, and they differ only in where the answer comes from – exactly as resolve_tabix_index_filename() and its _for_read twin do. A caller that only wants the two names takes .sidecars.

gain.genomic_resources.ann_data_resource.resolve_10x_layout_for_read(resource: GenomicResource, file_name: str) → TenXMtxLayout[source]

Return the layout to read file_name with, never building.

Shaped like resolve_tabix_index_filename_for_read(), and for the same reason: a read must stay a pure read, and GenomicResource.get_manifest() would build – md5-scanning the whole resource and writing state files – for a resource that carries no .MANIFEST (gain#430). With no manifest at hand, falls back to probing the resource itself for the same two markers.

gain.genomic_resources.ann_data_resource.resolve_ann_data_format(config: Mapping[str, Any]) → str[source]

Return the format an ann_data config is read as.

A declared format: wins; otherwise the file: suffix decides, and a name matching none of them falls back to h5ad. A config with no file: gets the fallback too – the missing key is the loader’s error to report, and this is also reached from the statistics hash, which degrades rather than raising.

The loader and the statistics hash both resolve through here so that the format a resource is read as and the format its hash records cannot disagree. They did: the hash used to state the h5ad fallback outright, so an explicit format: h5ad added to a 10x config changed the read without changing the hash, and the statistics never rebuilt.

gain.genomic_resources.bigwig_scores module

Reading a bigWig’s single value column as a genomic score.

Everything the score layer knows about bigWig, in one module: how a bigWig’s score definitions are finished off, what a bigWig resource is allowed to configure, and how the one value is read off a record. Symmetric to vcf_scores, and for the same reason – these are all statements about one backend’s peculiarities, and they belong together rather than scattered through genomic_scores.

Reject what corrupts values; warn about what is merely inert. That is the principle that decides which of the two halves below a piece of misconfiguration falls into, and it is worth stating because bigWig attracts a lot of config that does nothing. A bigWig is a binary format with a fixed layout: one numeric value per interval, no columns, no header, no text. So a resource that declares a second score, or a score type: of int, or a column index: other than the deprecated 3, is asking for a value the file cannot give – it would be truncated, or read from a column that does not exist – and validate_bigwig_scoredefs() refuses to open it, naming the resource and the score. A resource that declares chrom:/pos_begin:/pos_end: column blocks, or a header:, is merely describing a tabular file that this one is not; nothing reads those keys, no value changes because of them, and genomic_position_table.utils warns and ignores them.

The bigWig table itself is not here and does not belong here. genomic_position_table.table_bigwig produces records: it owns the payload’s shape – which, since this module exists, is the bare value. This module says what that value means as a score. That is the same seam vcf_scores draws, and it is why the table layer still imports nothing from the score layer.

gain.genomic_resources.bigwig_scores.build_bigwig_scoredefs(config: dict[str, Any], config_scoredefs: dict[str, GenomicScoreDef]) → dict[str, GenomicScoreDef][source]

Finish a bigWig resource’s score definitions.

Currently a pass-through, and kept as the seam rather than deleted: it is where GenomicScore._build_scoredefs routes a bigWig, matching the VCF branch beside it, so bigWig-specific definition work has an obvious home.

It used to empty the default ``na_values``, and that is worth recording because the reasoning was right and the decision still wrong. A float score defaults to the sentinels ("", "nan", ".", "NA") – four TEXT tokens, which exist because a tabular backend hands the score layer strings. A bigWig hands it a float, which can never equal any of them, so on this backend the set is dead config.

Emptying it changed nothing at runtime but everything to GenomicScoreImplementation.calc_statistics_hash, which folds na_values in verbatim: every one of the 150 deployed bigWig resources would have gone stale and been rescanned – ~74.7 G records, hours of compute – to arrive at byte-identical statistics. Verified against a deployed resource’s stored stats_hash: na_values was the ONLY field that differed.

The runtime saving it was reaching for is kept, and taken from the data instead: value_extraction.select_value_extractor binds the identity read unless the NA set holds a sentinel a float could actually match. A text-only set – which is every unconfigured bigWig score – takes the identity path either way.

gain.genomic_resources.bigwig_scores.extract_bigwig_value(record: tuple[Any, ...], score_def: GenomicScoreDef) → str | int | float | bool | None[source]

Read the one score off a bigWig record: the payload IS the value.

A true identity function, and that is the whole point. A bigWig stores a float per interval and pyBigWig hands it up as a Python float; a bigWig score is declared type: float (anything else is refused at open); and the NA default for such a score is empty (see build_bigwig_scoredefs()). So every step GenomicScoreDef.parse_value() would perform on this value – the NA membership test, the float(value) reparse – is provably a no-op, and this reads the value out of the record and returns it.

score_def is unused and stays in the signature because ValueExtractor is what GenomicScore.open routes to; a bigWig-shaped extractor with a different arity would need a call-site branch per record to invoke, which is the cost this removes.

The variant that does consult the definition is extract_bigwig_value_na(), and open() picks between the two once, from whether the score configures any NA sentinels at all.

gain.genomic_resources.bigwig_scores.extract_bigwig_value_na(record: tuple[Any, ...], score_def: GenomicScoreDef) → str | int | float | bool | None[source]

extract_bigwig_value(), plus the configured NA-sentinel check.

Bound instead of the identity for a score whose na_values is non-empty – which, since the default is empty for bigWig, means a resource that explicitly configured sentinels (na_values: "-1" is the deployed shape). The sentinel set carries both the text and the parsed form of each sentinel (see normalize_na_values), so a numeric payload matches by value.

Still no parse: a sentinel match yields the null score, and anything else is the payload unchanged. The choice between this and the identity is made once per open, never per record.

gain.genomic_resources.bigwig_scores.validate_bigwig_scoredefs(resource_id: str, score_defs: dict[str, GenomicScoreDef]) → None[source]

Refuse a bigWig score config that cannot mean what it says.

Called from GenomicScore.open before the table is opened, in the same slot as the extractor routing: a refusal that costs no file handle cannot leak one, and both of this function’s inputs are known at construction, so nothing here needs the handle.

Deliberately NOT called from __init__. GenomicScoreImplementation.__init__ builds its score eagerly, so a constructor that refused would make a misconfigured bigWig resource impossible to list or describe – and listing and describing it is precisely what grr_manage has to do in order to report it. Refusing at open leaves the resource inspectable and stops only the read.

What is refused, and why each one corrupts values rather than merely sitting there:

more than one score – a bigWig file carries one value per interval. A second score has no second value to read, so both scores would read the same number under different names.
a ``type:`` other than ``float`` – pyBigWig hands up a Python float, and the value read is an identity (see extract_bigwig_value()). int would silently truncate every value; str/bool would let a float through wearing the wrong declared type, and the declared type is what the aggregators, the histograms and the bulk read path all branch on.
a column address other than the deprecated 3 – there are no columns to address. Under the old four-element payload, 0/1/2 read the contig and the two coordinates and >= 4 was out of range; either way a resource that names one is asking for something that is not the score.
a column NAME – a bigWig has no header, so there is nothing to resolve the name against.

index: 3 is the one exception, accepted with a deprecation notice at DEBUG – all 150 deployed bigWig resources carry it, so anything louder fires for every one of them on every open: see DEPRECATED_VALUE_INDEX.

A score that declares no type: at all is let through. That is not a declaration of a non-float type – it is the same absent-type config every backend accepts, and it corrupts nothing: the value still arrives as the float the file holds.

gain.genomic_resources.cached_repository module

Provides caching genomic resources.

class gain.genomic_resources.cached_repository.CacheResource(resource: GenomicResource, protocol: CachingProtocol)[source]

Bases: GenomicResource

Represents resources stored in cache.

class gain.genomic_resources.cached_repository.CachingProtocol(remote_protocol: ReadOnlyRepositoryProtocol, local_protocol: FsspecReadWriteProtocol, public_url: str | None = None)[source]

Bases: ReadOnlyRepositoryProtocol

Defines caching GRR repository protocol.

classify_cached_resource_file(resource: GenomicResource, filename: str) → FileCacheVerdict[source]

Classify a resource file without taking any lock or downloading.

The lock-free decision half of refresh_cached_resource_file(): it resolves the remote resource and delegates to the local protocol’s classify_resource_file(). See gain#78.

download_cached_resource_file(resource: GenomicResource, filename: str, *, on_bytes: Callable[[int], None] | None = None) → tuple[str, str][source]

Download a resource file into cache unconditionally.

Takes the per-file lock and copies the file regardless of its local state – the decision was already made by classify_cached_resource_file(). See gain#78.

file_exists(resource: GenomicResource, filename: str) → bool[source]: Check if given file exist in give resource.

get_all_resources() → Generator[GenomicResource, None, None][source]: Return generator for all resources in the repository.

get_all_resources_dict() → dict[str, GenomicResource][source]: Return dictionary for all resources in the repository.

get_public_url() → str[source]

Return the public base URL of the repository.

Returns:: URL or path string pointing to a public repository root

get_resource_file_url(resource: GenomicResource, filename: str) → str[source]: Return url of a file in the resource.

get_resource_url(resource: GenomicResource) → str[source]: Return url of the specified resources.

get_url() → str[source]

Return the base URL of the repository.

Returns:: URL or path string pointing to repository root

invalidate() → None[source]: Invalidate internal cache of repository protocol.

load_manifest(resource: GenomicResource) → Manifest[source]: Load resource manifest.

open_bigwig_file(resource: GenomicResource, filename: str) → Any[source]

Open a bigwig file in a resource and return it.

Not all repositories support this method. Repositories that do no support this method raise and exception.

open_fasta_file(resource: GenomicResource, filename: str, index_filename: str | None = None, compressed_index_filename: str | None = None) → FastaFile[source]

Open a bgzipped fasta file in a resource and return a FastaFile.

Not all repositories support this method. Repositories that do not support this method raise an exception.

open_raw_file(resource: GenomicResource, filename: str, mode: str = 'rt', **kwargs: str | bool | None) → IO[source]: Open file in a resource and returns a file-like object.

open_repository_metadata() → Connection[source]: Open the db file for repo metadata and return the connection.

open_tabix_file(resource: GenomicResource, filename: str, index_filename: str | None = None) → TabixFile[source]

Open a tabix file in a resource and return a pysam tabix file.

Not all repositories support this method. Repositories that do no support this method raise and exception.

open_vcf_file(resource: GenomicResource, filename: str, index_filename: str | None = None) → VariantFile[source]

Open a vcf file in a resource and return a pysam VariantFile.

Not all repositories support this method. Repositories that do no support this method raise and exception.

refresh_cached_resource(resource: GenomicResource) → tuple[str, None][source]: Refresh all resource files in cache if neccessary.

refresh_cached_resource_file(resource: GenomicResource, filename: str) → tuple[str, str][source]: Refresh a resource file in cache if neccessary.

class gain.genomic_resources.cached_repository.GenomicResourceCachedRepo(child: GenomicResourceRepo, cache_url: str, **kwargs: str | None)[source]

Bases: GenomicResourceRepo

Defines caching genomic resources repository.

Carries the id of the repository it wraps, unchanged: a cache decides how a repository’s resources are served, not what the repository is called. The caching layer treats its other identities the same way – CachingProtocol.get_url() and get_public_url() both report the remote’s.

The id used to be f"{child.repo_id}.caching_repo", which renamed a repository an operator had already named, and renamed it after the definition was validated: the ids check_child_ids_are_unique reasoned about were then not the ids the repositories were built with, so a definition pairing {"id": "a", "cache_dir": ...} with {"id": "a.caching_repo"} passed validation and built two repositories sharing one id. See #447.

find_resource(resource_id: str, version_constraint: str | None = None, repository_id: str | None = None) → GenomicResource | None[source]

Return requested resource or None if not found.

Mirrors get_resource: the child resolves the resource (and owns repository_id semantics), then the hit is wrapped so the returned resource is cache-backed.

Forwarding repository_id unchanged is what makes this repository answer to its own id: it carries the child’s id (#447), so a filter naming this repository names the child too, and the child self-names – a leaf protocol repo by comparing the filter against its own id, a group by dropping the filter it matches itself. There is no separate self-match to keep in step here.

This used to enumerate every resource and pick the highest version across protocols, which had two defects. It filtered repository_id against the cache protocol’s id – registered as f"{proto_id}.cached" – so the filter never matched anything. And because get_resource already delegated, the two methods could return different versions of the same id when a group repository’s children overlapped. Group child order is a priority list; first child wins. See #429.

get_all_resources() → Generator[GenomicResource, None, None][source]: Return a generator over all resource in the repository.

get_resource(resource_id: str, version_constraint: str | None = None, repository_id: str | None = None) → GenomicResource[source]

Return one resource with id qual to resource_id.

If resource is not found, exception is raised.

repository_id restricts the lookup to the repository carrying that id, anywhere in this repository’s tree – including this repository itself: every repository answers to its own id, so passing repo.repo_id is equivalent to passing nothing (#447). A falsy repository_id is no filter at all.

get_resource_cached_files(resource_id: str) → set[str][source]: Get a set of filenames of cached files for a given resource.

invalidate() → None[source]

Clear cached state and force reload on next access.

Implementations should clear any cached resource lists, metadata, or file contents to ensure fresh data is loaded.

search_resources(search_term: str | None = None, resource_type: str | None = None, resource_query: str | None = None) → Generator[GenomicResource, None, list[tuple[str, str]] | None][source]

Search resources by FTS term, type and/or wildcard query.

All supplied filters conjoin.

The generator’s return value carries the (repository id, reason) pairs of the children a group skipped while still answering (ADR 0012, gain#686); None and [] both mean nothing was skipped. A for loop discards it, which is exactly right for a caller that does not present totals.

gain.genomic_resources.cached_repository.cache_resources(repository: GenomicResourceRepo, resource_ids: Iterable[str] | None, workers: int | None = None, *, progress: bool = True) → None[source]: Cache resources from a list of remote resource IDs.

gain.genomic_resources.cli module

Provides CLI for management of genomic resources repositories.

class gain.genomic_resources.cli.CommandResult(needs_update: int = 0, failed: frozenset[str] = frozenset({}), repo_failed: bool = False, wrote: bool = False)[source]

Bases: object

What a repository-management command found and what it could not do.

Three outcomes, deliberately kept apart (gain#364):

needs_update – how many resources are OUT OF DATE. Only a --dry-run reports this; a real run repairs them instead of counting them. This is the meaning the plain int these commands used to return carried.
failed – the ids of the resources that are BROKEN: whatever GAIn was asked to do to them raised, or silently did not happen. A run collects these rather than aborting on the first one, so a single broken resource cannot stop the healthy ones from being repaired.
repo_failed – something failed that no single resource can be blamed for: the repository’s own configuration, or a statistics task graph whose failure the per-resource check could not pin on any resource. Inventing a resource id for it would be a lie, but it still has to make the run exit non-zero.

An int could express only the first, which is why non-dry-run repair was structurally incapable of reporting failure.

wrote is not a fourth outcome but a fact about the run: whether it changed anything on disk. The resource-scoped commands read it to note that the repository-global artifacts are now behind the resources they describe (gain#760).

failed: frozenset[str] = frozenset({})

property has_failures: bool: Whether anything failed at all, attributable or not.

needs_update: int = 0

repo_failed: bool = False

wrote: bool = False

class gain.genomic_resources.cli.ManifestOutcome(updates_needed: dict[str, bool], failed: frozenset[str], wrote: bool)[source]

Bases: NamedTuple

What a manifest pass over a set of resources found.

updates_needed is keyed by the resources the pass got through, and valued by whether that resource’s manifest is stale. failed names the resources whose manifest could not be built at all - today, only a resource whose content drifted from its .dvc sidecars under --without-dvc; it has NO entry in updates_needed (#373). wrote is whether the pass saved any manifest.

failed: frozenset[str]: Alias for field number 1

updates_needed: dict[str, bool]: Alias for field number 0

wrote: bool: Alias for field number 2

gain.genomic_resources.cli.cli_browse(cli_args: list[str] | None = None) → None[source]: Provide CLI for repository browsing.

gain.genomic_resources.cli.cli_manage(cli_args: list[str] | None = None) → None[source]: Provide CLI for repository management.

gain.genomic_resources.cli_cache_repo module

CLI for caching genomic resources referenced by an annotation pipeline.

The tool resolves a GenomicResourceRepo and an AnnotationPipeline from a combination of command-line flags and the registered genomic context providers, then caches the resources the pipeline depends on.

The annotation pipeline can come from:

the positional pipeline argument — a file path or a GRR resource id of type annotation_pipeline. This is supplied by the standard CLIAnnotationContextProvider (the same mechanism annotate_columns / annotate_vcf use).
-i / --instance — when the GPFInstanceContextProvider plugin is installed, this resolves to the pipeline of the configured GPF instance.

When both are supplied, the positional pipeline wins (it is supplied by a higher-priority context provider); the tool logs which source it used. When neither is supplied (positional omitted / left at its "context" sentinel and no instance pipeline available), the tool logs a warning and exits cleanly without caching anything.

gain.genomic_resources.cli_cache_repo.cli_cache_repo(argv: list[str] | None = None) → None[source]: Cache genomic resources used by an annotation pipeline.

gain.genomic_resources.cli_dvc module

What grr_manage refuses about a .dvc sidecar, before it starts.

Its own module because the refusal has two gates that must say the same thing: this pre-flight, which runs before a command touches the repository, and the manifest builder’s own check in repository.collect_dvc_entries, which is what makes a manifest impossible to build from a sidecar GAIn cannot verify (#255, #284). The error they raise and the message they say it with live below both, in dvc and repository, so the builder’s gate does not depend on the CLI layer (#721); this module keeps the pre-flight.

gain.genomic_resources.cli_dvc.refuse_dvc_directory_outputs(proto: ReadWriteRepositoryProtocol, resources: Sequence[GenomicResource]) → None[source]

Refuse a dvc add <dir> output before the command writes anything.

The gate in repository.collect_dvc_entries fires where ONE resource’s manifest is built, which is far too late for a command that spans a repository: by then the resources ordered before the offender have their .MANIFEST and .grr state written, and *-stats / *-info have run a whole task graph – so a run that refuses the repository could still leave statistics and info pages behind for the resources it happened to reach first (#284). A refusal must be side-effect-free, so the sidecars are read up front, here, and the command fails before it touches anything.

Scoped to the resources the command SELECTED, which for the repo-* subcommands is the whole repository: a resource GAIn refuses does not make a resource-* command on some OTHER resource illegal, in keeping with one broken resource never stopping work on the healthy ones (gain#503).

A sidecar this pass cannot read or cannot parse is not its business: it asks one question – does any sidecar describe a directory? – and repository.collect_dvc_entries remains the one place that reports an unusable sidecar, so a run does not warn about it twice. Neither is a resource whose files cannot even be LISTED – an unreadable directory, a DVC cache this run may not traverse, a remote store that fails to describe a key it just listed. It is skipped for the same reason: the command’s own per-resource handler is what reports it and fails that resource alone, and a pre-flight that raised instead would take the whole run down over one broken resource – the very failure mode gain#503 removed.

It costs one extra listing of the selected resources, since the listing is what says which sidecars exist and the answer is needed before the first write. Only the sidecars are then read, and a .dvc file is a few hundred bytes of YAML – nothing next to the hashing and statistics the pass protects.

Raises:: UnsupportedDvcDirectoryOutputError – some selected resource has a dvc add <dir> output.

gain.genomic_resources.cli_errors module

How grr_manage reports a failure that belongs to ONE resource.

Its own module because more than one command needs it and they must agree: a repository-wide command that dies on the first bad resource reports half a repository and hides the rest (gain#364, gain#503).

It is imported from the statistics package (statistics.region_fold), so it may import nothing of GAIn beyond the logging shim and the leaves it names an exception from – anything more can reach histogram, which imports that package’s base class, and close a cycle. The architecture suite pins the allowlist and says what the cycle is (gain#1293).

gain.genomic_resources.cli_errors.report_resource_failure(err: Exception, action: str, resource_id: str) → None[source]

Report a failed operation on one resource, at the right tier.

action names what could not be done – never the phase the failure happened in. A handler that wraps several operations cannot know which one raised, and naming the wrong one sends the reader looking in the wrong place (gain#364); the cause, which is always carried, says it.

gain.genomic_resources.cli_list module

The grr_manage list command.

Split out of cli because listing is the one read-only command in that module: everything else there manages a repository – rebuilds manifests, statistics and info pages – while this only describes what is already there. Keeping it apart also keeps the reporting policy it needs (a bad resource is named and skipped, never fatal) from being read as the management commands’ policy, which is to fail the run.

gain.genomic_resources.cli_list.run_list_command(proto: ReadOnlyRepositoryProtocol | GenomicResourceRepo, args: Namespace) → None[source]: List the resources of a repository.

gain.genomic_resources.data_frame_resource module

Loading helpers for data_frame genomic resources.

gain.genomic_resources.data_frame_resource.load_data_frame_from_resource(resource: GenomicResource | None) → DataFrame[source]: Load a pandas DataFrame from a data_frame genomic resource.

gain.genomic_resources.data_frame_resource.load_data_frame_from_resource_id(resource_id: str, grr: GenomicResourceRepo | None = None) → DataFrame[source]: Load a data_frame from a genomic resource id.

gain.genomic_resources.draw_score_histograms module

exception gain.genomic_resources.draw_score_histograms.ScorelessResourceError[source]

Bases: TypeError

A resource whose type carries no scores at all.

Distinguished from the errors that mean a resource is broken: there is nothing wrong with a genome, it simply has no histograms to draw. A TypeError because that is what selecting such a resource by id has always raised, and callers still get exactly that.

gain.genomic_resources.draw_score_histograms.main(argv: list[str] | None = None) → None[source]: Liftover dae variants tool main function.

gain.genomic_resources.draw_score_histograms.parse_cli_arguments() → ArgumentParser[source]: Create CLI parser.

gain.genomic_resources.dvc module

The .dvc sidecar vocabulary.

dvc add <file> drops a <file>.dvc sidecar next to the data file it stores and gitignores the file itself. GAIn reads those sidecars in three places – the repository scan, grr_manage’s entry collection and the manifest builder – and this module is the only place that interprets one, so the three can never classify the same sidecar differently.

Recognising a sidecar by name is part of that vocabulary: naming it here keeps the info pages, which hide sidecars, and the manifest builder, which reads them, from ever disagreeing about which files are sidecars.

class gain.genomic_resources.dvc.DvcContentDrift(name: str, content_md5: str, dvc_md5: str)[source]

Bases: object

A materialised file whose content disagrees with its .dvc.

Produced only by the verifier (grr_manage --without-dvc), which is the one mode that reads a DVC-managed file’s bytes (#373).

content_md5: str

dvc_md5: str

name: str

exception gain.genomic_resources.dvc.DvcContentDriftError(resource_id: str, drifts: Sequence[DvcContentDrift])[source]

Bases: ValueError

Every file of ONE resource whose content drifted from its sidecar.

Collected rather than raised on the first offender, so that a single grr_manage --without-dvc run reports all of them (#373). It is a ValueError because it is a fault of the RESOURCE, and cli_errors.report_resource_failure reports those as one line carrying the cause, with the traceback demoted to DEBUG (gain#364).

exception gain.genomic_resources.dvc.UnsupportedDvcDirectoryOutputError[source]

Bases: Exception

A resource declares a dvc add <dir> output, which GAIn refuses.

Raised by cli_dvc.refuse_dvc_directory_outputs and by repository.collect_dvc_entries, and turned into a non-zero exit by cli.cli_manage (#255). Defined here, below both, so that the manifest builder’s gate does not depend on the CLI layer (#721).

gain.genomic_resources.dvc.dvc_sidecar_target(name: str) → str[source]: Return the path the sidecar name describes.

gain.genomic_resources.dvc.is_dvc_directory_out(out: dict[str, Any]) → bool[source]

Return True if a .dvc output describes a dvc add <dir> output.

DVC writes two signals for a directory output, and either one on its own is enough to recognise it:

its md5 is the hash of a DVC cache object - a listing of the directory’s files - and carries a .dir suffix to say so;
it declares nfiles, the number of files in the directory.

Neither is checked in isolation: an out that lost its nfiles is still a directory, and so is one whose md5 sum lost its suffix. GAIn does not support directory outputs - it cannot verify a .dir md5 sum against anything it can read - so grr_manage refuses a resource that has one (#255).

gain.genomic_resources.dvc.is_dvc_sidecar(name: str) → bool[source]: Return True if name names a .dvc sidecar.

gain.genomic_resources.dvc.parse_dvc_pointer_out(content: str | bytes, basename: str) → dict[str, Any] | None[source]

Parse a .dvc sidecar; return the output entry describing basename.

A well-formed .dvc pointer is a mapping with an outs list of mappings; the output that describes basename is the one whose path equals it exactly. Anything else - a mapping without outs, an outs that is not a list of mappings, an output for some other path, YAML that does not parse, non-UTF-8 bytes - is not a pointer for basename and yields None.

Parsing NEVER raises. This is the single place a .dvc file is interpreted, so that the repository scan (_is_dvc_managed_leaf, which must never abort on stray content) and repository.collect_dvc_entries cannot classify the same sidecar differently (#251).

Parameters:

content – raw content of the .dvc file; bytes are safe to pass - yaml decodes them itself, so a binary file cannot raise a UnicodeDecodeError past this function.
basename – the base name of the data file the pointer must describe.

Returns:

The matching outs entry, or None if the content is not a pointer for basename. The entry is NOT validated beyond its path: callers that need an md5 sum and a size must check for them.

gain.genomic_resources.fsspec_protocol module

Provides GRR protocols based on fsspec library.

exception gain.genomic_resources.fsspec_protocol.ChecksumMismatchError[source]

Bases: RetryableCopyError

A completed download whose md5 disagrees with the manifest.

Almost always a truncated or corrupted transfer.

exception gain.genomic_resources.fsspec_protocol.CorruptedPublishError[source]

Bases: RetryableCopyError

A published object that differs in size from the verified download.

The download’s byte-count and md5 checks both run before the move, so they describe the temp file, not the object the move landed. Since the digest is carried across rather than recomputed from the store (gain#865), a move that published something else – a partial object-store copy, a rename that lost the tail, one that left no object at all – would be recorded under the digest of bytes that are no longer there: size and timestamp come from the published object and agree with it, the md5 comes from the download and agrees with the manifest, so every later cache verdict passes and the file is served corrupt indefinitely (gain#880).

Only size-changing corruption is caught. Catching the rest needs a second full read of every downloaded file – for a remote store, a second transfer – which is exactly the cost gain#865 removed.

Raised on a second, digest-free path since gain#933: the repository’s own artifacts publish through _publish_file(), which compares the published object against the bytes it staged rather than against a manifest. The reasoning above is about the download, but the guard is the same one – it lives on the move, so everything that moves gets it. The retryability inherited from RetryableCopyError does not reach that second path, though: its callers publish once and propagate, so a corrupting publish of a repository artifact fails rather than retrying.

class gain.genomic_resources.fsspec_protocol.FileCacheVerdict(needs_download: bool, size: int)[source]

Bases: NamedTuple

The lock-free classification of a single resource file.

needs_download is True when the local copy is missing or has drifted from the remote manifest and must be (re)downloaded; size is the manifest-recorded byte size of that pending download (0 when nothing needs downloading). See gain#78.

needs_download: bool: Alias for field number 0

size: int: Alias for field number 1

class gain.genomic_resources.fsspec_protocol.FsspecReadOnlyProtocol(*args: Any, **kwargs: Any)[source]

Bases: ReadOnlyRepositoryProtocol

Provides fsspec genomic resources repository protocol.

(proto_id, url) names ONE protocol instance for the life of the process: __new__ memoizes it in _FSSPEC_PROTOCOLS, keyed on the url’s canonical scheme://netloc/path form so its spelling cannot split one repository in two, and never evicts. A second build over that pair therefore reaches the object every earlier caller is already holding, and Python re-runs __init__ on it.

That makes a rebuild a refresh – it drops the resource memo, which is how a caller that has just changed a repository reads it back. It is not a reconfiguration: a rebuild asking for a different mode, public url or credentials is refused rather than applied to the incumbent. See docs/adr/0005-fsspec-protocol-memo-rebuild.md (#514).

Construction is one atomic step, and a protocol is reachable through the memo only once the whole of it has returned (#527). __new__ records an in-flight construction instead of publishing, so a thread that arrives while another is building that key waits and is answered with the same, configured instance – rather than either building a second protocol over a key that names one, or reading one whose filesystem, url, public_url and kwargs are not bound yet.

close() → None[source]: Close the genomic resource.

file_exists(resource: GenomicResource, filename: str) → bool[source]: Check if given file exist in give resource.

get_all_resources() → Generator[GenomicResource, None, None][source]: Return generator over all resources in the repository.

get_all_resources_dict() → dict[str, GenomicResource][source]

Return the repository’s resources, keyed by full id.

The whole memo protocol – the lock, the check-then-populate, the keying, the ordering and the return – lives here and only here, for every fsspec protocol. A subclass enumerates the repository by overriding _enumerate_resources and inherits the rest (#515).

FsspecReadWriteProtocol used to carry a second copy of all of it and differ only in the enumeration, which is how the release-then-read defect of #458 came to be in two places while the report named one.

get_file_content(resource: GenomicResource, filename: str, *, uncompress: bool = True, mode: str = 't') → Any[source]

Return content of a file in given resource.

Overrides the base, which opens through open_raw_file and reads on the handle it returns: that open is redacted and the read is not, so on an authed GRR a failure mid-read surfaces the credential-bearing fetch url verbatim (gain#1058). _read_fetch_file covers both.

This sits IN FRONT of the fasta-index copy gain#1017 fixed – ReferenceGenome.open reads the .fai through here first – and also backs load_manifest and load_yaml.

get_loaded_manifest reads a missing manifest’s FileNotFoundError as “no manifest”, and the rebuild keeps that type: it reconstructs via type(exc)(message), which FileNotFoundError supports.

An error that CANNOT be reconstructed from a message alone loses its type – an aiohttp.ClientResponseError (an HTTP 5xx, which fsspec does not translate) needs request_info and history. It comes back an OSError, or a RetryableCopyError where the original was transient, which _rebuild_error_without_url_credentials preserves so redaction cannot change a retry decision. Safe here for the reason _copy_resource_file_to_local sets out: no retry or control-flow decision on this path keys off the type.

uncompress names nothing on this path and never has, so compression=None preserves the as-stored read exactly – see _copy_resource_file_to_local, which found the same. Repairing the dead parameter is separate work.

get_public_url() → str[source]

Return the public base URL of the repository.

Returns:: URL or path string pointing to a public repository root

get_resource_url(resource: GenomicResource) → str[source]

Return url of the specified resources.

The resource id is the other operand of this join and is no less untrusted than a file name – on the remote path it is read verbatim out of the repository’s .CONTENTS.json.gz – so it is contained here, at the join, exactly as get_resource_file_url contains the name (gain#467).

get_url() → str[source]

Return the base URL of the repository.

Returns:: URL or path string pointing to repository root

invalidate() → None[source]

Drop the memoized resources, leaving handed-out ones alone.

Clears only this protocol’s own cache. The resources in the memo are handed out by reference, so unbinding their proto on the way out – as this used to do – breaks the objects live callers are already holding, and they raise AttributeError on None at first use (#513). Their lifetime is the caller’s business; dropping the memo is enough to make the next read reload, and enough to let a resource no one else holds be collected, since the memo held the only reference to it. Not the protocol, though – _FSSPEC_PROTOCOLS memoizes every protocol for the life of the process and never evicts, so no amount of unbinding here ever released one.

kwargs: dict[str, Any]

load_contents() → list[dict[str, Any]][source]: Load the content JSON of the repository.

load_manifest(resource: GenomicResource) → Manifest[source]: Load resource manifest.

md5_contents() → str[source]: Calculate md5 hash of the repository content.

open_bigwig_file(resource: GenomicResource, filename: str) → Any[source]

Open filename of resource with pyBigWig.

A file GRR is opened by path. An s3, http or https GRR is opened by url – presigned for s3 – which needs a pyBigWig built with libcurl (pyBigWig.remote == 1); the PyPI wheel is not, and a remote open on it is refused with an OSError naming the remedies. That refusal never carries the url, which may hold a credential.

open_fasta_file(resource: GenomicResource, filename: str, index_filename: str | None = None, compressed_index_filename: str | None = None) → FastaFile[source]

Open a bgzipped fasta file in a resource and return a FastaFile.

Not all repositories support this method. Repositories that do not support this method raise an exception.

open_raw_file(resource: GenomicResource, filename: str, mode: str = 'rt', **kwargs: str | bool | None) → IO[source]: Open file in a resource and returns a file-like object.

open_repository_metadata() → Connection[source]: Open the db file for repo metadata and return the connection.

open_tabix_file(resource: GenomicResource, filename: str, index_filename: str | None = None) → TabixFile[source]

Open a tabix file in a resource and return a pysam tabix file.

Not all repositories support this method. Repositories that do no support this method raise and exception.

open_vcf_file(resource: GenomicResource, filename: str, index_filename: str | None = None) → VariantFile[source]

Open a vcf file in a resource and return a pysam VariantFile.

Not all repositories support this method. Repositories that do no support this method raise and exception.

class gain.genomic_resources.fsspec_protocol.FsspecReadWriteProtocol(*args: Any, **kwargs: Any)[source]

Bases: FsspecReadOnlyProtocol, ReadWriteRepositoryProtocol

Provides fsspec genomic resources repository protocol.

build_content_file(failed: frozenset[str] = frozenset({})) → list[dict[str, Any]][source]

Build the content of the repository (i.e ‘.CONTENTS.json.gz’).

failed names resources this run could not verify; each is published from the manifest it already had, or left out if it never had one, so a failed run never rebuilds a manifest from scratch and poisons the contents with it (#373).

Only the gzipped index is written. An uncompressed .CONTENTS.json left by an older release is reported rather than deleted (#758).

build_index_info(repository_template: str = 'grr_index.jinja', about_template: str | None = 'grr_about.jinja', failed: frozenset[str] = frozenset({})) → dict[source]

Build info dict for the repository.

failed names resources this run could not verify; each is described from the manifest it already had, or left off the index page if it never had one, so the page never triggers a build-from-scratch of a failed resource’s manifest (#373).

classify_resource_file(remote_resource: GenomicResource, dest_resource: GenomicResource, filename: str) → FileCacheVerdict[source]

Decide whether a resource file needs (re)downloading.

This is the lock-free decision half of update_resource_file(): it performs the same checks and the same state-refresh side effect (rebuild + save the .state on a missing state or one that no longer describes the stored file, and delete a file no longer in the remote manifest), but it never copies/downloads. The verdict’s size is the manifest byte size for files that will download (0 otherwise). See gain#78.

The one question it opens with – is the file there at all – is asked as a stat rather than as a boolean, because the same dict carries the size and the change token a rebuilt state needs. So the rebuild reads only what that dict cannot say: the modification time, and the md5 off the bytes themselves. Asking for a bool and then rebuilding from scratch asked the store about one key five times where twice will do (gain#1039). Only the rebuild is cheaper for it: a verdict that finds its recorded state current spends what it always did, one stat either way.

A stat that fails for a reason other than the file being absent now reaches the caller instead of reading as “not cached”. That is the one thing exists() did that this does not: fsspec’s base implementation answers False to every exception, so an unreadable cache directory used to be answered with a download that was going to fail on the same directory a moment later. On s3 it is not even a change – s3fs’s own exists swallows only FileNotFoundError.

The stat is taken before the md5 that is recorded beside it, so a file rewritten in between is recorded with the older token beside the newer digest. That pairing is self-correcting rather than a lost update: the next verdict reads the token, finds it moved, and rebuilds. It is the safe half of the ordering – a token read after the digest would pair a fresh token with a superseded md5, and nothing afterwards would notice.

collect_all_resources() → Generator[GenomicResource, None, None][source]: Return generator over all resources managed by this protocol.

copy_resource_file(remote_resource: GenomicResource, dest_resource: GenomicResource, filename: str, on_bytes: Callable[[int], None] | None = None) → ResourceFileState | None[source]

Copy a resource file into repository.

A transient stall or drop mid-download (common when fetching a large resource over a slow HTTP GRR link) is retried from scratch with exponential backoff rather than aborting the file. See gain#43.

on_bytes, when given, is called with the number of bytes written for each chunk during the download (see gain#77). Because a retried attempt re-downloads the whole file from scratch, the bytes credited by a failed attempt are rolled back with a single compensating negative call before the retry, so a caller-side byte counter never double-counts.

delete_resource_file(resource: GenomicResource, filename: str) → None[source]: Delete a resource file and it’s internal state.

get_resource_file_change_token(resource: GenomicResource, filename: str) → str | None[source]

Return the store’s change token for a resource file, if any.

A change token is whatever the store itself offers as “this is the version of the object you are looking at”: it changes on every write and holds still for as long as the object is not written. Stores that offer none answer None, and for them the modification time remains the only change hint there is.

The value is opaque. It is never parsed, never compared against an md5 sum and never assumed to be one, even where a particular store happens to derive it from one.

See ADR 0022 for why a state is judged by this rather than by the modification time.

get_resource_file_size(resource: GenomicResource, filename: str) → int[source]: Return the size of a resource file.

get_resource_file_timestamp(resource: GenomicResource, filename: str) → float[source]: Return the timestamp (ISO formatted) of a resource file.

load_resource_file_state(resource: GenomicResource, filename: str) → ResourceFileState | None[source]

Load resource file state from internal GRR state.

If the specified resource file has no internal state returns None.

obtain_resource_file_lock(resource: GenomicResource, filename: str, timeout: float = -1) → AbstractContextManager[source]

Lock a resource’s file.

The lock is a lockfile, which only provides mutual exclusion on a local filesystem. Off file this used to return a no-op context manager – every caller “acquired” it instantly, so the caching protocol serialised nothing and concurrent readers saw partially written files. Refuse rather than hand out a lock that does not lock; a GRR cache must be local. See #473.

publish_raw_file(resource: GenomicResource, filename: str, mode: str = 'wt') → AbstractContextManager[IO][source]

Open a resource file for a write that replaces it, or does not.

The publishing counterpart of open_raw_file(): what the caller writes reaches filename only once the handle has closed cleanly. A write that fails part-way – and an interrupt – leaves whatever was published before exactly as it was, rather than truncating it in place with nothing to roll back to (gain#933).

A separate name rather than a flag on open_raw_file(): the two differ in what they guarantee, not in a parameter, and the callers that need the guarantee are not the ones that need a plain handle.

publish_repository_file(filename: str, mode: str = 'wb') → AbstractContextManager[IO][source]

Publish one of the repository’s own artifacts, by file name.

The counterpart of publish_raw_file() for the artifacts that belong to no resource. The artifacts this protocol builds itself reach _publish_file() directly; this is the way in for one that is built somewhere else and handed over as bytes – the FTS search index, which the CLI assembles from every resource’s implementation and cannot build from in here without importing the layer above it (gain#948).

Same seam, so the same guarantee: the artifact already published is replaced by a completed one in a single move, or not at all.

Binary by default, unlike publish_raw_file(): a repository artifact is a built object – gzipped bytes, a rendered page – and not the text a resource file usually is.

filename names an artifact of the repository itself and is joined to its url unvalidated, which is safe only while callers pass a constant. There is no containment rule to check it against: the resource-file joins each re-run validate_resource_file_name() (gain#467), but that rule is about staying inside a resource, and these artifacts sit beside the resources rather than in one. A caller that ever wants to publish a name it did not author needs such a rule written first.

save_resource_file_state(resource: GenomicResource, state: ResourceFileState) → None[source]: Save resource file state into internal GRR state.

scan_resource_entries(resource: GenomicResource) → ResourceScan[source]: Scan the resource and return what was found.

update_resource_file(remote_resource: GenomicResource, dest_resource: GenomicResource, filename: str) → ResourceFileState | None[source]: Update a resource file into repository if needed.

gain.genomic_resources.fsspec_protocol.GRR_INTERNAL_DIR = '.grr': Directory inside a resource holding the protocol’s own bookkeeping – per-file .state documents, lockfiles, and partial downloads. Not part of the resource: everything that enumerates resource files skips it.

exception gain.genomic_resources.fsspec_protocol.RetryableCopyError[source]

Bases: OSError

A copy failure the download loop retries from scratch.

Retryability is a property of the class, not of a list kept beside it: copy_resource_file catches this base, so a new transient failure shape becomes retryable by subclassing it and nothing else (gain#934).

What the retry buys is a fresh remote handle and a fresh temp file, so only faults a second full attempt could plausibly clear belong here – a stalled link, a short read, corrupt bytes. A fault that would repeat identically is a plain OSError and surfaces on the first attempt.

The classification is the download path’s, and it reaches everything that path does – the move included, so a corrupting publish inside a download is retried like any other transient fault.

One shape reaches here from outside a download: redacting a credential-bearing failure whose own type cannot be reconstructed from a message rebuilds it as this class, so that redaction cannot silently reclassify a transient failure as permanent (gain#1078, ADR 0023). Such an error can therefore surface from a plain read – get_file_content, a tabix header – where nothing will retry it. It is still true of it that a second attempt could plausibly have cleared it; only the opportunity is absent.

exception gain.genomic_resources.fsspec_protocol.TruncatedDownloadError[source]

Bases: RetryableCopyError

A download that ended short of the manifest’s recorded byte size.

A silent short read in the fsspec range-reassembly layer (gain#292, H1) makes infile.read() return EOF before the whole file has been streamed; the copy loop stops on that empty read and writes a truncated file. Caught explicitly by byte count – before the md5 check – so the failure is reported as the truncation it is, with both the received and the expected size, rather than as an opaque checksum mismatch.

gain.genomic_resources.fsspec_protocol.build_fsspec_protocol(proto_id: str, root_url: str, **kwargs: str | bool | None) → FsspecReadOnlyProtocol | FsspecReadWriteProtocol[source]

Create fsspec GRR protocol based on the root url.

read_only is the one boolean among the keyword arguments – hence the widened value type; every other keyword is a url or a credential. It is absent by default rather than False so that asking for a read-write protocol can be told apart from not asking at all: the two mean different things on an http(s) url, where only one of them is serviceable (#528).

gain.genomic_resources.fsspec_protocol.build_inmemory_protocol(proto_id: str, root_path: str, content: dict[str, Any]) → FsspecReadWriteProtocol[source]: Build and return an embedded fsspec protocol for testing.

gain.genomic_resources.fsspec_protocol.build_local_resource(dirname: str, config: dict[str, Any]) → GenomicResource[source]: Build a resource from a local filesystem directory.

gain.genomic_resources.fsspec_protocol.canonical_public_url(public_url: str) → str[source]

Return a public url in the one spelling two builds can be compared in.

Only for comparison – the value a protocol reports through get_public_url stays exactly as its caller wrote it.

gain.genomic_resources.genomic_context module

Genomic context provides a way to collect various genomic resources from various sources and make them available through a single interface.

The module follows a registry-based approach. Providers register themselves and are later consulted (in priority order) to build individual GenomicContext instances. Every created context is combined into a PriorityGenomicContext, offering a single access point for resources such as genomic resource repositories, reference genomes, gene models, annotation pipelines, etc. Providers can be registered programmatically via register_context_provider() or discovered automatically through entry points.

Example usage of genomic context in a tool with command line interface:

import argparse
import sys

from gain.genomic_resources.genomic_context import (
    context_providers_add_argparser_arguments,
    context_providers_init,
    get_genomic_context,
)


parser = argparse.ArgumentParser()
context_providers_add_argparser_arguments(parser)

args = parser.parse_args(sys.argv[1:])
context_providers_init(**vars(args))
genomic_context = get_genomic_context()

If you don’t need command line arguments you can do:

context_providers_init()
genomic_context = get_genomic_context()

When you need a CLI with all defaults and without modifying the argument parser you can do:

context_providers_init_with_argparser("GenomicTool")
genomic_context = get_genomic_context()

class gain.genomic_resources.genomic_context.DefaultRepositoryContextProvider[source]

Bases: GenomicContextProvider

Provide access to the default genomic resources repository.

The default repository is resolved via build_genomic_resource_repository() using the environment configuration. The resulting context exposes a single key, "genomic_resources_repository", which can be consumed by other code participating in the context chain.

add_argparser_arguments(parser: ArgumentParser, **kwargs: Any) → None[source]

Declare command line arguments for this provider.

The default repository provider is fully configuration driven and has nothing to expose on the CLI, so the method intentionally leaves the parser untouched. The override exists to make the behaviour explicit in the generated documentation.

init(**kwargs: Any) → GenomicContext[source]

Instantiate a context backed by the default GRR.

Parameters:: **kwargs – Accepted for interface compatibility; the provider ignores runtime keyword arguments because everything is derived from the global configuration.
Returns:: A context exposing a single genomic_resources_repository entry pointing at the default repository instance.
Return type:: GenomicContext

gain.genomic_resources.genomic_context.build_cli_genomic_context(cli_args: dict[str, Any]) → GenomicContext[source]

Initialise the context providers from parsed CLI arguments and merge.

The one call a CLI tool makes between parsing its arguments and asking for a GRR: every registered provider sees cli_args (see context_providers_init()), and the result is the merged PriorityGenomicContext of get_genomic_context().

gain.genomic_resources.genomic_context.clear_registered_contexts() → None[source]

Forget all contexts created by context_providers_init().

This function exists primarily for testing scenarios where the global registry should be reset between test cases.

gain.genomic_resources.genomic_context.context_providers_add_argparser_arguments(parser: ArgumentParser, **kwargs: Any) → None[source]

Delegate command line argument registration to each provider.

Parameters:: parser – The parser that should receive additional arguments from every registered provider.

gain.genomic_resources.genomic_context.context_providers_init(**kwargs: Any) → None[source]

Materialize contexts from every registered provider.

The function walks all registered providers in priority order and asks each of them to initialise a GenomicContext. The resulting contexts are stored for later retrieval via get_genomic_context().

Notes

Providers are invoked at most once per process. Subsequent calls are ignored until clear_registered_contexts() is executed, which is especially helpful in unit tests.

Parameters:: **kwargs – Keyword arguments forwarded to every provider’s init method.

gain.genomic_resources.genomic_context.context_providers_init_with_argparser(toolname: str = 'GenomicTool') → None[source]

Initialise providers using arguments parsed from sys.argv.

Parameters:: toolname – The program name presented to argparse.ArgumentParser.

Notes

This helper is useful for simple tools that do not customise their argument parser but still want to expose the command line options defined by registered context providers.

gain.genomic_resources.genomic_context.get_genomic_context() → GenomicContext[source]

Return a priority context that merges every registered context.

The returned PriorityGenomicContext respects the registration order, giving precedence to contexts added most recently when multiple contexts expose the same key.

gain.genomic_resources.genomic_context.get_grr_from_context(context: GenomicContext) → GenomicResourceRepo[source]: Get the genomic resource repository from the genomic context.

gain.genomic_resources.genomic_context.register_context(context: GenomicContext) → None[source]

Record context so it participates in future lookups.

Parameters:: context – The context instance to be considered when get_genomic_context() is invoked.

gain.genomic_resources.genomic_context.register_context_provider(context_provider: GenomicContextProvider) → None[source]

Register context_provider so it participates in initialization.

Parameters:: context_provider – The provider implementation that should be considered when contexts are assembled. Providers are stored in registration order and later sorted by their priority before initialization.

gain.genomic_resources.genomic_context_base module

Base classes and interfaces for genomic context management.

This module defines the foundational abstractions for organizing and accessing genomic resources through a unified context system. The central concept is GenomicContext, which acts as a key-value store exposing resources like genomic repositories, reference genomes, gene models, and annotation pipelines. Providers implementing GenomicContextProvider are responsible for building concrete context instances, often by consulting configuration files or command-line arguments.

The module also provides two concrete context implementations: SimpleGenomicContext for straightforward dictionary-backed contexts and PriorityGenomicContext for merging multiple contexts with fallback semantics.

Key Constants

GC_GRR_KEYstr: Standard key for the genomic resources repository object.
GC_REFERENCE_GENOME_KEYstr: Standard key for the reference genome object.
GC_GENE_MODELS_KEYstr: Standard key for the gene models object.
GC_ANNOTATION_PIPELINE_KEYstr: Standard key for the annotation pipeline object.

gain.genomic_resources.genomic_context_cli module

Command-line helpers for configuring genomic resource contexts.

This module exposes CLIGenomicContextProvider, a concrete implementation of GenomicContextProvider that resolves genomic resources based on command-line arguments. Tools can register the provider to let their users supply a genomic resources repository, reference genome, and gene models at runtime.

class gain.genomic_resources.genomic_context_cli.CLIGenomicContextProvider[source]

Bases: GenomicContextProvider

Resolve genomic resources from command-line arguments.

The provider allows CLI tools to override the default genomic resources repository, reference genome, and gene models. When invoked without any overrides, it falls back to the previously initialised genomic context so that defaults from gpf_instance or other providers remain available.

add_argparser_arguments(parser: ArgumentParser, **kwargs: Any) → None[source]

Expose CLI options that control genomic resource resolution.

Parameters:

parser – The argument parser that should receive the provider specific options.
**kwargs – skip_cli_reference_genome and skip_cli_gene_models leave out -R and -G for a tool that resolves neither from the command line. The GRR options are always added.

init(**kwargs: Any) → GenomicContext | None[source]

Create a SimpleGenomicContext based on CLI arguments.

Parameters:: **kwargs – Arguments produced from the command-line parser. The provider recognises grr_filename, grr_directory, reference_genome_resource_id, and gene_models_resource_id.
Returns:: A context containing the resolved objects, or None if the genomic resources repository could not be determined.
Return type:: GenomicContext | None

gain.genomic_resources.group_repository module

Provides group genomic resources repository.

class gain.genomic_resources.group_repository.GenomicResourceGroupRepo(children: list[GenomicResourceRepo], repo_id: str | None = None)[source]

Bases: GenomicResourceRepo

Defines group genomic resources repository.

find_resource(resource_id: str, version_constraint: str | None = None, repository_id: str | None = None) → GenomicResource | None[source]

Return one resource with id qual to resource_id.

If resource is not found, None is returned.

repository_id selects a repository by id under the same rule as get_resource() – a repository answers to its own id, and a falsy id is no filter.

get_all_resources() → Generator[GenomicResource, None, None][source]: Return a generator over all resource in the repository.

get_resource(resource_id: str, version_constraint: str | None = None, repository_id: str | None = None) → GenomicResource[source]

Return one resource with id qual to resource_id.

If resource is not found, exception is raised.

repository_id restricts the lookup to the repository carrying that id, anywhere in this repository’s tree – including this repository itself: every repository answers to its own id, so passing repo.repo_id is equivalent to passing nothing (#447). A falsy repository_id is no filter at all.

invalidate() → None[source]

Clear cached state and force reload on next access.

Implementations should clear any cached resource lists, metadata, or file contents to ensure fresh data is loaded.

search_resources(search_term: str | None = None, resource_type: str | None = None, resource_query: str | None = None) → Generator[GenomicResource, None, list[tuple[str, str]]][source]

Search resources by FTS term, type and/or wildcard query.

All supplied filters conjoin.

The generator’s return value carries the (repository id, reason) pairs of the children a group skipped while still answering (ADR 0012, gain#686); None and [] both mean nothing was skipped. A for loop discards it, which is exactly right for a caller that does not present totals.

search_resources_by_child(search_term: str | None = None, resource_type: str | None = None, resource_query: str | None = None) → Generator[tuple[GenomicResourceRepo, GenomicResource], None, list[tuple[str, str]]][source]

Search, pairing each hit with the child repository serving it.

The pair names the repository that actually holds the resource: a nested group projects its own pairs upward rather than naming itself, so a caller never has to take a group apart to label a row.

This is where a child that cannot answer the filter is skipped, and search_resources() is its projection – the two cannot drift, because there is only one loop. The skips of a search that still answered are the generator’s return value (gain#686); a for loop discards them, which is exactly right for the callers that already hear about them from the log.

gain.genomic_resources.histogram module

Handling of genomic scores statistics.

Currently we support only genomic scores histograms.

class gain.genomic_resources.histogram.CategoricalHistogram(config: CategoricalHistogramConfig, counter: dict[str | int, int] | None = None, *, truncated: bool = False, unique_values: int | None = None, total_count: int | None = None)[source]

Bases: Statistic

Class for categorical data histograms.

UNIQUE_VALUES_LIMIT = 100

add_batch(values: ndarray, weights: ndarray) → None[source]

Add a batch of (value, weight) pairs, vectorized.

Equivalent to calling add_value() over each pair in order: the same None skip, the same per-value counts, the same TypeError naming the first value that is neither str nor int, and the same HistogramError when the batch takes the histogram past UNIQUE_VALUES_LIMIT. This is the hot path for the statistics scan, where a per-value Python call dominates the cost.

The limit’s message reports UNIQUE_VALUES_LIMIT + 1 because that is the count add_value() always raises at: it tests after every single add, so the first add that exceeds the limit is the one that raises, and the histogram holds exactly one value too many. Which values the counter holds when it raises is not otherwise observable – a histogram that raises is replaced by a NullHistogram carrying the message.

A batch containing BOTH a value of an unusable type and enough new values to trip the limit reports the type failure, wherever the two sit relative to each other; the per-record path reports whichever comes first. The distinction is unreachable through the scan, which batches a str score’s column – every cell of which is a str or the None this skips.

add_value(value: str | int | None, count: int = 1) → None[source]

Add a value to the categorical histogram.

Returns true if successfully added and false if failed. Will fail if too many values are accumulated.

static deserialize(content: str) → CategoricalHistogram[source]: Rebuild a categorical histogram from serialize() output.

property display_values: dict[str | int, int]

Return categorical histogram display values in order.

A truncated instance carries exactly the values its config selected for display at serialization time, so they are returned verbatim – re-running the selection against the truncated counter would compute percentages and orderings against the wrong totals.

static from_dict(data: dict[str, Any]) → CategoricalHistogram[source]

Build a categorical histogram from a dict.

Reads both forms to_dict() and serialize_truncated() produce: the truncation flag and the two totals are absent from a full histogram and default accordingly.

merge(other: Statistic) → None[source]: Merge with other histogram.

plot(outfile: IO, score_id: str, *, y_axis_label: str | None = None, small_values_description: str | None = None, large_values_description: str | None = None) → None[source]: Plot histogram and save it into outfile.

property raw_values: dict[str | int, int]

Every counted value with its count, unordered.

This is the histogram’s own content, as distinct from display_values, which is the ordered subset the summary page draws. A custom plot_function receives the histogram itself and so can read either.

Note that on a histogram loaded back from a truncated sidecar (serialize_truncated()) the counter holds only the values that sidecar carried, so this returns the truncated set – unique_values and total_count are the ones that still describe the full data.

serialize() → str[source]: Render the full histogram as the JSON stored in the resource.

serialize_truncated() → str[source]

Serialize the truncated sidecar form of this histogram.

The sidecar carries the config, the values this histogram’s config selects for display, and the unique_values/total_count totals of the full histogram, marked with "truncated": true. It is the small, always-readable companion of a full histogram whose values file may be absent from a checkout (DVC-tracked, not pulled).

The carried values follow the display selection so the sidecar renders what the full histogram would: every value_order key with its real count for an ordered config, the values covering displayed_values_percent for a percent config, and the displayed_values_count most common values otherwise.

to_dict() → dict[str, Any][source]

Render this histogram as the mapping from_dict() reads back.

Carries every counted value, not the displayed subset; the truncated companion form is serialize_truncated().

property total_count: int

Sum of all value counts, across truncation.

On a truncated instance this is the total of the full histogram the sidecar was derived from, not the sum of the top-N counter it carries.

type = 'categorical_histogram'

property unique_values: int

Number of distinct values, across truncation.

On a truncated instance this is the distinct-value count of the full histogram the sidecar was derived from, not the length of the top-N counter it carries.

values_domain() → str[source]: Render the displayed values, noting truncation when present.

class gain.genomic_resources.histogram.CategoricalHistogramConfig(displayed_values_count: int | None = 20, displayed_values_percent: float | None = None, value_order: list[str | int] | None = None, y_log_scale: bool = False, label_rotation: int = 0, plot_function: str | None = None, enforce_type: bool = True, natural_order: bool = False, allow_only_whole_values_y: bool = False)[source]

Bases: object

Configuration class for categorical histograms.

allow_only_whole_values_y: bool = False

static default_config() → CategoricalHistogramConfig[source]

The config for a score that declares no categorical histogram.

enforce_type=False here does NOT relax value-type checking – CategoricalHistogram.add_value() refuses a non-str/int value either way. What the flag gates is CategoricalHistogram.UNIQUE_VALUES_LIMIT, and it reads backwards: it is the default config that enforces the limit, so a score nobody declared categorical cannot silently histogram thousands of distinct values. A curator who writes an explicit categorical block has asserted the score really is categorical, and that config carries no limit.

displayed_values_count: int | None = 20

displayed_values_percent: float | None = None

enforce_type: bool = True

static from_dict(parsed: dict[str, Any]) → CategoricalHistogramConfig[source]: Create categorical histogram config from configuratin dict.

label_rotation: int = 0

natural_order: bool = False

plot_function: str | None = None

to_dict() → dict[str, Any][source]: Transform categorical histogram config to dict.

value_order: list[str | int] | None = None

y_log_scale: bool = False

class gain.genomic_resources.histogram.HistogramStatisticMixin[source]

Bases: object

Mixin for creating statistics classes with histograms.

static get_histogram_file(score_id: str) → str[source]

static get_histogram_image_file(score_id: str) → str[source]

gain.genomic_resources.histogram.NUMBER_HISTOGRAM_VALUE_TYPES = ('float', 'int', 'bool')

Which score value types a NUMBER histogram can accumulate, one value at a time. bool is in: numpy folds it as 0/1, and a two-bin histogram over a flag is meaningful. str is not, and a resource pairing the two aborted its entire statistics build in np.isnan (gain#1285); since gain#1336 that pairing is refused when the score is CONSTRUCTED, by refuse_unfoldable_histograms(), which is the only thing that reads this set.

It lives here, with the config whose acceptance it describes, rather than in the statistics scan that used to own it – the refusal moved to the score layer, and the score layer must not import the scan to ask what a histogram can fold.

Deliberately WIDER than the scan’s _BULK_HISTOGRAM_VALUE_TYPES, and the two are different questions rather than one rule stated twice: this asks what a histogram can fold value by value, that asks what it can fold a whole column of (a bulk read yields a number histogram’s column as float64, which a bool score’s column is not). Merging them would widen the vectorized path to a type it cannot read.

class gain.genomic_resources.histogram.NullHistogram(config: NullHistogramConfig | None)[source]

Bases: Statistic

Class for annulled histograms.

add_value(value: Any, count: int = 1) → None[source]

Discard the value.

A null histogram counts nothing by design, so that a statistics build can feed every score the same way without first asking whether this one has a histogram.

static deserialize(content: str) → NullHistogram[source]: Rebuild a null histogram from serialize() output.

static from_dict(data: dict[str, Any]) → NullHistogram[source]: Build a null histogram from a dict.

merge(other: Any) → None[source]

Do nothing: there are no counts to fold together.

Merging is a no-op rather than an error so that a parallel build can reduce its partial results uniformly, null histograms included.

plot(_outfile: IO, _score_id: str) → None[source]

Draw nothing, leaving outfile untouched.

The caller is expected to render reason in place of the image, rather than to link an image file this never wrote.

serialize() → str[source]: Render this histogram as the JSON stored in the resource.

to_dict() → dict[str, Any][source]

Render this histogram as the mapping from_dict() reads back.

Only the config survives a round trip, because the reason is the whole of a null histogram’s content.

type = 'null_histogram'

values_domain() → str[source]: Report that there is no domain, in the other kinds’ place.

class gain.genomic_resources.histogram.NullHistogramConfig(reason: str)[source]

Bases: object

Configuration class for null histograms.

static default_config() → NullHistogramConfig[source]

A null config for a caller with no reason of its own to give.

reason is required rather than optional, so that every null histogram on a summary page can say why it is null; this is the placeholder for the few call sites that genuinely have nothing to add.

static from_dict(parsed: dict[str, Any]) → NullHistogramConfig[source]: Create Null histogram from configuration dict.

reason: str

to_dict() → dict[str, Any][source]: Render this config as the mapping from_dict reads back.

class gain.genomic_resources.histogram.NumberHistogram(config: NumberHistogramConfig, bins: ndarray | None = None, bars: ndarray | None = None)[source]

Bases: Statistic

Class to represent a histogram.

add_batch(values: ndarray, weights: ndarray) → None[source]

Add a batch of (value, weight) pairs, vectorized.

Bit-for-bit equivalent to calling add_value() over each pair in order: the same bin selection for both x-scales (the truncation of choose_bin_lin / choose_bin_log and the clamp to number_of_bins - 1), the same below/above out_of_range_bins split, the same min_value/max_value tracking, and the same nan-skip. This is the hot path for the statistics scan, where a per-value Python call dominates the cost.

The equivalence covers dtype as well as arithmetic: the float64 coercion below is what add_value reproduces by normalizing a numpy scalar, so a narrow column – float32, float16, bool – folds the same through either arm (gain#1338).

Where it does not hold: this arm also folds values add_value refuses outright, because asarray converts them silently – a complex loses its imaginary part, and an object array of Decimal converts. Nothing in a scan produces either; they are reachable only by calling this directly.

Both x-scales are vectorized. The log one is bit-exact for the same reason the linear one is and one more: np.log10 returns the same float for a scalar as for that scalar inside an array, at every array width numpy dispatches differently on – checked over many decades by test_add_batch_matches_add_value_loop_log_fuzz, which varies batch size precisely to cover numpy’s scalar-loop and SIMD kernels.

add_value(value: float | generic | None, count: int = 1) → None[source]

Add value to the histogram.

np.generic is in the signature because a numpy scalar is a real caller’s value, not a curiosity (gain#1338).

choose_bin_lin(value: float) → int[source]: Compute bin index for a passed value for linear x-scale.

choose_bin_log(value: float) → int[source]: Compute bin index for a passed value for log x-scale.

static deserialize(content: str) → NumberHistogram[source]: Rebuild a number histogram from serialize() output.

static from_dict(data: dict[str, Any]) → NumberHistogram[source]: Build a number histogram from a dict.

max_value: float

merge(other: Statistic) → None[source]: Merge two histograms.

min_value: float

out_of_range_bins: list[int]

out_of_range_values: list[float]

plot(outfile: IO, score_id: str, y_axis_label: str | None = None, small_values_description: str | None = None, large_values_description: str | None = None) → None[source]: Plot histogram and save it into outfile.

serialize() → str[source]: Render this histogram as the JSON stored in the resource.

to_dict() → dict[str, Any][source]

Render this histogram as the mapping from_dict() reads back.

Bin edges and bar counts are carried as plain lists rather than arrays, so the result is JSON-serialisable as it stands.

type = 'number_histogram'

values_domain() → str[source]

The observed value range, rendered for the summary page.

Unlike view_min() / view_max() this reports the values actually seen, so a score whose configured view range is wider than its data still shows the narrower true extent.

view_max() → float[source]

The high edge of the last bin.

The counterpart of view_min(); values above it are counted as out of range.

view_min() → float[source]

The low edge of the first bin.

This is the histogram’s view range, not the score’s observed minimum: values below it are counted in out_of_range_bins rather than binned.

view_range: tuple[float, float]

class gain.genomic_resources.histogram.NumberHistogramConfig(view_range: tuple[float | None, float | None], number_of_bins: int = 100, x_log_scale: bool = False, y_log_scale: bool = False, x_min_log: float | None = None, plot_function: str | None = None)[source]

Bases: object

Configuration class for number histograms.

static default_config(min_max: MinMaxValue | None) → NumberHistogramConfig[source]: Build a number histogram config from a parsed yaml file.

static from_dict(parsed: dict[str, Any]) → NumberHistogramConfig[source]: Build a number histogram config from a parsed yaml file.

has_view_range() → bool[source]

Whether both ends of the view range are pinned.

A histogram can only be built once its bin edges are known, so a config whose range is half-open still needs the score’s min/max before it can be used.

number_of_bins: int = 100

plot_function: str | None = None

to_dict() → dict[str, Any][source]: Transform number histogram config to dict.

view_range: tuple[float | None, float | None]

x_log_scale: bool = False

x_min_log: float | None = None

y_log_scale: bool = False

gain.genomic_resources.histogram.build_default_histogram_conf(value_type: str, **kwargs: Any) → NumberHistogramConfig | CategoricalHistogramConfig | NullHistogramConfig[source]: Build default histogram config for given value type.

gain.genomic_resources.histogram.build_empty_histogram(config: NullHistogramConfig | CategoricalHistogramConfig | NumberHistogramConfig) → NumberHistogram | CategoricalHistogram | NullHistogram[source]: Create an empty histogram from a deserialize histogram dictionary.

gain.genomic_resources.histogram.build_histogram_config(config: dict[str, Any] | None) → NullHistogramConfig | CategoricalHistogramConfig | NumberHistogramConfig | None[source]: Create histogram config form configuration dict.

gain.genomic_resources.histogram.load_histogram(resource: GenomicResource, filename: str) → NullHistogram | CategoricalHistogram | NumberHistogram[source]

Load and return a histogram in a resource.

On an error or missing histogram, an appropriate NullHistogram is returned.

gain.genomic_resources.histogram.plot_histogram(res: GenomicResource, image_filename: str, hist: NullHistogram | CategoricalHistogram | NumberHistogram, score_id: str, small_values_desc: str | None = None, large_values_desc: str | None = None) → None[source]: Plot histogram and save it into the resource.

gain.genomic_resources.histogram.save_histogram(resource: GenomicResource, filename: str, histogram: NullHistogram | CategoricalHistogram | NumberHistogram) → None[source]: Save histogram into a resource.

gain.genomic_resources.histogram.truncated_histogram_filename(histogram_filename: str) → str[source]

Return the truncated-sidecar filename for a histogram filename.

statistics/histogram_cell.json maps to statistics/truncated/histogram_cell.json: a directory rather than a name suffix, because the score-id part of the histogram filename is arbitrary (a score may itself be named foo_truncated) while no score id can contain /.

gain.genomic_resources.liftover_chain module

Provides LiftOver chain resource.

class gain.genomic_resources.liftover_chain.LiftoverChain(resource: GenomicResource)[source]

Bases: ResourceConfigValidationMixin

Defines Lift Over chain wrapper around pyliftover objects.

close() → None[source]

convert_coordinate(chrom: str, pos: int) → tuple[str, int, str, int] | None[source]: Lift over a genomic coordinate.

property files: set[str]

static get_schema() → dict[str, Any][source]: Return schema to be used for config validation.

is_open() → bool[source]

static map_chromosome(chrom: str, mapping: dict[str, str] | None) → str[source]: Map a chromosome (contig) name according to configuration.

open() → LiftoverChain[source]: Open the liftover chain resource.

gain.genomic_resources.liftover_chain.build_liftover_chain_from_resource(resource: GenomicResource) → LiftoverChain[source]: Load a Lift Over chain from GRR resource.

gain.genomic_resources.liftover_chain.build_liftover_chain_from_resource_id(resource_id: str, grr: GenomicResourceRepo | None = None) → LiftoverChain[source]

gain.genomic_resources.reference_genome module

class gain.genomic_resources.reference_genome.ReferenceGenome(resource: GenomicResource)[source]

Bases: ResourceConfigValidationMixin

Provides an interface for quering a reference genome.

property chrom_prefix: str: Return a prefix of all chromosomes of the reference genome.

property chromosomes: list[str]: Return a list of all chromosomes of the reference genome.

close() → None[source]: Close reference genome sequence file-like objects.

fetch(chrom: str, start: int, stop: int | None, buffer_size: int = 512) → Generator[str, None, None][source]

Yield the nucleotides in a specific region.

While line feed calculation can be inaccurate because not every fetch will start at the start of a line, line feeds add extra characters to read and the output is limited by the amount of nucleotides expected to be read.

get_all_chrom_lengths() → dict[str, int][source]: Return list of all chromosomes lengths.

get_chrom_length(chrom: str) → int[source]: Return the length of a specified chromosome.

static get_schema() → dict[str, Any][source]

The schema a genome resource’s config is checked against.

The base resource schema plus filename, index_file, chrom_prefix and PARS.

get_sequence(chrom: str, start: int, stop: int) → str[source]: Return sequence of nucleotides from specified chromosome region.

is_open() → bool[source]: Whether open() has run and close() has not since.

is_pseudoautosomal(chrom: str, pos: int) → bool[source]: Return true if specified position is pseudoautosomal.

open() → ReferenceGenome[source]: Open reference genome resources.

property resource_id: str: The id of the genome resource this object wraps.

split_into_regions(region_size: int, chromosome: str | None = None) → Generator[Region, None, None][source]

Split the reference genome into regions and yield them.

Can specify a specific chromosome to limit the regions to be in that chromosome only.

gain.genomic_resources.reference_genome.build_reference_genome_from_file(filename: str) → ReferenceGenome[source]: Open a reference genome from a file.

gain.genomic_resources.reference_genome.build_reference_genome_from_resource(resource: GenomicResource) → ReferenceGenome[source]: Open a reference genome from resource.

gain.genomic_resources.reference_genome.build_reference_genome_from_resource_id(resource_id: str, grr: GenomicResourceRepo | None = None) → ReferenceGenome[source]

gain.genomic_resources.reference_genome.genome_index_file(config: dict[str, Any]) → str[source]

Return the FASTA .fai index name for a genome config.

The optional index_file key overrides the default <filename>.fai. The three consumers that read a genome resource from its config – reference_genome_files(), the index parsing and the sequence backend – resolve the name through here, so that a resource is listed, hashed, parsed and opened against one and the same index.

Resolution is by name only: the file is not checked for existence and is not looked up in the resource manifest.

gain.genomic_resources.reference_genome.reference_genome_files(config: dict[str, Any]) → set[str][source]

Return all files a reference-genome resource consists of.

The set always contains the genome FASTA and its .fai index (honoring the optional index_file config key). For a bgzipped genome (filename ending in .gz/.bgz, in any case) htslib random access also needs the .gzi BGZF index, so it is included as well.

gain.genomic_resources.repository module

Provides basic classes for genomic resources and repositories.

This module defines the core architecture for managing genomic resources through a flexible repository system. It supports different storage backends (local files, HTTP, S3) and provides both read-only and read-write access.

Class Hierarchy:

      +---------------------+                    +-----------------+
+-----| GenomicResourceRepo |--------------------| GenomicResource |
|     +---------------------+                    +-----------------+
|        ^               ^                                    |
|        |               |                                    |
|        |  +-----------------------------+     +----------------------------+
|        |  | GenomicResourceProtocolRepo | ----| ReadOnlyRepositoryProtocol |
|        |  +-----------------------------+     +----------------------------+
|        |                                                    ^
|        |                                                    |
|    +--------------------------+            +-----------------------------+
+----| GenomicResourceGroupRepo |            | ReadWriteRepositoryProtocol |
     +--------------------------+            +-----------------------------+

Key Concepts:

GenomicResource: Represents a single genomic resource (e.g., a reference genome, score set, or gene model) with metadata and file access methods.
GenomicResourceRepo: Abstract base for repositories that manage collections of genomic resources.
RepositoryProtocol: Defines the storage backend interface (file system, HTTP, S3, etc.) for accessing resource files.
Manifest: Tracks files and their checksums within a resource to ensure data integrity and enable caching.

Resource Identifiers:

Resources are identified by an ID and optional version suffix:

Simple: “hg19/gene_models/refseq”
Versioned: “hg19/gene_models/refseq(1.2.3)”

Configuration Files:

Each resource contains a genomic_resource.yaml configuration file with metadata including type, description, and resource-specific settings.

gain.genomic_resources.repository.GR_GENERATED_INFO_PAGES = frozenset({'index.html', 'statistics/index.html'}): The pages grr_manage resource-info writes into a resource. They are regenerated on every run, so they are build artefacts rather than resource data and are never manifested – whether or not DVC manages them (#373).

gain.genomic_resources.repository.GR_INDEX_NON_LABEL_COLUMNS = frozenset({'description', 'full_id', 'id', 'score_descriptions', 'score_ids', 'summary', 'type'})

Index columns that describe the resource rather than one of its meta.labels entries. A label query names a label, so a clause on one of these must not be answered out of the column that shares its name – and no resource can carry them as labels anyway, because the index build refuses a label key repeating any of these names, whatever fields the resource’s own implementation contributes (gain#542).

An implementation that contributes a further field of its own belongs here too; the index cannot tell on its own which of its columns came from a label. Registering it here is what both refuses it as a label key and keeps a clause naming it off that column.

gain.genomic_resources.repository.GR_INDEX_RESOURCE_FIELDS = ('full_id', 'id', 'type', 'description', 'summary'): The columns every resource contributes to the FTS index, in the order ResourceImplementation.collect_index_info emits them.

gain.genomic_resources.repository.GR_INDEX_SCORE_FIELDS = ('score_ids', 'score_descriptions'): The columns a score implementation contributes on top of those.

gain.genomic_resources.repository.GR_STATISTICS_INDEX_FILE_NAME = 'statistics/index.html': The path grr_manage resource-info writes the statistics page to; named here so the writer and the exclusion below cannot drift (#373).

class gain.genomic_resources.repository.GenomicResource(resource_id: str, version: tuple[int, ...], protocol: ReadOnlyRepositoryProtocol | ReadWriteRepositoryProtocol, config: dict[str, Any] | None = None, manifest: Manifest | None = None)[source]

Bases: object

Represents a single genomic resource with metadata and file access.

A genomic resource is a versioned collection of data files with a configuration file (genomic_resource.yaml) that defines its type, description, and resource-specific settings.

Common resource types include:

genome: Reference genome sequences
gene_models: Gene annotations and transcript models
position_score: Position-based genomic scores
allele_score: Variant effect scores
gene_score: Gene-level scores

Variables:

resource_id – Unique identifier like “hg19/gene_models/refseq”
version (tuple[int, ...]) – Version tuple like (1, 2, 3)
config – Configuration dictionary from genomic_resource.yaml
proto – Repository protocol for accessing resource files

file_exists(filename: str) → bool[source]: Check if filename exists in this resource.

get_config() → dict[str, Any][source]

Return the resource configuration.

Raises ValueError if the resource carries no config. The return type is not optional and this never returns None, so a caller has nothing to re-check (gain#1010).

get_description() → str[source]: Return resource description.

get_file_content(filename: str, *, uncompress: bool = True, mode: str = 't') → Any[source]: Return the content of file in a resource.

get_file_url(filename: str) → str[source]

The URL of filename in this resource, per its protocol.

A filesystem path for a directory repository, an http(s):// or s3:// URL otherwise. The name is validated; the file need not exist.

get_full_id() → str[source]

Return a string combining resource ID and version.

Returns a string of the form aa/bb/cc(3.2) for a genomic resource with id aa/bb/cc and version 3.2. If the version is 0 the string will be aa/bb/cc.

This is also the resource’s path component under a repository’s url: a protocol addresses the resource’s directory by joining this string onto the repository root, so the suffix is part of where a versioned resource is stored, not merely of how it is displayed.

get_id() → str[source]: Return genomic resource ID.

get_labels() → dict[str, Any][source]

Return resource labels.

meta and meta.labels are both free-form YAML, so what is in either is whatever the curator wrote – a scalar, a list and an int are all things a resource can declare, and only the resource types that run the base schema are refused for it. Both levels are narrowed rather than trusted: a non-mapping reads as no labels and is reported, so that every caller sees a mapping whatever the resource says (gain#654). The outer level is narrowed by get_meta(), which every meta reader shares (gain#1004); this is the inner half of the two-tier narrowing described there, and it holds to the same never-validates, never-raises contract.

The values are returned as written. How a value is read – a list as a set of alternatives, everything else as its str() – is label_alternatives().

get_loaded_manifest() → Manifest | None[source]

Return the resource manifest without ever building one.

get_manifest() falls back to building the manifest on a read-write protocol – an md5 scan of every byte of the resource that also writes .grr/*.state files, and that fails outright on a read-only GRR mount. A pure read path that merely wants to consult the manifest uses this instead and copes with None.

get_manifest() → Manifest[source]: Load resource manifest if it exists. Otherwise builds it.

get_memo_key() → tuple[str, str, str][source]

Return a key identifying everything this resource denotes.

For memoising an object built from a resource – gene models, a gene score, a gene set collection, a liftover chain. Two resources that would build different objects have different keys.

The config is part of the key because a resource at the repository root, spelled ".", takes its whole meaning from it: its id and repository url alone do not separate it from another root resource over the same directory. The repository url is part of the key too, so the same resource reached through two repositories is memoised once per repository.

The key is finer than value identity, never coarser, so a miss costs a rebuild rather than a wrong answer. Raises ValueError if the resource carries no config.

get_meta() → dict[str, Any][source]

Return the resource’s meta block, narrowed to a mapping.

meta is free-form YAML, so what is there is whatever the curator wrote – meta: | followed by prose parses to a string, and only the resource types that run the base schema are refused for it. It is narrowed rather than trusted, so that every reader of the block sees a mapping whatever the resource says: a non-mapping reads as absent metadata and is reported.

The narrowing is SHALLOW – it promises a mapping at the top level and says nothing about what any field inside holds, which is just as free-form. A reader of a field narrows that field itself, the way get_labels() narrows meta.labels on top of this and get_description settles for str() on whatever it finds.

This is the single seam through which the meta block is read – by get_description, get_summary, get_labels and the FTS index-row collector alike – because narrowing it in one reader and not the others is what gain#1004 was: get_labels coped while the two beside it raised a bare AttributeError, which aborted the repository-wide index build outright.

Narrowing the block is not on its own enough to make every derived value agree. The index row used to collect description and summary out of this mapping with its own .get calls, so it missed the description fall-back get_summary() applies and a resource carrying a description and no summary indexed an empty summary column (gain#1008). Both now derive through _description_in() and _summary_in(), which are the single spelling of what each field is: a reader holding this block derives from those rather than reaching into it, so a second spelling cannot drift from the accessors again.

Reading never validates (ADR 0008) and never raises: this is on the path of every repository-wide walk – a label search, the index build, grr_manage list – and one malformed resource must cost that walk only itself (gain#464, gain#503).

get_public_url() → str[source]: Return this resource’s address on the GRR’s public mirror.

get_repo_public_url() → str[source]: Return repository’s URL.

get_repo_url() → str[source]: Return repository’s URL.

get_summary() → str | None[source]: Return resource summary.

get_type() → str[source]: Return resource type as defined in ‘genomic_resource.yaml’.

get_url() → str[source]: Return this resource’s address on the repository’s own url.

get_version_str() → str[source]: Return version string of the form ‘3.1’.

invalidate() → None[source]: Clean up cached attributes like manifest, etc.

open_bigwig_file(filename: str) → Any[source]: Open a bigwig file and return it.

open_fasta_file(filename: str, index_filename: str | None = None, compressed_index_filename: str | None = None) → FastaFile[source]: Open a bgzipped fasta file and return a pysam.FastaFile.

open_raw_file(filename: str, mode: str = 'rt', **kwargs: str | bool | None) → IO[source]: Open a file in the resource and returns a File-like object.

open_tabix_file(filename: str, index_filename: str | None = None) → TabixFile[source]: Open a tabix file and returns a pysam.TabixFile.

open_vcf_file(filename: str, index_filename: str | None = None) → VariantFile[source]: Open a vcf file and returns a pysam.VariantFile.

version: tuple[int, ...]

class gain.genomic_resources.repository.GenomicResourceProtocolRepo(proto: ReadOnlyRepositoryProtocol | ReadWriteRepositoryProtocol)[source]

Bases: GenomicResourceRepo

Base class for real genomic resources repositories.

close() → None[source]: Release any resources held by this repository.

find_resource(resource_id: str, version_constraint: str | None = None, repository_id: str | None = None) → GenomicResource | None[source]

Return one resource with id qual to resource_id.

If resource is not found, None is returned.

repository_id selects a repository by id under the same rule as get_resource() – a repository answers to its own id, and a falsy id is no filter.

get_all_resources() → Generator[GenomicResource, None, None][source]: Return a generator over all resource in the repository.

get_resource(resource_id: str, version_constraint: str | None = None, repository_id: str | None = None) → GenomicResource[source]

Return one resource with id qual to resource_id.

If resource is not found, exception is raised.

repository_id restricts the lookup to the repository carrying that id, anywhere in this repository’s tree – including this repository itself: every repository answers to its own id, so passing repo.repo_id is equivalent to passing nothing (#447). A falsy repository_id is no filter at all.

invalidate() → None[source]

Clear cached state and force reload on next access.

Implementations should clear any cached resource lists, metadata, or file contents to ensure fresh data is loaded.

search_resources(search_term: str | None = None, resource_type: str | None = None, resource_query: str | None = None) → Generator[GenomicResource, None, None][source]

Search resources by FTS term, type and/or wildcard query.

All supplied filters conjoin.

The generator’s return value carries the (repository id, reason) pairs of the children a group skipped while still answering (ADR 0012, gain#686); None and [] both mean nothing was skipped. A for loop discards it, which is exactly right for a caller that does not present totals.

class gain.genomic_resources.repository.GenomicResourceRepo(repo_id: str)[source]

Bases: ABC

Abstract base class for genomic resource repositories.

A repository manages a collection of genomic resources, providing methods to discover, retrieve, and (for writable repos) create resources.

Repositories can be:

Protocol-based: Direct access to a single storage backend
Group: Aggregates multiple child repositories
Cached: Wraps another repository with local caching

All repositories support resource lookup with optional version constraints:

repo.get_resource(“hg19/genome”) # Latest version repo.get_resource(“hg19/genome”, “>=2.0”) # Version 2.0 or higher repo.get_resource(“hg19/genome”, “=2.1”) # Exact version 2.1

Variables:

repo_id – Unique identifier for this repository
definition – Configuration dict used to create this repository

close() → None[source]: Release any resources held by this repository.

property definition: dict[str, Any] | None

Get a copy of the repository configuration definition.

Returns:: Deep copy of definition dict, or None if not set

abstractmethod find_resource(resource_id: str, version_constraint: str | None = None, repository_id: str | None = None) → GenomicResource | None[source]

Return one resource with id qual to resource_id.

If resource is not found, None is returned.

repository_id selects a repository by id under the same rule as get_resource() – a repository answers to its own id, and a falsy id is no filter.

abstractmethod get_all_resources() → Generator[GenomicResource, None, None][source]: Return a generator over all resource in the repository.

abstractmethod get_resource(resource_id: str, version_constraint: str | None = None, repository_id: str | None = None) → GenomicResource[source]

Return one resource with id qual to resource_id.

If resource is not found, exception is raised.

repository_id restricts the lookup to the repository carrying that id, anywhere in this repository’s tree – including this repository itself: every repository answers to its own id, so passing repo.repo_id is equivalent to passing nothing (#447). A falsy repository_id is no filter at all.

abstractmethod invalidate() → None[source]

Clear cached state and force reload on next access.

Implementations should clear any cached resource lists, metadata, or file contents to ensure fresh data is loaded.

property repo_id: str

Get the repository identifier.

Returns:: Repository ID string

abstractmethod search_resources(search_term: str | None = None, resource_type: str | None = None, resource_query: str | None = None) → Generator[GenomicResource, None, list[tuple[str, str]] | None][source]

Search resources by FTS term, type and/or wildcard query.

All supplied filters conjoin.

The generator’s return value carries the (repository id, reason) pairs of the children a group skipped while still answering (ADR 0012, gain#686); None and [] both mean nothing was skipped. A for loop discards it, which is exactly right for a caller that does not present totals.

search_resources_by_child(search_term: str | None = None, resource_type: str | None = None, resource_query: str | None = None) → Generator[tuple[GenomicResourceRepo, GenomicResource], None, list[tuple[str, str]] | None][source]

Search, pairing each hit with the repository that serves it.

For a repository that serves resources itself the answer is always this one, which is what this implementation says. A group overrides it to name the child the resource actually came from, so a caller that has to label a hit – grr_manage list prints the id beside every row – does not have to take a group apart to find out.

The filters mean exactly what they mean for search_resources(), which is the projection of this – and the return value carries the same skips.

gain.genomic_resources.repository.INDEX_COLUMN_PATTERN = '[A-Za-z_][A-Za-z0-9_]*': What an FTS index column may be named. Every name the index build creates is vetted against this before it becomes a column, because a column name cannot be bound as a parameter and so has to be spliced into SQL (gain#464).

class gain.genomic_resources.repository.Manifest[source]

Bases: object

Manages file listings and checksums for a genomic resource.

A manifest maintains a catalog of all files in a resource with their sizes and MD5 checksums. This enables data integrity verification, efficient caching, and incremental updates.

The manifest is typically stored in a .MANIFEST file within the resource directory and is automatically loaded when accessing the resource.

add(entry: ManifestEntry) → None[source]

Add or update a manifest entry.

Parameters:: entry – ManifestEntry to add to the manifest

entries: dict[str, ManifestEntry]

static from_file_content(file_content: str) → Manifest[source]

Create a manifest from raw YAML file content.

Parameters:: file_content – YAML-formatted string containing manifest entries
Returns:: Manifest object with entries parsed from the content

static from_manifest_entries(manifest_entries: list[dict[str, Any]]) → Manifest[source]

Create a manifest from parsed manifest entry dictionaries.

Parameters:: manifest_entries – List of dicts with ‘name’, ‘size’, ‘md5’ keys
Returns:: Manifest object populated with the provided entries

get_files() → list[tuple[str, int]][source]

Get list of all files with their sizes.

Returns:: List of (filename, size) tuples for all files in manifest

names() → set[str][source]

Get set of all filenames in the manifest.

Returns:: Set of filenames tracked by this manifest

to_manifest_entries() → list[dict[str, Any]][source]

Convert manifest to list of dictionaries for serialization.

Returns:: List of dictionaries with ‘name’, ‘size’, ‘md5’ keys, sorted by filename

update(entries: dict[str, ManifestEntry]) → None[source]

Add or update multiple manifest entries.

Parameters:: entries – Dictionary mapping filenames to ManifestEntry objects

class gain.genomic_resources.repository.ManifestEntry(name: str, size: int, md5: str | None)[source]

Bases: object

Represents a file entry in a genomic resource manifest.

A manifest tracks all files within a resource with their sizes and checksums to ensure data integrity and enable efficient caching.

Variables:

name (str) – Relative path to the file within the resource
size (int) – File size in bytes
md5 (str | None) – MD5 checksum of file content, or None if not computed

md5: str | None

name: str

size: int

class gain.genomic_resources.repository.ManifestUpdate(manifest: Manifest, entries_to_delete: set[str], entries_to_update: set[str])[source]

Bases: object

Represents a set of changes to apply to a manifest.

Used during resource synchronization to track which files need to be deleted or updated.

Variables:

manifest (gain.genomic_resources.repository.Manifest) – The updated manifest with all changes applied
entries_to_delete (set[str]) – Set of filenames to remove
entries_to_update (set[str]) – Set of filenames that need updating

entries_to_delete: set[str]

entries_to_update: set[str]

manifest: Manifest

class gain.genomic_resources.repository.Mode(*values)[source]

Bases: Enum

Enumeration of repository protocol access modes.

Variables:

READONLY – Protocol supports only read operations
READWRITE – Protocol supports both read and write operations

READONLY = 1

READWRITE = 2

gain.genomic_resources.repository.RESOURCE_ID_CHARACTER_CLASS = 'a-zA-Z0-9/._-': Every character a resource id may be spelled with, as the body of a regex character class. One definition, composed into the pattern that accepts an id, the one that names what a malformed one carries and the one that accepts a single segment, so the three cannot drift apart – they used to be written out separately, and a rule whose single source of truth is a test is a rule waiting to disagree with itself (gain#1352). - stays last: anywhere else it would read as a range.

class gain.genomic_resources.repository.ReadOnlyRepositoryProtocol(proto_id: str, url: str)[source]

Bases: ABC

Abstract base class for read-only repository storage protocols.

A protocol defines how to access genomic resources from a specific storage backend (local filesystem, HTTP server, S3 bucket, etc.). Read-only protocols can retrieve resources but cannot modify them.

Subclasses must implement methods for:

Listing available resources
Reading configuration files
Opening resource files
Loading manifests

Variables:

proto_id – Unique identifier for this protocol instance
url – Base URL or path to the repository root
CHUNK_SIZE – Default read-buffer size for chunked file operations (1 MiB). This is the application-level read size for the download and md5 loops, not the network transfer unit – fsspec does its own block-level fetching/buffering underneath. Larger chunks cut Python-loop and progress-callback overhead on multi-GB resources.

CHUNK_SIZE = 1048576

build_genomic_resource(resource_id: str, version: tuple[int, ...], config: dict | None = None, manifest: Manifest | None = None) → GenomicResource[source]

Build a genomic resource instance using this protocol.

Parameters:

resource_id – Resource identifier like “hg19/gene_models/refseq”
version – Version tuple like (1, 2, 3)
config – Optional pre-loaded configuration dict. If None, will load from genomic_resource.yaml
manifest – Optional pre-loaded manifest. If None, will load when first accessed

Returns:

GenomicResource instance configured with this protocol

compute_md5_sum(resource: GenomicResource, filename: str) → str[source]: Compute a md5 hash for a file in the resource.

abstractmethod file_exists(resource: GenomicResource, filename: str) → bool[source]: Check if given file exist in give resource.

find_resource(resource_id: str, version_constraint: str | None = None) → GenomicResource | None[source]: Return requested resource or None if not found.

abstractmethod get_all_resources() → Generator[GenomicResource, None, None][source]: Return generator for all resources in the repository.

abstractmethod get_all_resources_dict() → dict[str, GenomicResource][source]: Return dictionary for all resources in the repository.

get_file_content(resource: GenomicResource, filename: str, *, uncompress: bool = True, mode: str = 't') → Any[source]: Return content of a file in given resource.

get_id() → str[source]

Return the repository protocol identifier.

Returns:: Protocol ID string

get_manifest(resource: GenomicResource) → Manifest[source]: Load and returns a resource manifest.

abstractmethod get_public_url() → str[source]

Return the public base URL of the repository.

Returns:: URL or path string pointing to a public repository root

get_resource(resource_id: str, version_constraint: str | None = None) → GenomicResource[source]

Return requested resource or raises exception if not found.

In case resource is not found a FileNotFoundError exception is raised.

get_resource_file_url(resource: GenomicResource, filename: str) → str[source]: Return url of a file in the resource.

get_resource_url(resource: GenomicResource) → str[source]

Return url of the specified resources.

The resource id is the other operand of this join and is no less untrusted than a file name – on the remote path it is read verbatim out of the repository’s .CONTENTS.json.gz – so it is contained here, at the join, exactly as get_resource_file_url contains the name (gain#467).

abstractmethod get_url() → str[source]

Return the base URL of the repository.

Returns:: URL or path string pointing to repository root

abstractmethod invalidate() → None[source]: Invalidate internal cache of repository protocol.

abstractmethod load_manifest(resource: GenomicResource) → Manifest[source]: Load resource manifest.

load_yaml(resource: GenomicResource, filename: str) → Any[source]: Return parsed YAML file.

mode() → Mode[source]

Return repository protocol mode.

Returns:: Mode.READONLY for this base class

abstractmethod open_bigwig_file(resource: GenomicResource, filename: str) → Any[source]

Open a bigwig file in a resource and return it.

Not all repositories support this method. Repositories that do no support this method raise and exception.

open_fasta_file(resource: GenomicResource, filename: str, index_filename: str | None = None, compressed_index_filename: str | None = None) → FastaFile[source]

Open a bgzipped fasta file in a resource and return a FastaFile.

Not all repositories support this method. Repositories that do not support this method raise an exception.

abstractmethod open_raw_file(resource: GenomicResource, filename: str, mode: str = 'rt', **kwargs: str | bool | None) → IO[source]: Open file in a resource and returns a file-like object.

abstractmethod open_repository_metadata() → Connection[source]: Open the db file for repo metadata and return the connection.

abstractmethod open_tabix_file(resource: GenomicResource, filename: str, index_filename: str | None = None) → TabixFile[source]

Open a tabix file in a resource and return a pysam tabix file.

Not all repositories support this method. Repositories that do no support this method raise and exception.

abstractmethod open_vcf_file(resource: GenomicResource, filename: str, index_filename: str | None = None) → VariantFile[source]

Open a vcf file in a resource and return a pysam VariantFile.

Not all repositories support this method. Repositories that do no support this method raise and exception.

search_resources(search_term: str | None = None, resource_type: str | None = None, resource_query: str | None = None) → Generator[GenomicResource, None, None][source]

Search for resources using SQLite full-text search.

The three filters conjoin: a resource must satisfy every one that is supplied. A search_term is matched against the FTS index, and when one is supplied the resource_type and the id glob of the resource_query – the annotator wildcard language, an id glob plus an optional label query – join it in the same statement, so the index narrows once rather than handing rows to a filter.

Without a search_term the index is never opened: the type and the query are matched in Python, over every resource. That is what makes them work on a repository with no .CONTENTS.sqlite3.gz at all, where opening the metadata db raises. Only the term genuinely needs the index – FTS5 tokenisation is not reproducible in Python, while a type is one token every resource carries (gain#1212).

Both routes evaluate the same parsed query, and every label clause is answered against the resource’s own meta.labels rather than out of the value the index recorded (gain#646) – so the two agree on every resource the index knows about, whatever a published index that has fallen behind its resources says about their labels. What an index too old to name a resource at all cannot do is return it, and that much is inherent: only the index can answer a search_term. Rebuilding it with grr_manage is what makes a newly added resource findable.

An empty resource_query is an unset one: it is what a shell substitutes for a variable that was never set, and the useful reading of -q "$SELECTOR" with no selector is the one that behaves like omitting the flag. A blank search_term is unset for the same reason (gain#633), and normalising it here – ahead of the branch below that decides whether to open the index at all – is what keeps -s "" from demanding an index it has no filter to apply: "" is not None, so it used to reach MATCH and be rejected by FTS5, or, on a repository with no index, be reported as a search that cannot be run.

Whitespace counts as blank: -s "$VAR " is the same accident as -s "$VAR". What FTS5 would otherwise make of it depends on which space was typed, and neither answer is worth keeping – a run of ASCII spaces is the empty expression it rejects, while a non-breaking space is a term it accepts and nothing contains, so the one accident was an error or a silently empty result. A term that has content is passed on untouched, spaces and all – ref_genome : hg38 is one term.

resource_type is normalised the same way, for the same reason (gain#653): blank, it selected nothing where it was meant to select everything, and asked an index-less repository for an index to apply a filter nobody set.

Raises ResourceQueryParseError for a malformed resource_query – eagerly, when called, rather than on the first iteration, so a caller can still report it against the argument that caused it.

class gain.genomic_resources.repository.ReadWriteRepositoryProtocol(proto_id: str, url: str)[source]

Bases: ReadOnlyRepositoryProtocol

Abstract base class for read-write repository storage protocols.

Extends ReadOnlyRepositoryProtocol with write capabilities including:

Creating and updating resources
Managing manifests
File upload and deletion
Resource versioning

This protocol type is used for local repositories and writable remote storage backends where resources can be modified or created.

abstractmethod build_content_file() → list[dict[str, Any]][source]: Build the content of the repository (i.e ‘.CONTENTS.json.gz’).

build_manifest(resource: GenomicResource, prebuild_entries: dict[str, ManifestEntry] | None = None, *, verify_content: bool = False) → Manifest[source]: Build full manifest for the resource.

Build resource file state.

Each of md5, timestamp, size and change_token is read off the stored file when it is not supplied. Supplying a digest that is already in hand – the download path verifies one against the manifest before publishing the file – saves reading the whole file back out of the store, which for a remote store is a second transfer of it. Supplying the other three saves a stat each, and the download path has all of them from the stat it makes to verify its own move (gain#936).

change_token defaults to Unread.UNREAD rather than to None because None is a value a token legitimately takes: it is what a store offering no tokens reports, and it is the value a caller holding one such answer needs to be able to pass.

The parameters are keyword-only and named explicitly so that a misspelled one is a TypeError here rather than a silently ignored value – which for md5 costs a full re-read of the file, and for the others a stat. See gain#865.

A file that is not there is still refused with a ValueError naming the resource and the filename – but it is noticed by the reads rather than ahead of them. It is worth being plain about what the check that used to run first was: not a safeguard, but a message. Every read below raises FileNotFoundError of its own accord on a file that is not there; all the check added was that the caller hears the ValueError instead. A message can be phrased where the failure happens, and asking first cost a probe of a key the caller may have just been told about – the cache verdict opens by stating the very file whose state it then rebuilds, and paid for that twice (gain#1039). A caller that supplies every field reads nothing, so it raises nothing here, exactly as before.

check_update_manifest(resource: GenomicResource, prebuild_entries: dict[str, ManifestEntry] | None = None, *, verify_content: bool = False, save_state: bool = True) → ManifestUpdate[source]

Check if the resource manifest needs update.

With save_state=False nothing it derives is recorded (#257).

abstractmethod collect_all_resources() → Generator[GenomicResource, None, None][source]

Scan repository and yield all resources.

Returns:: Generator yielding GenomicResource instances for each resource found in the repository

collect_resource_entries(resource: GenomicResource) → Manifest[source]

Scan resource directory and build manifest from files found.

The entries-only view of scan_resource_entries(), for callers that just want the names and sizes on disk. A caller that WRITES a manifest must use the scan itself, so that a file it could not describe fails the resource instead of vanishing from the manifest (gain#503).

Parameters:: resource – Resource to scan
Returns:: Manifest containing entries for all files in the resource

copy_resource(remote_resource: GenomicResource) → GenomicResource[source]: Copy a remote resource into repository.

abstractmethod copy_resource_file(remote_resource: GenomicResource, dest_resource: GenomicResource, filename: str) → ResourceFileState | None[source]: Copy a remote resource file into local repository.

abstractmethod delete_resource_file(resource: GenomicResource, filename: str) → None[source]: Delete a resource file and it’s internal state.

get_manifest(resource: GenomicResource) → Manifest[source]: Load or build a resource manifest.

get_or_create_resource(resource_id: str, version: tuple[int, ...]) → GenomicResource[source]

Return a resource with specified ID and version.

If the resource is not found create an empty resource.

abstractmethod get_resource_file_change_token(resource: GenomicResource, filename: str) → str | None[source]

Return the store’s change token for a resource file, if any.

A change token is whatever the store itself offers as “this is the version of the object you are looking at”: it changes on every write and holds still for as long as the object is not written. Stores that offer none answer None, and for them the modification time remains the only change hint there is.

The value is opaque. It is never parsed, never compared against an md5 sum and never assumed to be one, even where a particular store happens to derive it from one.

See ADR 0022 for why a state is judged by this rather than by the modification time.

abstractmethod get_resource_file_size(resource: GenomicResource, filename: str) → int[source]: Return the size of a resource file.

abstractmethod get_resource_file_timestamp(resource: GenomicResource, filename: str) → float[source]: Return the timestamp (ISO formatted) of a resource file.

abstractmethod load_resource_file_state(resource: GenomicResource, filename: str) → ResourceFileState | None[source]

Load resource file state from internal GRR state.

If the specified resource file has no internal state returns None.

mode() → Mode[source]

Return repository protocol mode.

Returns:: Mode.READWRITE for this protocol type

abstractmethod publish_raw_file(resource: GenomicResource, filename: str, mode: str = 'wt') → AbstractContextManager[IO][source]

Open a resource file for a write that replaces it, or does not.

The publishing counterpart of open_raw_file(): what the caller writes reaches filename only once the handle has closed cleanly. A write that fails part-way – and an interrupt – leaves whatever was published before exactly as it was, rather than truncating it in place with nothing to roll back to (gain#933).

A separate name rather than a flag on open_raw_file(): the two differ in what they guarantee, not in a parameter, and the callers that need the guarantee are not the ones that need a plain handle.

save_index(resource: GenomicResource, contents: str) → None[source]: Save an index HTML file into the genomic resource’s directory.

save_manifest(resource: GenomicResource, manifest: Manifest) → None[source]: Save manifest into genomic resource’s directory.

abstractmethod save_resource_file_state(resource: GenomicResource, state: ResourceFileState) → None[source]: Save resource file state into internal GRR state.

abstractmethod scan_resource_entries(resource: GenomicResource) → ResourceScan[source]

Scan resource directory for its files.

Parameters:: resource – Resource to scan
Returns:: A ResourceScan holding the entries that could be described and the names of the files that could not.

update_manifest(resource: GenomicResource, prebuild_entries: dict[str, ManifestEntry] | None = None, *, verify_content: bool = False) → Manifest[source]: Update or create full manifest for the resource.

update_resource(remote_resource: GenomicResource, files_to_copy: set[str] | None = None) → GenomicResource[source]

Copy a remote resource into repository.

Allows copying of a subset of files from the resource via files_to_copy. If files_to_copy is None, copies all files.

abstractmethod update_resource_file(remote_resource: GenomicResource, dest_resource: GenomicResource, filename: str) → ResourceFileState | None[source]: Update a resource file into repository if needed.

class gain.genomic_resources.repository.ResourceFileState(filename: str, size: int, timestamp: float, md5: str, change_token: str | None = None)[source]

Bases: object

Tracks the state of a resource file in internal repository storage.

Used for caching and synchronization to determine if files need to be refreshed or re-downloaded.

Variables:

filename (str) – Relative path to the file within the resource
size (int) – File size in bytes
timestamp (float) – Last modification time as Unix timestamp
md5 (str) – MD5 checksum of file content
change_token (str | None) – Opaque token the store supplies for the stored object, or None where the store has none. It changes whenever the object changes, and nothing else about it is defined – it is never parsed, and never treated as a checksum, even when a particular store happens to derive it from one.

change_token: str | None = None

filename: str

md5: str

size: int

timestamp: float

class gain.genomic_resources.repository.ResourceScan(manifest: Manifest, unreadable: Mapping[str, str])[source]

Bases: object

What one scan of a resource’s directory found.

unreadable maps each file the scan listed but could not stat – a dangling symlink, a symlink loop, a directory it may not traverse – to why. They are NOT in manifest: there is no size to put there.

They are carried rather than raised because the scan cannot yet know whether they matter. A DVC-managed file materialised as a link into a shared cache is unreadable exactly when that cache has been garbage collected, and its .dvc sidecar still describes it perfectly – so it is manifested from the sidecar and nothing is wrong. Only a name that NOTHING can describe is a broken resource, and that is not known until the sidecars have been merged in (gain#503).

The reason travels with the name so that whoever DOES know the outcome can report it: a resource is scanned more than once per command, so the scan itself is the wrong place to say anything the user should see exactly once.

manifest: Manifest

unreadable: Mapping[str, str]

exception gain.genomic_resources.repository.SearchIndexUnavailableError(repo_id: str, reason: str)[source]

Bases: ValueError

A repository that cannot apply a search filter for want of an index.

Raised when the repository publishes no .CONTENTS.sqlite3.gz at all, and when the one it publishes carries no contents table because no resource could be indexed into it. Both are repository health: the filter is unobjectionable and a grr_manage repo-repair would let it be applied.

Typed rather than left as a bare ValueError because a group repository absorbs this to skip the child and carry on (ADR 0012), and the layers it absorbs it through raise ValueError of their own that must keep propagating – the cache layer resolving a resource it cannot place, for one.

A ValueError still, so a caller that only ever distinguished bad arguments from working ones is unaffected.

exception gain.genomic_resources.repository.SearchTermError(search_term: str, cause: Exception)[source]

Bases: ValueError

A search term that SQLite’s FTS5 could not parse as a match expression.

The term is bound, never interpolated, so this is not a failure to contain it – it is that FTS5 reads a bound term as an expression, with a grammar of its own: quotes, AND/OR/NOT, NEAR, column : value. A term that does not parse is the caller’s mistake, and apsw.SQLError reports it as neither – it names no term, no argument, and reads like the database broke (gain#632).

A ValueError, like ResourceQueryParseError, so the two bad arguments this search can be given are handled the same way by the endpoint and the CLI that take them.

The column filter is why the term is not simply quoted into a literal before it reaches MATCH: label keys are index columns, and ref_genome : hg38 is a supported search. Quoting would make that a search for the text.

class gain.genomic_resources.repository.Unread(*values)[source]

Bases: Enum

A keyword that was not supplied, where None is a real value.

An enum rather than a bare object() so that the sentinel has a type a signature can name and a type checker can narrow: comparing a parameter against the member leaves str | None in the other branch, which is what the field actually is. Public because it appears in the signature of a public method, and a subclass that overrides it has to be able to name it.

UNREAD = 1

Not supplied.

ReadWriteRepositoryProtocol.build_resource_file_state() reads a field off the stored file when the caller does not supply it, and for three of the four fields None says so unambiguously. A change token is the exception: None is the answer a store offering no tokens gives, so a caller that has asked and been told “no token” must be able to say that, and it must not read as “go and ask”.

exception gain.genomic_resources.repository.UnreadableResourceFilesError(resource_id: str, names: Sequence[str])[source]

Bases: ValueError

Files of ONE resource that could not be read or described.

Collected rather than raised on the first offender, so a single run reports every one of them. A ValueError so that cli_errors.report_resource_failure renders it as one line naming the resource and carrying the cause, with the traceback demoted to DEBUG (gain#364) – and so that one broken resource fails itself instead of aborting the repository-wide command (gain#503).

gain.genomic_resources.repository.collect_dvc_entries(proto: ReadWriteRepositoryProtocol, res: GenomicResource) → dict[str, ManifestEntry][source]

Collect manifest entries defined by .dvc files.

A .dvc file that cannot be read, does not parse as a pointer for the data file it sits next to, or declares no usable md5 sum and size is skipped with a warning - never propagated into the manifest, and never allowed to abort the command. .dvc sidecars are read on every grr_manage run, and the repository scan that produced this entry has already tolerated the very same content (see FsspecReadWriteProtocol._is_dvc_managed_leaf); the two classify identically because both delegate to dvc.parse_dvc_pointer_out().

A well-formed sidecar for a dvc add <dir> output is a different matter: it is not ignored, it is REFUSED. GAIn cannot verify a .dir md5 sum - it hashes a DVC cache object, not any file GAIn can read - so writing it into the manifest would be a false clean bill of health, and quietly skipping the directory would leave its data unmanifested and unverified. Either way the resource would be certified without its content ever being checked, so the command fails instead (#255). This is the gate every grr_manage subcommand that builds or checks a manifest passes through – and, since #721, the fallback build a repository walk triggers – and it applies whether or not the directory is materialised. It is kept even though cli_dvc.refuse_dvc_directory_outputs refuses such a resource before any command reaches this function: a manifest must never be built from a sidecar GAIn cannot verify, whoever asks for it (#284).

An entry is produced for every readable sidecar. Every materialised file’s entry is consulted by ReadWriteRepositoryProtocol._update_manifest_entry_and_state() - the sidecar IS the md5 sum of the file it describes - and the entries for files the scan did not yield are merged by ReadWriteRepositoryProtocol._merge_unscanned_dvc_entries() (#373).

Lives beside the manifest builder, not in the CLI, because the builder itself must reach it: a manifest built as a FALLBACK - a repository walk meeting a resource that never had a .MANIFEST - has no CLI frame above it to collect the sidecars, and building without them hashes every DVC-managed byte the sidecar already describes (#721).

Raises:: UnsupportedDvcDirectoryOutputError – the resource has a dvc add <dir> output.

gain.genomic_resources.repository.drain_search(hits: Generator[HitT, None, list[tuple[str, str]] | None]) → tuple[list[HitT], list[tuple[str, str]]][source]

Exhaust a search, answering its rows and its skips.

The consumption idiom for a caller that presents totals: a for loop silently discards the skips a group reports on its generator’s return value (gain#686). None and [] both arrive as [], so the dual spelling ends here.

gain.genomic_resources.repository.dvc_directory_output_message(resource_id: str, entry_name: str, filename: str) → str[source]

Say why a dvc add <dir> output is refused, and what to do.

One text for both gates – cli_dvc’s pre-flight and the manifest builder – so what the user is told cannot depend on which of them saw the sidecar first (#284).

Every name here is untrusted GRR content and this message is a refusal report, so the names are escaped for the same reason the sibling warnings in collect_dvc_entries() are (gain#642).

gain.genomic_resources.repository.is_generated_info_page(name: str) → bool[source]

Return True if name is a page resource-info generates.

Membership in a resource’s manifest is decided by the file’s PATH: the two pages GAIn writes itself are excluded, everything else is resource data. The rule it replaced – “drop every name ending in html” – was a proxy for the same question and a bad one, since it silently dropped any html file a resource legitimately carries as data (#373).

gain.genomic_resources.repository.is_gr_id_token(token: str) → bool[source]

Check if token can be used as a genomic resource ID.

Genomic Resource Id Token is a string with one or more letters, numbers, ‘.’, ‘_’, or ‘-’. The function checks if the parameter token is a Genomic REsource Id Token.

gain.genomic_resources.repository.is_safe_repo_id(repo_id: str) → bool[source]

Check if repo_id is usable as a single filesystem path segment.

A repository id names a directory: a cached repository derives each repository’s cache directory by joining the id onto the cache url. An id that is not a single path segment therefore decides where the process writes – .. climbs out of the configured cache directory, and an absolute id makes os.path.join discard the cache url altogether. Such an id is a configuration error and is rejected, never rewritten (#460).

Safe means exactly one non-empty path segment that still names that same segment after a round trip through a url. Each check earns its keep separately: the separator check rules out sub/dir, ../../escaped and an absolute /etc/grrcache – and, because a UNC prefix (\\server\share, //server/share, \\?\C:\x) always carries separators, those too; ntpath.splitdrive() adds the one absolute-ish prefix that has no separator in it, C:cache, which os.path.join on Windows resolves against that drive’s current directory; the control-character check covers the id that changes shape when it is parsed as part of a url (see _UNSAFE_NAME_CHARACTER_RE); and . and .. are spelled out because they are ordinary segments to every one of the checks above.

An empty id is not a segment either, but it is not a traversal: a falsy id already means “unnamed” everywhere it is read (_resolve_repo_id synthesises one, find_resource treats it as “no filter”), so its callers decide what to do with it rather than this predicate.

Deliberately NOT built on is_gr_id_token(). That helper enforces a character class ([a-zA-Z0-9._-]), which is both too weak and too strong here: is_gr_id_token("..") is True – .. matches the class in full, and a single-segment .. still escapes one directory level – while ids that are perfectly safe as directory names (a space, say) would start failing for a reason that has nothing to do with path safety. This check answers only the path question.

gain.genomic_resources.repository.is_version_constraint_satisfied(version_constraint: str | None, version: tuple[int, ...]) → bool[source]

Check if a version matches a version constraint.

Supports two types of constraints:

“=X.Y.Z”: Exact match required
“>=X.Y.Z” or “X.Y.Z”: Minimum version required (default)

Parameters:

version_constraint – Constraint string like “>=1.2.0” or “=1.2.3”. None or empty string matches any version.
version – Version tuple to check like (1, 2, 3)

Returns:

True if the version satisfies the constraint

Raises:

ValueError – If constraint has invalid syntax or unknown operator

gain.genomic_resources.repository.malformed_resource_id_reason(resource_id: str) → str | None[source]

Return why resource_id is not a well-formed id, or None.

Containment is the other rule an id is held to, and the two are separate: uncontained_resource_id_reason() asks whether the id escapes the repository, this one whether GAIn can process it at all. Both run where a .CONTENTS is read, and an id refused by either is dropped with a warning, the rest of the .CONTENTS still served. A scan runs neither: it parses each candidate path with parse_gr_id_version_token(), which enforces the same character class by construction, and a directory the parse refuses is skipped with a warning the same way. The two enumeration paths agree on both the grammar and its consequence.

Two segments are refused here that the scan grammar does not refuse – both / and . are inside its character class, so a//b and a/./b match it – because a scan never meets them: a filesystem has no empty name to offer, and every dot-named directory is skipped on the way down. Only a hand-written .CONTENTS can carry either. The . segment is the quieter of the two: it survives the join, the filesystem resolves it away, and the cached directory is then enumerated under a different id than the one it was served as (gain#1385).

"" and "." are exempt because both name the repository root, a supported resource in its own right that is published under the empty id and addressed as "." by build_local_resource. The exemption is by whole id: "." as a segment of a longer id is the refusal above.

gain.genomic_resources.repository.parse_gr_id_version_token(token: str) → tuple[str, tuple[int, ...]][source]

Parse a genomic resource id with an optional version suffix.

The suffix has the form (3.3.2); without one the version is (0,). The empty token names the repository root. Returns the (resource_id, version) tuple, and raises ValueError on a token outside the resource id grammar.

gain.genomic_resources.repository.parse_resource_id_version(resource_path: str) → tuple[str, tuple[int, ...] | None][source]

Parse a resource path into an (resource_id, version) tuple.

Like parse_gr_id_version_token(), but a path without a version suffix yields None for the version rather than (0,), so a caller can tell “unversioned” from “version 0”. Raises ValueError on a path outside the resource id grammar.

gain.genomic_resources.repository.report_uncontained_manifest_entries(resource_id: str, manifest: Manifest) → None[source]

Warn about – and never raise on – entries that escape the resource.

Rejecting a poisoned entry while parsing the manifest reads as defence in depth, but a manifest is parsed while ENUMERATING a repository: one bad entry then kills the generator before a single resource is yielded, and list, repo-repair and even resource-repair on an unrelated healthy resource all die with it. That is the gain#464 shape – one poisoned resource costing the whole repository – and it is not worth paying, because the load-bearing check sits at the join in get_resource_file_url() and fails the poisoned name loudly the moment anything tries to USE it.

So this only supplies the attribution the raise used to: a warning that names the resource AND the entry, which the raise could not do because a ManifestEntry does not know which resource it belongs to.

gain.genomic_resources.repository.resolve_tabix_index_filename(manifest: Manifest, filename: str) → str | None[source]

Return the tabix index of filename as recorded in manifest.

Resolution is manifest-driven on purpose: the manifest is already loaded and is protocol-agnostic, whereas probing with file_exists costs a network round-trip per candidate on the http and s3 protocols.

Returns None when the manifest records neither index – the caller decides whether that is a warning (the file set of an implementation) or an error (an open that needs an index). See gain#430.

gain.genomic_resources.repository.resolve_tabix_index_filename_for_read(resource: GenomicResource, filename: str) → str[source]

Return the index to read filename with, never building a manifest.

Consults the manifest only when it is already loaded or can be loaded from the resource: an open must stay a pure read, and GenomicResource.get_manifest() would build – md5-scanning the whole resource and writing state files – for a resource that carries no .MANIFEST (gain#430).

With no manifest at hand – the hand-authored GRR directory shape that collect_all_resources tolerates – falls back to probing the resource for each candidate index. The probe is deliberately confined to this branch: a manifest-backed resource must never pay the per-candidate network round-trip the probe costs on the http and s3 protocols.

With a manifest that records no index at all, or when no probe succeeds, falls back to the historical .tbi guess so that whatever pysam raises still names a concrete path.

gain.genomic_resources.repository.uncontained_resource_file_name_reason(filename: str) → str | None[source]

Return why filename is not resource-contained, or None.

Resource file names arrive from GRR content – the resource’s genomic_resource.yaml and its .MANIFEST – which is fetched from remote repositories and is therefore untrusted. A name is contained when it is relative and names no .., . or empty segment; nested names such as statistics/histogram_score.json are ordinary and stay allowed.

The joined location is a URL, not an os path, so the name is checked both as written and percent-decoded: an http(s) server decodes the path before resolving it, which makes %2e%2e a traversal there. A single decoding pass is the right depth – %252e%252e decodes to the literal text %2e%2e, which no server resolves any further.

A .. that would stay inside the resource (sub/../other.txt) is rejected as well, because the three backends GAIn speaks to disagree about what it means: yarl/aiohttp normalises it away client-side before the request is even sent, minio rejects the key outright (XMinioInvalidResourceName), and a local filesystem resolves it. One name, three outcomes – so it is refused everywhere rather than left to mean whatever the protocol of the day decides.

A degenerate name – empty, blank, ., or carrying an empty segment – is refused too: open_raw_file("") addressed the resource DIRECTORY, and no resource file is legitimately spelled that way. See gain#467.

A name carrying a control character is refused as well – not a containment failure but a reporting one, since the name is logged unescaped. See gain#642.

gain.genomic_resources.repository.uncontained_resource_id_reason(resource_id: str) → str | None[source]

Return why resource_id is not repository-contained, or None.

A resource id is the other operand of the same join a file name goes through: get_resource_url joins it onto the repository url. On the remote path it is read verbatim out of the repository’s .CONTENTS.json.gz, so it is exactly as untrusted as a manifest entry name – and containing only the file name left the escape wide open through its sibling (gain#467).

"" and "." are contained: both name the repository root, which is a supported resource in its own right – proto_builder addresses it as "" and build_local_resource as ".". Only the escape itself is refused here, not the degenerate spellings a file name is also held to: an id is joined once, at the root, so a . segment in it is a no-op rather than a way to address something else. (It is still refused – by malformed_resource_id_reason(), on the different ground that no scan can ever produce it.)

An id carrying a control character is refused, which is a reporting concern rather than a containment one – the id is logged unescaped at many call sites, and a newline in it forges a log line. See gain#642.

gain.genomic_resources.repository.validate_resource_file_name(resource_id: str, filename: str) → None[source]: Raise ValueError unless filename stays inside the resource.

gain.genomic_resources.repository.validate_resource_id(resource_id: str) → None[source]: Raise ValueError unless resource_id stays inside the repo.

gain.genomic_resources.repository.version_tuple_to_string(version: tuple[int, ...]) → str[source]

Convert version tuple to string representation.

Parameters:: version – Version tuple like (1, 2, 3)
Returns:: String representation like “1.2.3”

gain.genomic_resources.repository.version_tuple_to_suffix(version: tuple[int, ...]) → str[source]

Transform version tuple into resource ID version suffix.

The suffix is used to append version information to resource IDs. Default version (0,) produces no suffix.

Parameters:: version – Version tuple like (1, 2, 3)
Returns:: Empty string for version (0,), otherwise “(1.2.3)” format

gain.genomic_resources.repository_factory module

Provides a factory for building genomic resources repostiories.

class gain.genomic_resources.repository_factory.EmbeddedRepoDefinition(*, id: str | None = None, public_url: str | None = None, type: Literal['embedded', 'memory'], content: dict[str, Any] | None = None, cache_dir: str | Path | None = None)[source]

Bases: _RepoDefinitionBase

Definition for an in-memory genomic resource repository.

cache_dir: _PathOrStr | None

content: dict[str, Any] | None

model_config = {'extra': 'forbid', 'hide_input_in_errors': True}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

type: Literal['embedded', 'memory']

Bases: _RepoDefinitionBase

Definition for a local filesystem genomic resource repository.

cache_dir: _PathOrStr | None

directory: _PathOrStr

model_config = {'extra': 'forbid', 'hide_input_in_errors': True}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

read_only: bool | None

type: Literal['file', 'dir', 'directory']

Bases: _RepoDefinitionBase

Definition for a group of genomic resource repositories.

Child repository ids must be distinct across the whole subtree below a group, not merely among siblings: an id selects a repository (find_resource/get_resource take a repository_id) and, for a cached repository, names that repository’s cache directory. Two repositories sharing an id – at the same level or at different ones – leave the second unreachable. Duplicates are a configuration error, rejected here when the definition is validated.

An id is also a directory name – a cached repository derives each repository’s cache directory from it – so it must be a single path segment. One that is not (a separator, an absolute path, . or ..) would move cached data out of the configured cache_dir, and is rejected by the base definition model rather than rewritten (#460).

The uniqueness check covers children, not the definition root: the walk starts at self.children, so the root’s own id (explicit or synthesised) is not compared against any descendant’s here. It is not exempt – a repository_id naming the root selects it, like any other repository (#447), so a root sharing an id with a descendant shadows it. That pair is refused by _check_root_id_is_not_a_descendant_id in the top-level builder instead, which is the only place the root’s path through the definition tree is genuinely empty and its id can therefore be resolved as its parent would.

Spelling id on a child is optional. A child that omits it gets a deterministic id synthesised from its own identity – its url or directory, or its path from the definition root for an embedded / memory child or a nested group, which have neither. The synthesised id is never empty, and two children that would synthesise the same id (the same directory listed twice, say) are duplicates like any other.

cache_dir: _PathOrStr | None

check_child_ids_are_unique() → GroupRepoDefinition[source]

Reject a group whose descendants do not have distinct ids.

Compares the resolved ids – an explicit id where the child spells one, the synthesised id otherwise – so a pair that would end up sharing an id is refused whether or not the collision was spelled out.

The walk covers the whole subtree, not just the direct children. Repository ids share one namespace across nesting levels: a repository_id filter is matched against every repository in the tree, and a cached repository derives each child’s cache directory from its id, so two repositories at different levels sharing an id are as ambiguous as two siblings. Pydantic validates bottom-up, so a nested group has already checked its own subtree by the time this runs; repeating the walk from here is what catches the cross-level pairs a nested group cannot see.

children: list[RepoDefinition]

model_config = {'extra': 'forbid', 'hide_input_in_errors': True}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

type: Literal['group']

Bases: _RepoDefinitionBase

Definition for an HTTP/HTTPS genomic resource repository.

cache_dir: _PathOrStr | None

check_credentials_together() → HttpRepoDefinition[source]

model_config = {'extra': 'forbid', 'hide_input_in_errors': True}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

password: str | None

type: Literal['http']

url: str

user: str | None

warn_on_insecure_credentials() → HttpRepoDefinition[source]

Warn when basic-auth credentials ride a cleartext http:// URL.

Credentials are still accepted (localhost/dev GRRs legitimately use plain http), but a non-https URL to a non-local host means the base64-encoded credentials travel unencrypted, so emit a loud warning. The message never includes the password.

Bases: _RepoDefinitionBase

Definition for an S3 genomic resource repository.

cache_dir: _PathOrStr | None

endpoint_url: str | None

model_config = {'extra': 'forbid', 'hide_input_in_errors': True}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

type: Literal['s3']

url: str

class gain.genomic_resources.repository_factory.UrlRepoDefinition(*, id: str | None = None, public_url: str | None = None, type: Literal['url'], url: str, cache_dir: str | Path | None = None)[source]

Bases: _RepoDefinitionBase

Definition for a generic URL (http/https/s3) repository.

cache_dir: _PathOrStr | None

model_config = {'extra': 'forbid', 'hide_input_in_errors': True}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

type: Literal['url']

url: str

gain.genomic_resources.repository_factory.build_genomic_resource_group_repository(repo_id: str, children: list[GenomicResourceRepo]) → GenomicResourceRepo[source]

gain.genomic_resources.repository_factory.build_genomic_resource_repository(definition: dict | None = None, file_name: str | None = None) → GenomicResourceRepo[source]: Build a GRR using a definition dict or yaml file.

gain.genomic_resources.repository_factory.build_resource_implementation(res: GenomicResource) → GenomicResourceImplementation[source]: Build a resource implementation from a resource.

gain.genomic_resources.repository_factory.get_default_grr_definition() → dict[str, Any][source]: Return default genomic resources repository definition.

gain.genomic_resources.repository_factory.get_default_grr_definition_path() → str | None[source]: Return a path to default genomic resources repository definition.

gain.genomic_resources.repository_factory.load_definition_file(filename: str) → Any[source]: Load GRR definition from a YAML file.

gain.genomic_resources.repository_factory.redact_definition(definition: Any) → Any[source]

Return a deep copy of a GRR definition with credentials masked.

user/password values are replaced with "***" recursively (including inside a group repository’s children) so that a definition can be logged or embedded in an error message without leaking secrets. Credentials embedded in a URL’s userinfo (scheme://user:pass@host) are also scrubbed.

gain.genomic_resources.resource_errors module

The exceptions more than one tier must be able to name.

MalformedResourceError places the blame on the resource: its records or its configuration do not hold to what its kind can mean. It subclasses ValueError so it falls inside cli_errors.RESOURCE_ERRORS by construction, which is what makes grr_manage report it as one attributed line rather than as an unexpected internal error carrying a traceback (ADR 0008).

The module is a leaf – it imports nothing from GAIn – so the score layer, the table layer and the CLI tier can raise and catch the same exception without any of them acquiring a dependency on another.

HistogramError is here so that cli_errors.RESOURCE_ERRORS can name it without importing the histogram module; cli_errors says why it may not.

exception gain.genomic_resources.resource_errors.HistogramError[source]

Bases: Exception

A histogram-specific failure of one resource.

Raised for a categorical histogram past its cardinality limit, which the statistics scan catches and nullifies, and for a histogram file that cannot be read, which reaches grr_manage’s one-line reporting tier. A plain Exception, not a MalformedResourceError, because the second case is a fault of the resource’s state rather than of its records or configuration.

exception gain.genomic_resources.resource_errors.MalformedResourceError[source]

Bases: ValueError

A resource refused because of its own records or configuration.

Named for the state the resource is in rather than for any one rule, so that every refusal a reader could act on the same way – by fixing the resource – arrives under one name.

gain.genomic_resources.resource_errors.backwards_records_error(resource_id: str, chrom: str, pos: int, prev_pos: int, kind: str) → MalformedResourceError[source]

Refuse a score whose records move backwards along a contig.

The sibling of overlapping_records_error(), and here for the same reason. The position rule has been built from one helper since it was detected on two paths; this rule is detected on two paths for each of TWO kinds, and was written out at all four sites – six copies of one sentence, agreeing only for as long as nobody edited one of them.

kind is the possessive naming the kind whose promise was broken (“a fragment score’s”), so the message still says which validator fired. It is passed by each raise site rather than read off a shared class attribute: an attribute two validators interpret for themselves is what ADR 0008 unwound, and a message fragment is not a rule.

gain.genomic_resources.resource_errors.index_column_mismatch_error(resource_id: str, index_filename: str, mismatches: Sequence[tuple[str, int, int]], *, end_is_implied: bool = False) → MalformedResourceError[source]

Refuse a tabix table whose index was built over other columns.

Each entry of mismatches is a field, the column its index was built over, and the column its table resolves to – resolves, not states: a coordinate no configuration and no header names is read through a hardcoded fallback, and a fallback the index disagrees with splits the read from the filter exactly as a contradictory config entry does. Both columns are named because the remedy is a one-line edit and a reader cannot write it from either number alone. end_is_implied says the index records no end column at all, so its end column is its begin column by definition rather than by anyone’s choice.

Built here rather than at the raise site for the same reason overlapping_records_error() is: the rule is one rule, and it must read identically wherever it is reported.

gain.genomic_resources.resource_errors.inverted_span_error(chrom: str, pos_begin: int, pos_end: int, ref: str | None, alt: str | None) → OSError[source]

Refuse a single record whose end precedes its begin.

Returned rather than raised, so the raise stays at the site that read the slots – and so the several sites that perform this check share one message. Off the hot path by construction: a caller compares two integers per record and only calls this when the comparison fails.

Not to be confused with backwards_records_error(), despite the neighbouring vocabulary: that one refuses a resource whose records move backwards along a contig, raises MalformedResourceError, and belongs to the scan’s validation. This one is about a single record’s own two ends, and stays an OSError – the type the read path has always raised for it, and the type the tests pin.

Takes the five decoded values rather than the record they came from, for two reasons. Reading the slots here would mean importing them from genomic_position_table.record, whose package __init__ imports table_tabix, which imports this module – a cycle, and the end of this module’s leaf status (see the module docstring). And a record’s last slot is the backend’s payload, so a helper handed the whole record must be careful never to interpolate it: f"{record}" would print an entire pysam.VariantRecord – whose repr is the whole VCF line – or a TupleProxy. Named values cannot make that mistake.

ref and alt are what tell two records at one position apart, so the message names them when the record carries them, and says nothing in their place when it does not (ADR 0027).

gain.genomic_resources.resource_errors.overlapping_records_error(resource_id: str, chrom: str, pos: int, prev_end: int) → MalformedResourceError[source]

Refuse a position score whose record claims a position already taken.

Built here rather than at either raise site because two paths detect this one rule – the per-record region read and the vectorized statistics scan – and a reader who meets the message from one of them must not have to wonder whether the other words it differently.

gain.genomic_resources.resource_errors.score_configuration_error(resource_id: str, score_id: str, detail: str) → MalformedResourceError[source]

Refuse a score whose DEFINITION says what the score cannot be.

The sibling of overlapping_records_error() for the other half of what makes a resource malformed: not a record that breaks its kind’s promise, but a definition claiming a value the score cannot hold. The rules live in three layers, and each raise site phrases its own detail: the VCF header/config merge (the _refuse_* helpers in vcf_scores, for an id, an address or a type: the header contradicts, and for a field declared Number=G), the construction convergence point (refuse_unfoldable_histograms, for a number histogram over a value type none accumulates), and open, once a tabular table’s header is known (validate_scoredefs, for a column address the header cannot honour, and resolve_score_indices, which holds the DEFINITIONS – not the config – to the same address rules as it resolves each one to its payload column, and refuses a VCF definition with no INFO key). The Number=G rule is the one a scores: entry need not have caused: a header-only resource has none, and the claim is the header’s.

What is shared is the ADDRESS: which resource, and which score in it. That is the half a reader needs to find the file to edit, it is the half no raise site can word differently without sending someone to the wrong place, and it is why it is built here rather than at each site.

The prefix matches ResourceConfigValidationMixin so that a caller reading a config error sees one wording; the TYPE is MalformedResourceError, so a caller that already catches “this resource’s own config is bad” by type catches these too rather than matching on a string.

gain.genomic_resources.resource_errors.vcf_header_file_error(resource_id: str, header_filename: str, detail: str) → MalformedResourceError[source]

Refuse a VCF table whose *.header.vcf.gz sidecar is not a header.

The sidecar is read line by line (gain#1406), and two things can be wrong with it: no ## line at all, or a line pysam cannot parse – each phrased by its raise site as detail. What is shared is the ADDRESS, the resource and the file, built here for the reason score_configuration_error() gives.

gain.genomic_resources.resource_implementation module

gain.genomic_resources.resource_implementation.CONFIG_VALIDATOR_CACHE = <gain.genomic_resources.resource_implementation._ConfigValidatorCache object>: The process-wide resource-config validator cache. See the class.

class gain.genomic_resources.resource_implementation.GenomicResourceImplementation(genomic_resource: GenomicResource)[source]

Bases: ABC

Base class used by resource implementations.

Resources are just a folder on a repository. Resource implementations are classes that know how to use the contents of the resource.

abstractmethod calc_info_hash() → bytes[source]: Compute and return the info hash.

abstractmethod calc_statistics_hash() → bytes[source]

Compute the statistics hash.

This hash is used to decide whether the resource statistics should be recomputed.

collect_index_info() → tuple[tuple[str, ...], tuple[str, ...]][source]

Collect resource info for FTS index building.

Returns a (header, row) pair where header contains field names and row contains the corresponding values for this resource. Label keys/values are appended after the fixed fields.

Raises ValueError if a label key cannot name an index field – every implementation reaches the index through here, and the index build reports a raise from here against this one resource (gain#464).

An override that contributes further fields must call super() and append to what it returns. This is the only place a label key is checked against the names the index reserves: the build’s own re-check sees the finished header, in which an implementation’s fields legitimately appear, so it cannot tell a field from a label (gain#542). A field added by an override belongs in GR_INDEX_NON_LABEL_COLUMNS.

config: dict

abstractmethod create_statistics_build_tasks(**kwargs: Any) → list[TaskDesc][source]: Create tasks for calculating resource statistics for task graph.

property files: set[str]: Return a list of resource files the implementation utilises.

get_config() → dict[source]

The resource’s configuration.

As read from the resource at construction; an implementation that validates its configuration replaces it with the validated form, and answers that here.

abstractmethod get_info(**kwargs: Any) → str[source]: Construct the contents of the implementation’s HTML info page.

get_statistics() → ResourceStatistics | None[source]: Try and load resource statistics.

abstractmethod get_statistics_info(**kwargs: Any) → str[source]: Construct the contents of the implementation’s HTML statistics info page.

reload_statistics() → ResourceStatistics | None[source]

Drop the cached statistics and reload via get_statistics().

For after the statistics were rebuilt on disk. Answers what get_statistics() answers: None unless the implementation overrides it.

property resource_id: str: The id of the resource this implementation wraps.

class gain.genomic_resources.resource_implementation.InfoImplementationMixin[source]

Bases: object

Mixin that provides generic template info page generation interface.

class FileEntry(name: str, size: str, md5: str | None)[source]

Bases: object

Provides an entry into manifest object.

md5: str | None

name: str

size: str

get_info(**kwargs: Any) → str[source]: Construct the contents of the implementation’s HTML info page.

get_statistics_info(**kwargs: Any) → str[source]: Construct the contents of the implementation’s HTML info page.

get_statistics_template_data() → dict[source]

Return a data dictionary to be used by the statistics template.

Will transform the description in the meta section using markdown.

get_template_data() → dict[source]

Return a data dictionary to be used by the template.

Will transform the description in the meta section using markdown.

resource: GenomicResource

styles_template_name: ClassVar[str] = 'base_implementation_styles.jinja'

template_name: ClassVar[str] = 'base_implementation.jinja'

class gain.genomic_resources.resource_implementation.ResourceConfigValidationMixin[source]

Bases: object

Mixin that provides validation of resource configuration.

abstractmethod static get_schema() → dict[source]: Return schema to be used for config validation.

classmethod validate_and_normalize_schema(config: dict, resource: GenomicResource) → dict[source]

Validate the resource schema and return the normalized version.

What comes back is the caller’s own document all the way down, memo hit or not, and detached from the config it was normalized from – so an implementation may keep it and write into it, as GenomicScore.__init__ does.

Do not edit a resource’s config in place. Offering the same config object twice is answered from a memo rather than re-normalized (gain#1059), and an edit made between the two calls is not seen: the second caller gets the document as the config was the first time. Nothing in gain or gpf does this – the implementations that validate all keep the normalized copy rather than the config – and code that wants a config re-read should hand over a new dict, which is always normalized afresh.

class gain.genomic_resources.resource_implementation.ResourceStatistics(resource_id: str)[source]

Bases: object

Base class for statistics.

Subclasses should be created using mixins defined for each statistic type that the resource contains.

static get_statistics_folder() → str[source]

gain.genomic_resources.resource_implementation.get_base_resource_schema() → dict[str, Any][source]

gain.genomic_resources.resource_implementation.merge_index_columns(resource_id: str, columns: Sequence[str], claimed: Mapping[str, tuple[str, str]]) → dict[str, tuple[str, str]][source]

Return claimed extended with columns, keyed case-insensitively.

The index table has one set of columns for the whole repository – the union of every resource’s fields – so a field name that is fine within one resource can still be unusable next to another resource’s. SQLite compares column names case-insensitively, so assay in one resource and Assay in another are one column asked for twice under two spellings, and a CREATE VIRTUAL TABLE naming both fails – taking the whole repository’s index with it (gain#464). Two resources spelling a field the same way share the column, which is the point of the index.

Raises ValueError if columns cannot join claimed – naming the resource that already holds a spelling, or, past MAX_INDEX_COLUMNS, saying that the repository’s labels no longer fit an FTS5 table. The caller skips that one resource and keeps the rest. claimed is never modified – a rejected resource claims nothing.

gain.genomic_resources.resource_implementation.validate_index_columns(resource_id: str, columns: Sequence[str]) → None[source]

Check that columns can name the columns of the FTS index.

Every column name is interpolated into the CREATE VIRTUAL TABLE and INSERT statements that build the repository index, so a name that is not a bare identifier – and not one SQL or FTS5 has already taken – is at best unbuildable and at worst an injection (gain#464). Repeats are rejected too: the index keeps one column per name. Where the repeated name is a field of the resource itself, the label silently replaces that field’s value, which then cannot be found by it; where it is a name the index reserves for a field some other implementation contributes, the label lands in a column that means something else for every resource that does contribute it (gain#542). Both are reported against the offending column, the second naming the reserved set, since the curator has no such field of their own to look at.

Raises ValueError naming the resource and the offending column. The caller is the per-resource handler of the index build, so an offending resource is skipped and reported by id instead of taking the whole repository’s index down with it.

The columns are typed as strings but come from a YAML mapping’s keys, which need not be – a column that is not a string is refused like any other bad name rather than raising out of the check.

gain.genomic_resources.resource_query module

The query language that selects resources out of a repository.

A query is an fnmatch glob over the resource id, optionally followed by a bracketed query over the resource’s meta.labels:

hg38/scores/*[phenotype="aut*" and "UCSC" in provenance]

This module owns the grammar, the matching, and what a label value is read as (label_alternatives()) – and nothing else. It answers one question – does this resource match this query – and deliberately holds no policy about what a caller does with the answer: no result cap, no error when a query selects nothing. Those are the annotation layer’s rules about building a pipeline, not the repository’s rules about listing resources.

It lives here rather than in annotation so that the pipeline config, the repositories and the CLIs cannot disagree about what * means.

class gain.genomic_resources.resource_query.LabelClause(key: str, operator: LabelOperator, value: str)[source]

Bases: object

One condition on one label: key = value or value in key.

A clause is data, not a closure, so a caller that evaluates the query somewhere other than in Python – against the FTS index, say – can read what was asked without reimplementing what it means. Whatever engine runs the search, matches() stays the only definition of the comparison.

key: str

matches(label: str) → bool[source]: Check whether a rendered label value satisfies this clause.

matches_an_absent_label() → bool[source]

Check whether this clause holds for a label that is not there.

An absent label is matched as "", and under this grammar a clause that holds for "" holds for every string: a value must be at least one character, so in can never accept "", and the only = values fnmatch accepts "" for are globs of * alone. A caller with no other way to evaluate the clause can therefore drop it outright.

The converse says nothing. A clause that fails here still holds for whichever resources carry the key, so a false answer is not a licence to settle the clause for all of them.

matches_in(labels: Mapping[str, Any]) → bool[source]

Check whether labels satisfies this clause.

meta.labels is a free-form YAML mapping, so a label value is whatever YAML made of it – perturbed: False is a bool and year: 2019 an int, both of which the production GRRs carry in bulk. The query language only ever spells values as text, so a value is compared in its rendered form; without that both in and = raise a bare TypeError out of the predicate. A label the resource does not carry is matched as "". A list value is a set of alternatives: the clause holds if it holds for any one element, each rendered exactly as a scalar is (see label_alternatives()).

These rules live here rather than at the call sites so that a caller evaluating one clause reads a label exactly as the caller evaluating all of them does.

operator: LabelOperator

value: str

class gain.genomic_resources.resource_query.LabelOperator(*values)[source]

Bases: Enum

The comparisons a label clause can make.

CONTAINS = 'contains'

EQUALS = 'equals'

class gain.genomic_resources.resource_query.ResourceQuery(resource_id_pattern: str, label_clauses: tuple[LabelClause, ...])[source]

Bases: object

A parsed resource query: an id glob plus label clauses.

label_clauses: tuple[LabelClause, ...]

match(resource: GenomicResource) → bool[source]: Check whether resource matches the query.

match_id(resource_id: str) → bool[source]: Check whether resource_id matches the query’s id glob.

match_labels(labels: Mapping[str, Any]) → bool[source]

Check whether labels satisfies every one of the query’s clauses.

How one label value is read – its rendered form, a list as alternatives, absence as "" – is LabelClause.matches_in()’s to say, and is said there. The FTS index cannot represent the difference between an absent label and an empty one – it stores "" for every label column a resource does not carry – so treating absence as a distinct case would put this matcher permanently out of step with the same query evaluated in SQL.

static parse(query: str) → ResourceQuery[source]

Parse query into a matcher.

Raises ResourceQueryParseError if the query is not well-formed, or if it is longer than MAX_RESOURCE_QUERY_LENGTH.

resource_id_pattern: str

exception gain.genomic_resources.resource_query.ResourceQueryParseError[source]

Bases: ValueError

Raised when a resource query cannot be parsed.

gain.genomic_resources.resource_query.label_alternatives(value: Any) → tuple[str, ...][source]

Render a meta.labels value as the strings it stands for.

A label value is whatever YAML made of it. A scalar – a string, or the bool and int the production GRRs carry in bulk – stands for its str(); a nested mapping does too. A list or tuple is a set of alternatives, one rendered string per element, so a resource can be labelled with everything it is (modality: [RNA, ATAC]). An empty list stands for "", which is also what an absent label reads as.

Every reader of a label value renders it through here.

gain.genomic_resources.resource_types module

GAIn’s config vocabulary: which spellings are accepted, and which mean the same thing.

Mostly resource type: values, plus the annotator names that were retired alongside one of them.

Two different relations live here, and they are not the same. fragment_score and cnv_collection are equivalent – either resolves to the same thing, one is merely deprecated. np_score is retired: it is no longer accepted at all, and it was never equivalent to its replacement, since it carried a different default read mode. The first relation is served by equivalent_resource_types(), the second by reject_retired_resource() and, for the annotator names, retired_annotator_message().

Deliberately dependency-free and low in the import graph. The equivalence below is needed by repository (which applies the type predicate in SQL), by genomic_scores and its implementations, by annotation_config and by the web API – and genomic_scores imports repository, so a home in the score layer could not be reached from the layer that needs it most. That is not a detail: the review of gain#471 found that the SQL-side predicate had been missed precisely because the helper was out of reach.

See docs/adr/0003-fragment-score-vocabulary.md, superseded by docs/adr/0011-deprecate-cnv-collection-vocabulary.md.

gain.genomic_resources.resource_types.FRAGMENT_SCORE_TYPES = ('fragment_score', 'cnv_collection')

The resource type: values that name a fragment score.

Two spellings. fragment_score is what a resource should declare, and what the public GRR declares since the migration; cnv_collection is what a repository that has not migrated declares. It is deprecated and stops being accepted in LEGACY_VOCABULARY_REMOVAL_RELEASE; consuming it warns, at the places that open a resource rather than here.

A tuple rather than a set: it is used for membership, but also rendered into user-facing messages and into SQL placeholders, and a set would order them arbitrarily. Preferred spelling first, so a message reads as a recommendation.

gain.genomic_resources.resource_types.GENE_SCORE_TYPE = 'gene_score': The resource type: for a gene score. One spelling, as above.

gain.genomic_resources.resource_types.GENE_SET_TYPES = ('gene_set_collection', 'gene_set')

The resource type: values that name a gene set collection.

An ordered pair for the same reasons as FRAGMENT_SCORE_TYPES above, shared by the collection that opens the resource and the annotator that declares what it accepts (gain#1329).

Deliberately not an equivalence group. Unlike the fragment score’s pair, this one is absent from equivalent_resource_types below, so a search or type filter for gene_set_collection does not answer a repository’s gene_set resources. That is the behaviour as it stands, not a considered position: making the pair searchable would change what the repository predicate, the resources endpoint and the editor return, which is a decision of its own rather than a consequence of naming the pair here.

gain.genomic_resources.resource_types.LEGACY_ANNOTATOR_NAMES = {'cnv_collection': 'fragment_score', 'cnv_collection_annotator': 'fragment_score_annotator'}

Annotator names still accepted but deprecated, mapped to what to write instead.

The annotator half of the fragment-score vocabulary deprecation (ADR 0011, gain#538): both spellings are registered entry-point keys, so a pipeline naming either builds, and only the value is worth typing in a config written today. They stop being accepted in LEGACY_VOCABULARY_REMOVAL_RELEASE.

Here for the same reason RETIRED_ANNOTATOR_NAMES below is: two seams on opposite sides of an import edge need it – the fragment-score annotator warns on one, and annotation_config keeps them out of the names it advertises while parsing, and the former imports the latter. Deriving the set from a resource-type spelling instead would make a naming coincidence load-bearing: these are annotator names, and nothing obliges a deprecated one to be spelled like the type it reads.

gain.genomic_resources.resource_types.LEGACY_FRAGMENT_SCORE_TYPE = 'cnv_collection': The deprecated resource type: for a fragment score. Still accepted, and still declared by repositories that have not migrated.

gain.genomic_resources.resource_types.LEGACY_GENE_SET_TYPE = 'gene_set': The deprecated resource type: that also names one. Still accepted; opening one warns.

gain.genomic_resources.resource_types.LEGACY_VOCABULARY_REMOVAL_RELEASE = '2027.1.0': The GAIn release in which every legacy fragment-score spelling stops being accepted (gain#539). Named in every deprecation warning: a notice that does not say when it bites cannot be scheduled against.

gain.genomic_resources.resource_types.PREFERRED_ALLELE_SCORE_TYPE = 'allele_score': The resource type: for an allele score.

gain.genomic_resources.resource_types.PREFERRED_FRAGMENT_SCORE_TYPE = 'fragment_score': The preferred resource type: for a fragment score.

gain.genomic_resources.resource_types.PREFERRED_GENE_SET_TYPE = 'gene_set_collection': The resource type: for a gene set collection.

gain.genomic_resources.resource_types.PREFERRED_POSITION_SCORE_TYPE = 'position_score': The resource type: for a position score. One spelling, so there is no pair below; named here because this module owns the vocabulary and its sibling kinds are named here too.

gain.genomic_resources.resource_types.RETIRED_ALLELE_SCORE_TYPE = 'np_score'

The retired resource type: that used to name an allele score.

Deprecated since 2024-11 and removed in RETIRED_VOCABULARY_REMOVAL_RELEASE (gain#920). Unlike the fragment score’s legacy spelling above this one is no longer accepted, so it survives here only to be recognised and refused with a message that names the replacement.

gain.genomic_resources.resource_types.RETIRED_ANNOTATOR_NAMES = {'np_score': 'allele_score', 'np_score_annotator': 'allele_score_annotator'}

Annotator names GAIn no longer accepts, mapped to what to write instead.

The annotator half of the same retirement (gain#919): these named the allele-score annotator in a pipeline’s YAML, as np_score named its resource type. Each maps to the replacement of the same shape, so a migration is a one-word edit.

Here rather than in the annotation package because two seams need it and they are on opposite sides of an import edge: annotation_factory turns the name into a factory, and annotation_config turns it into a resource set while parsing – and the former imports the latter.

gain.genomic_resources.resource_types.RETIRED_VOCABULARY_REMOVAL_RELEASE = '2026.8.5': The GAIn release that removed np_score (gain#781, announced in gain#918). Named in the refusal so a reader who meets it in an old environment can tell which upgrade changed under them.

class gain.genomic_resources.resource_types.RetirableResource(*args, **kwargs)[source]

Bases: Protocol

The little of a resource reject_retired_resource() needs.

A structural type rather than GenomicResource itself: this module is deliberately dependency-free and low in the import graph (see the module docstring), and repository imports it, so naming the class here would close a cycle.

get_full_id() → str[source]: Return the resource’s id, with version where it has one.

get_type() → str[source]: Return the resource’s declared type:.

gain.genomic_resources.resource_types.deprecated_spelling_message(surface: str, legacy: str, preferred: str, *, found_in: str) → str[source]

Return the warning text for one use of one legacy spelling.

surface names the kind of configuration the spelling was written as ("resource type", "annotator name", "parameter"), found_in names where it was written – a resource id, or an annotator within a pipeline. Both are required because the stack at the point of the warning points into GAIn’s own config parsing rather than at the YAML the reader has to edit, so the message must carry the location itself.

A plain string rather than a logging call: the module that recognised the spelling logs it, so the record carries that module’s logger name.

gain.genomic_resources.resource_types.equivalent_resource_types(resource_type: str) → tuple[str, ...][source]

Return every type: value denoting the same kind of resource.

Only a fragment score is treated as having more than one spelling here; every other type maps to itself, so a caller can filter by the result unconditionally without special-casing.

A gene set collection also has two spellings (GENE_SET_TYPES) and is deliberately NOT one of them – see that constant. So this is the set of equivalences that SEARCH honours, which is narrower than the set of types some annotator will open.

Exists because filtering resources by an exact type string went wrong the moment a second spelling appeared: asking for fragment_score matched nothing at all in a repository whose resources declare cnv_collection. An empty result is indistinguishable from “this repository has none of those”, so the failure is silent – a wrong answer rather than an error.

gain.genomic_resources.resource_types.reject_retired_resource(resource: RetirableResource) → None[source]

Raise if resource declares a spelling GAIn has removed.

Called from each seam that turns a type: string into something: the score factory (build_score_from_resource), AlleleScore itself, the implementation builder (what grr_manage sweeps with), and the annotation pipeline’s resource resolver. Four call sites rather than one because there is no single seam they all pass through – ADR 0011 established the same for the fragment-score warning – and each is reachable without the others: the pipeline’s own type check would otherwise pre-empt this message with a generic one, and the implementation builder never constructs a score at all.

Not pushed down into GenomicResource itself, which would be the only common ancestor: that is also the enumeration and display path (grr_manage list, the web API’s resource-types endpoint, the repository’s SQL type predicate), and refusing there would abort a whole run over a repository that merely contains a retired resource. That is the failure ADR 0011 records as the reason its predecessor expired.

Raising here rather than letting the entry-point lookup fail is the whole point. Deleting the registration already makes an np_score resource fail, but it fails as unsupported resource implementation type <np_score> – which tells a holder that GAIn does not know the type, not that GAIn removed it and what to write instead.

Named by full id, matching the fragment-score notice: a repository may hold several versions of one resource id, each its own directory with its own config to migrate, and the bare id would name none of them precisely. Rendered only on the failure path – this runs on every resource open, including the statistics scan’s per-region rebuilds.

gain.genomic_resources.resource_types.require_fragment_score_type(resource_type: str) → str[source]

Return resource_type, or raise if it names no fragment score.

Lives here rather than at its one call site (the test-data builder’s with_resource_type) because builders.py sits four lines under pylint’s 1500-line module ceiling, and because the rule it enforces is this module’s to state.

gain.genomic_resources.resource_types.reset_deprecation_notices() → None[source]

Forget what this process has already announced.

Exists for tests: the announced-set is process-wide, and a test that asserts a warning fired must not depend on whether an earlier test in the same worker already consumed it. core/tests/conftest.py calls this before every test.

gain.genomic_resources.resource_types.retired_annotator_message(annotator_type: str) → str[source]

Return the refusal text for one use of a retired annotator name.

Silent about allele_score_mode, unlike its resource-type sibling above: the mode was only ever derived from the resource, and an annotator name is refused before any resource is opened, so a pipeline naming both retired spellings is told about the annotator first and about the resource – with its mode advice – once that is fixed.

gain.genomic_resources.resource_types.retired_resource_type_message(*, found_in: str) → str[source]

Return the refusal text for one use of the retired np_score.

found_in names where the type was written – a resource id – for the same reason the deprecation messages above carry it: the stack at the point of recognition runs through GAIn’s own config parsing, not through the YAML the reader has to edit.

The mode sentence is not padding. np_score is the one retired spelling that was never a pure alias: AlleleScore used to read the default mode off the resource type, so np_score meant substitutions while allele_score means alleles. A holder who swaps only the type string gets a resource that loads and reads differently, which is a silent wrong answer rather than an error – so the replacement and the mode key have to arrive together or the message causes the bug it is warning about.

gain.genomic_resources.resource_types.warn_deprecated_spelling(logger: Logger, surface: str, legacy: str, preferred: str, *, found_in: str) → None[source]

Announce one legacy spelling once per offender, per process.

The seams that recognise a legacy spelling are not once-per-offender on their own. FragmentScore.__init__ looked like it was – until the statistics scan, which rebuilds the score inside every min/max and histogram task: grr_manage repo-repair over an hg38-scale resource re-opens it once per region, so an unguarded warning there prints thousands of identical lines for a single offender. That is the noise the deprecation was supposed to avoid, and it hides the other offenders behind it.

Deduplicating on the rendered message keeps the property that matters – every distinct offender is named – without asking each call site to know how often it runs. The scope is the process: a multiprocess task run announces once per worker, which is bounded by the worker count rather than by the task count.

What is remembered is capped at _ANNOUNCEMENT_MEMORY distinct messages, oldest evicted first: found_in is caller-supplied, so an uncapped memory would grow with what a long-lived process has been asked to parse rather than with the repository it serves.

Tests reset the set through reset_deprecation_notices(), so an assertion never depends on what ran before it.

gain.genomic_resources.score_def module

What a genomic score DEFINITION is, how it reads a value, how it is built.

The bottom of the score layer: a score’s definition, the vocabulary it parses with (value types, NA sentinels, column addressing), the read that turns a record’s cell into a value, and the lifecycle that builds a resource’s definitions out of its scores: block. Nothing here knows about a GenomicScore – which is what lets vcf_scores sit above this module and genomic_scores above both, with no cycle between them.

Split out of genomic_scores so that the VCF-specific score code could be gathered into one module: vcf_scores constructs GenomicScoreDef at runtime, and genomic_scores imports what vcf_scores builds, so the two could not both live in one file without importing each other.

The lifecycle joined it in gain#1044, from private methods on GenomicScore. Only one of them was polymorphic, and only through the class attribute DEFAULT_AGGREGATORS, so each is a function parametrized by what it used to read off self. The one piece that stayed behind is the dispatch over the table’s TYPE (GenomicScore._build_scoredefs): it calls into vcf_scores and bigwig_scores, both of which import this module, so hosting it here would close a cycle.

The other half of a definition’s story is told elsewhere, for that same reason: genomic_scores.value_extraction (gain#1114) sits above this module and holds both decisions taken at open. So what a definition IS and how it parses a cell are here; how one is ADDRESSED and which read reaches it are there.

gain.genomic_resources.score_def.BULK_PARSEABLE_VALUE_TYPES = ('float', 'int', 'str'): Value types GenomicScoreDef.parse_array() defines a column parse for, and so the ones a bulk column read can serve. bool is absent because no column consumer asks for it.

class gain.genomic_resources.score_def.GenomicScoreDef(score_id: str, value_type: str, desc: str, small_values_desc: str | None, large_values_desc: str | None, hist_conf: HistogramConfig | None, aggregator: str | None, col_name: str | None, col_index: int | None, value_parser: Any, na_values: Any)[source]

Bases: ScoreDef

A genomic score definition. Includes backend loading internals.

Extends the shared ScoreDef (score id, value type, description and histogram config) with the concerns that are genomic-only: the per-position and per-allele default aggregators, and the internal column addressing / parsing state used when reading a value off a table backend.

aggregator: str | None

col_index: int | None

col_name: str | None

empty_element_warned: bool = False

na_values: Any

number_mismatch_warned: bool = False

parse_array(cells: ndarray) → ndarray[source]

Turn a whole column of raw cells into values, vectorized.

The column half of this definition’s parsing contract, and the reason the bulk statistics scan is worth having. Equivalent to [parse_value(c) for c in cells], with the “no value” that the scalar contract spells None rendered in whatever form the returned array can carry. That equivalence is not an aspiration: test_parse_array_agrees_with_parse_value_fuzz asserts it token by token, per value type, over several na_values configs and several array widths.

The returned array is one of two shapes, chosen by value_type:

float and int – a float64 array whose nan is the non-value. A float64 array has no None, and for every consumer of a numeric column a non-value and a nan are the same skip.
str – an object array of str, whose None is the non-value, exactly as parse_value() returns it.

Any other value type is refused: bool has no column consumer, and an unset value_type is not a parse this can define.

Parsed with numpy, deliberately NOT with ``pd.to_numeric``, which is not correctly rounded – it returns 9.999999999999999e-26 for 1e-25 and truncates 0.00000071009127180852 to ten significant digits. ndarray.astype agrees with float() on every token tested, including the PEP-515 underscores and Unicode digits pandas rejects outright.

Turn one raw cell into this score’s value.

None for a null raw value (an absent VCF INFO key), for a configured NA sentinel, and for a cell that fails to parse – a bad cell is logged and skipped rather than aborting a whole scan.

The scalar half of this definition’s parsing contract; the column half is parse_array(). Both live here, on the object that owns the two inputs they need (value_parser and na_values), so neither can be changed against a config the other did not see.

score_index: int

value_parser: Any

gain.genomic_resources.score_def.SCORE_TYPE_PARSERS: dict[str, Callable[[Any], Any]] = {'bool': <function parse_bool>, 'float': <class 'float'>, 'int': <class 'int'>, 'str': <class 'str'>}: How each value type turns a raw cell into a value. Annotated rather than inferred: three entries are builtin TYPES and bool’s is a function, so an inferred value type joins to object, which is not callable as far as a type checker is concerned.

gain.genomic_resources.score_def.build_genomic_score_schema() → dict[str, Any][source]

Declare the config every genomic score kind accepts.

The kinds’ get_schema overrides deep-copy this and splice in their aggregator, so it must build a fresh dict per call.

gain.genomic_resources.score_def.extract_column_value(record: tuple[Any, ...], score_def: GenomicScoreDef) → str | int | float | bool | None[source]

Read one score off a record whose PAYLOAD is a raw row.

The tabular backends and bigWig: a score is a CELL of the payload, addressed by the integer column value_extraction.resolve_score_indices resolved into score_index. Turning that cell into a value is the definition’s job (GenomicScoreDef.parse_value()), so that this read and the bulk column read cannot drift apart.

A pure function of (record, score_def) – it holds no state and needs none, which is what let the per-line score-line objects go. There is no “score_index not resolved yet” check: the attribute does not exist until open() sets it, so an unopened def raises AttributeError naming it.

gain.genomic_resources.score_def.finish_scoredefs(score_defs: dict[str, GenomicScoreDef], default_aggregators: dict[str, str | None]) → dict[str, GenomicScoreDef][source]

Fill in what a definition cannot decide for itself.

The value type. type: is optional, and an unstated one is recorded as float, with the float value parser alongside it – the two are resolved together, here, because they were left unresolved together by parse_scoredef_config (gain#1221). Recording the type matters rather than leaving it None: GenomicScoreDef.__post_init__ returns early on a None type, which would skip na_values normalization and leave the raw config string in place. That turns the NA check into a SUBSTRING test, so a score configured na_values: "-1" would read a real value of 1 as a null – the exact defect normalize_na_values prevents.

It is resolved HERE rather than at parse time because for a VCF score an unstated type means “the type the file’s header declares”, and defaulting before the merge would override a declared int with float. Filling it afterwards leaves that inheritance intact and still leaves no definition without a type.

The aggregator.

The default depends on how the score is reduced, which is fixed by the resource type – mean over a region of positions, max over the alleles at one, join(,) rather than list for a fragment score’s strings – so it cannot be decided from a definition alone. default_aggregators is that decision, made by the caller: each score class passes its own DEFAULT_AGGREGATORS. Resolving it here rather than in GenomicScoreDef.__post_init__ is what lets one field replace the pos_aggregator/allele_aggregator pair.

Applied at the convergence point of all three construction routes – the scores: block, a VCF header, a bigWig – because a default applied in only one of them is the same bug in a new place: a VCF-derived def would arrive with aggregator=None, and the fragment score annotator drops an attribute whose aggregator is None silently.

gain.genomic_resources.score_def.normalize_na_values(na_values: Any, value_type: str) → set[Any][source]

Normalize a configured na_values into a type-aware sentinel set.

The resource schema permits na_values as a bare scalar (na_values: "-1") or a list. A bare str left un-normalized turns the NA membership test in _extract_vcf_value() into a SUBSTRING test ("1" in "-1" is True) and raises TypeError when matched against a non-string raw payload (bigWig floats). This wraps a scalar into a one-element collection and returns a set that carries, for every configured sentinel, both its text form (matched against string backends) and – for numeric score types – its parsed form (matched against a float/int raw payload). So a sentinel is matched against whichever representation the incoming raw value presents, never by substring.

na_values of None selects the per-value-type default set verbatim: the defaults are non-numeric tokens ("", "nan", ".", "NA") that a numeric backend never presents as a raw value, so they are left as a pure-text set – coercing them would only add a spurious parsed nan and change the default behaviour.

A set input is treated as ALREADY normalized and returned as a copy without re-coercion, so normalization is idempotent (a fixed point). This is what the VCF scores-block merge path relies on: it rebuilds a GenomicScoreDef from an already-normalized na_values set, whose __post_init__ re-runs this function – a second coercion pass would otherwise grow the set (e.g. parsing the default "nan" text token into a float('nan')) and silently change the statistics hash. Config-supplied na_values never arrive as a set (the schema permits only None, str or list), so a set can only be a prior normalization result.

gain.genomic_resources.score_def.parse_bool(value: Any) → bool[source]

Read a bool score’s raw value: its TEXT, against a closed set.

A value that is ALREADY a bool is returned unchanged. That is the idempotence every parser in SCORE_TYPE_PARSERS owes its caller – float(1.5) and int(3) give it for free, and a VCF score reaches the parser with a value pysam has already decoded – so it is the shared contract rather than a special case for flags.

Anything else raises ValueError, including a bare 0/1 number: a number is not a bool, and the only way one arrives is a resource declaring type: bool over a numeric field. GenomicScoreDef.parse_value() logs the refusal and reads the cell as a non-value, so one bad cell does not abort a scan.

Why the vocabulary is closed, why bool alone declares no NA sentinels, and what that costs: docs/adr/0024-a-bool-score-reads-its-cells-text.

gain.genomic_resources.score_def.parse_scoredef_config(config: dict[str, Any]) → dict[str, GenomicScoreDef][source]: Parse ScoreDef configuration.

gain.genomic_resources.score_def.validate_scoredefs(config: dict[str, Any], table: GenomicPositionTable, resource: GenomicResource) → None[source]

Check each configured score’s column address against the table’s header.

For a TABULAR table only: a bigWig’s scores are checked by validate_bigwig_scoredefs and a VCF’s have no column address at all (each reads the INFO field named by its id, and parse_vcf_scoredefs refuses an address that says otherwise), so GenomicScore.open() does not send either here. It runs at open because the header is only known then.

A score is refused – through score_configuration_error(), naming the resource and the score, never by assert (a resource config is data, and python -O strips an assert) – when it names a column of a table that has no header (header_mode: none is the one way a tabular table is still headerless once open), names one the header lacks, indexes past the header, or states no address at all. A headerless table’s index addresses are not checked here: there is no header to bound them by.

Also rewrites the legacy name:/index: spellings into column_name:/column_index: IN the config it is given – which is why it takes the config rather than reading one, and why the caller must hand it the same dict the score keeps.

gain.genomic_resources.score_filter module

Boolean record filters over a genomic score’s own values.

One grammar and one compiler, owned by the score rather than by whichever annotator happens to want filtering: a filter reads score values off a record, so it belongs where the score definitions are. See docs/adr/0017-score-filtering-is-a-score-capability.md.

gain.genomic_resources.score_filter.SCORE_FILTER_GRAMMAR = '\n?start: or_expr\n\n?or_expr: and_expr | or_expr "or" and_expr -> or\n\n?and_expr: not_expr | and_expr "and" not_expr -> and_\n\n?not_expr: primary | _NOT not_expr -> not\n\n?primary: comparison | "(" or_expr ")"\n\ncomparison: subject operator subject\n\n?subject: variable | value\n\nvalue: "\\"" text "\\"" | number\n\nvariable: name\n\noperator: equals | not_equals\n | greater_than | greater_or_equal\n | less_than | less_or_equal\n | in\n\nequals: "=="\n\nnot_equals: "!="\n\ngreater_than: ">"\n\ngreater_or_equal: ">="\n\nless_than: "<"\n\nless_or_equal: "<="\n\nin: "in"\n\n_NOT: /not(?![a-zA-Z0-9_@#$%^&*+])/\n\nname: /[0-9]*[a-zA-Z_@#$%^&*+][a-zA-Z0-9_@#$%^&*+]*/\n\ntext: /[0-9]*[a-zA-Z_@#$%^&*+!()][a-zA-Z0-9_@#$%^&*+!()]*/\n\nnumber: /-?(?:[0-9]+\\.?[0-9]*|\\.[0-9]+)/\n\n%ignore /\\s+/\n': The filter language. The rules cascade or -> and -> not -> comparison-or-group precisely so that precedence is declared here rather than left to the parser’s ambiguity resolution, and adding an operator means placing it in that cascade. _NOT is guarded against matching inside a longer word so that notch is a name and not a negated ch.

class gain.genomic_resources.score_filter.ScoreFilter(score: GenomicScore, expression: str, predicate: Callable[[Record], bool])[source]

Bases: object

A compiled record predicate, bound to the score that compiled it.

Opaque on purpose: a caller compiles one through GenomicScore.compile_filter() and passes it back to a fetch, and the tree it was compiled from is nobody else’s business. The source expression is kept for error messages and repr.

It carries the score because its variables are bound to that score’s definitions – a column index, a value type, an NA set – and none of those travel with a record. See require_owner().

require_owner(score: GenomicScore) → None[source]

Refuse to read the records of a score that did not compile this.

Checked once per fetch, not per record. The failure it prevents is silent: two resources both defining freq put it at different column indexes, so a foreign filter reads a real value from the wrong column and selects records nobody can tell are wrong.

select(score: GenomicScore, records: Iterable[Record]) → Iterator[Record][source]

Yield the records of score this filter accepts.

The only way to apply a filter, and it takes the score being read rather than the records alone, so require_owner() cannot be forgotten: there is no reachable way to test a record against a filter without first saying which score the record came from. The predicate itself stays private for that reason.

Applying the predicate used to be the caller’s own business, and the allele annotator’s region path was the one caller that did it – the single filter application the ownership check could not cover (ADR 0017). It reads through this now, as every other path does.

Lazy, and the ownership check is not: a foreign filter is refused from the call, before any record is read, because it is a programming error rather than a property of the data. A caller that defers this call into a generator body defers the refusal with it – GenomicScore.fetch_records() does, and says so.

exception gain.genomic_resources.score_filter.ScoreFilterError[source]

Bases: ValueError

A filter expression that does not compile against a score.

gain.genomic_resources.score_filter.compile_score_filter(score: GenomicScore, expression: str) → ScoreFilter[source]

Compile expression into a predicate over score’s records.

Every variable is checked against the score’s score_definitions HERE, so an expression naming a score the resource does not define is refused now, with the valid names, rather than per record at read time.

gain.genomic_resources.score_filter.select_records(score: GenomicScore, records: Iterable[Record], score_filter: ScoreFilter | None) → Iterator[Record][source]

Apply an optional filter to a record stream.

Every read takes its filter as ScoreFilter | None, so every read would otherwise answer “and what does absent mean?” for itself. It means the records unchanged, and it means that in one place.

gain.genomic_resources.score_implementation module

The implementation plane shared by gene scores and genomic scores.

ScoreImplementationBase sits one layer above ScoreResource (the catalogue plane in gain.genomic_resources.score_resource): where that base owns what a score is, this base owns what a score implementation does that means the same for both families – contributing score_ids / score_descriptions into the FTS index, and serialising-and-plotting a computed histogram into the resource.

It deliberately keeps create_statistics_build_tasks abstract: a gene score emits a single task that scans a DataFrame, whereas a genomic score emits a region-split DAG with a min/max merge stage. These are genuinely different strategies for genuinely different data shapes, so the base does not try to unify them.

The location mirrors score_resource for the same reason: gene_scores already depends on genomic_resources, so living here adds no new dependency edge, whereas a top-level gain/scores/ package would create a cycle.

class gain.genomic_resources.score_implementation.ScoreImplementationBase(genomic_resource: GenomicResource)[source]

Bases: GenomicResourceImplementation, InfoImplementationMixin

Shared implementation base for gene and genomic score resources.

A concrete subclass must set self.score (a ScoreResource) in its own __init__; from it this base reads the score definitions for the search index and the histogram save-and-plot loop.

collect_index_info() → tuple[tuple[str, ...], tuple[str, ...]][source]

Collect resource info for FTS index building.

Returns a (header, row) pair where header contains field names and row contains the corresponding values for this resource. Label keys/values are appended after the fixed fields.

Raises ValueError if a label key cannot name an index field – every implementation reaches the index through here, and the index build reports a raise from here against this one resource (gain#464).

An override that contributes further fields must call super() and append to what it returns. This is the only place a label key is checked against the names the index reserves: the build’s own re-check sees the finished header, in which an implementation’s fields legitimately appear, so it cannot tell a field from a label (gain#542). A field added by an override belongs in GR_INDEX_NON_LABEL_COLUMNS.

abstractmethod create_statistics_build_tasks(**kwargs: Any) → list[TaskDesc][source]

Create tasks for calculating resource statistics for task graph.

Kept abstract: gene and genomic scores build statistics with genuinely different task shapes (a single DataFrame scan versus a region-split DAG), so each family provides its own.

score: ScoreResource

gain.genomic_resources.score_resource module

The catalogue plane shared by gene scores and genomic scores.

ScoreResource is the one base both families extend. It owns only the things that mean the same for a gene score and a genomic score: the score definitions, the two ways to enumerate/look them up, and the histogram accessors. It deliberately owns nothing about a resource’s lifecycle (open/ close), its table, its fetch surface, its chromosomes or its aggregators – those belong to genomic scores alone, which are keyed by position and read over a region, whereas gene scores are keyed by gene symbol, have no open/close and nothing to aggregate.

The boundary is not incidental; it is the whole reason this module exists. See docs/2026-07-14-gain-score-abstraction.html for the design, and tests/small/genomic_resources/test_score_resource_api.py for the guard that keeps a lifecycle/table/fetch/aggregator method from being lifted into this base “because both subclasses happen to have one”.

The location is deliberate too: gene_scores already imports genomic_resources.{histogram,repository,resource_implementation}, so living here adds zero new dependency edges. A top-level gain/scores/ package would instead create a genomic_resources <-> scores import cycle.

class gain.genomic_resources.score_resource.ScoreDef(score_id: str, value_type: str, desc: str, small_values_desc: str | None, large_values_desc: str | None, hist_conf: NullHistogramConfig | CategoricalHistogramConfig | NumberHistogramConfig | None)[source]

Bases: object

Catalogue-plane fields common to a gene score and a genomic score.

Column addressing is intentionally not here: it is a loading detail that differs per family (a VCF keys a score by INFO name, a bigWig has no header, a gene score renames a pandas column), so it lives on the concrete GenomicScoreDef / GeneScoreDef subclasses instead.

desc: str

hist_conf: NullHistogramConfig | CategoricalHistogramConfig | NumberHistogramConfig | None

large_values_desc: str | None

score_id: str

small_values_desc: str | None

value_type: str

class gain.genomic_resources.score_resource.ScoreResource[source]

Bases: ResourceConfigValidationMixin, Generic

Shared catalogue base for gene and genomic score resources.

Parameterised by the concrete score-definition type so that get_score_definition returns the right kind for each family (a GenomicScoreDef for genomic scores, a GeneScoreDef for gene scores) without either side having to override it.

A concrete subclass must set two attributes in its own __init__:

resource – the underlying GenomicResource, from which the histogram accessors read the manifest and public URLs;
score_definitions – the score_id -> definition mapping.

Everything a subclass may add on top of this (a table, an open/close lifecycle, fetch methods, aggregators) is its own concern and must NOT be lifted here – see the module docstring and the API-surface guard test.

get_all_scores() → list[str][source]

get_histogram_filename(score_id: str) → str[source]: Return the histogram filename for a score.

get_histogram_image_filename(score_id: str) → str[source]

get_histogram_image_public_url(score_id: str) → str[source]

Return the histogram image URL on the resource’s public mirror.

Unlike get_histogram_image_url(), this is built from the resource’s public URL so it is reachable from a browser even when the GRR is a local directory repository.

get_histogram_image_url(score_id: str) → str | None[source]

get_score_definition(score_id: str) → ScoreDefT | None[source]

get_score_histogram(score_id: str, *, truncated: bool = False) → NullHistogram | CategoricalHistogram | NumberHistogram[source]

Return defined histogram for a score.

A score may declare a categorical (or null) histogram just as readily as a numeric one, so the honest return type is the full Histogram union. Callers that need numeric-only attributes (bars/bins/min_value/…) must narrow with isinstance(hist, NumberHistogram) first.

With truncated=True a truncated sidecar is acceptable: when one exists (categorical histograms past UNIQUE_VALUES_LIMIT have one), it is returned instead of the full histogram, so the caller never reads the full values file. Without a sidecar the full histogram is returned either way.

get_score_range(score_id: str) → tuple[float, float] | None[source]: Return the value range for a numeric score.

static histogram_schema() → dict[str, Any][source]

The histogram config-schema fragment shared by both families.

Contributed into each family’s get_schema() instead of pasted into both – the two blocks used to be byte-identical modulo line-wrapping. Built fresh on every call so a caller that mutates the returned schema (e.g. a copy.deepcopy then in-place edit) cannot affect another.

resource: GenomicResource

score_definitions: dict[str, ScoreDefT]

gain.genomic_resources.score_resource.refuse_unfoldable_histograms(score_defs: dict[str, ScoreDefT], resource_id: str) → dict[str, ScoreDefT][source]

Refuse a configured NUMBER histogram no value of the score can feed.

A histogram: {type: number} over a score whose value type is not one a number histogram accumulates (NUMBER_HISTOGRAM_VALUE_TYPES) is a config stating something the score cannot do, so gain#1336 raises on it rather than working around it, naming the resource and the score.

It is a fact about a score DEFINITION – the value_type and hist_conf every ScoreDef carries – so it lives here, on the base both families share, and each family calls it once at its own construction point (gain#1308): the caller decides when (after the value type is final, before anything reads the column), and says why there. Raising at CONSTRUCTION rather than where the statistics build reads the configs is what makes the refusal reach every consumer: annotation never unpacks score definitions.

Two things it does not refuse. A definition with hist_conf=None is skipped – the default chosen for its type later never pairs a non-numeric type with a number histogram. And a CATEGORICAL histogram over a number is deliberately let through: it folds one value at a time and nullifies just that score, which is a fact about a value rather than about the config.

gain.genomic_resources.utils module

gain.genomic_resources.utils.build_chrom_mapping(resource: GenomicResource | None, config: dict[str, Any] | None = None) → Callable[[str], str | None] | None[source]

Build chromosome mapping function from resource config.

The resource config may contain chrom_mapping section with filename, add_prefix and del_prefix keys. The filename points to a file with two columns: original chromosome names and mapped names.

These keys are mutually exclusive, only one of them may be present.

Parameters:: resource – genomic resource with config
Returns:: function that maps chromosome names or None if no mapping is defined

gain.genomic_resources.utils.read_resource_id_label(resource: GenomicResource, label: str) → str | None[source]

The named label’s value, when that label names another resource.

Read through the accessor that narrows both meta levels, the way every other label reader does (gain#654, gain#1004) – and then narrowed once more, because that accessor promises a mapping and says nothing about what is in it (gain#1050). A value that cannot be a resource id reads as absent and is reported. An absent label, and the explicit YAML null the production GRRs carry, are not curator mistakes and stay silent.

Taken by label name rather than hard-wired to reference_genome, because four labels across three modules name a resource this way: reference_genome on gene models (gain#1050) and on scores, and source_genome/target_genome on a liftover chain. All four read a free-form YAML value into a str | None, and unnarrowed all four died the same way – the int in a regex, the list and the dict wherever the id was first hashed – with a TypeError that named neither the resource nor the label (gain#1053).

Lives here rather than beside any one of its callers, and beside build_chrom_mapping(), which is the same kind of thing: a read of a resource’s own configuration that several resource types share.

Whitespace is deliberately NOT stripped: only an empty value is narrowed away, so a padded id or the trailing newline a folded scalar leaves still reaches resolution and fails there naming itself. Stripping would be a normalization policy for ids rather than a narrowing, and that is a separate decision.

gain.genomic_resources.variant_utils module

gain.genomic_resources.variant_utils.maximally_extend_variant(chrom: str, pos: int, ref: str, alts: list[str], genome: ReferenceGenome) → tuple[str, int, str, list[str]][source]: Maximally extend a variant.

gain.genomic_resources.variant_utils.normalize_variant(chrom: str, pos: int, ref: str, alts: list[str], genome: ReferenceGenome) → tuple[str, int, str, list[str]][source]

Normalize a variant.

Using algorithm defined in the https://genome.sph.umich.edu/wiki/Variant_Normalization

gain.genomic_resources.vcf_scores module

Reading a VCF’s INFO fields as genomic scores.

Everything the score layer knows about VCF, in one module: how a VCF header’s INFO metadata becomes score definitions, and how one of those scores is read off a record. Both encode the same thing – INFO field semantics, and Number=1/A/R/. in particular – and they used to sit 362 lines apart in genomic_scores.

The VCF table itself is not here and does not belong here. genomic_position_table.table_vcf produces records: it owns the payload’s shape, the pysam proxies it carries and the constants that name them. This module interprets those records as scores. That is the same seam the record contract draws everywhere else – a backend yields records, the score layer says what they mean – and it is why the table layer still imports nothing from the score layer.

gain.genomic_resources.vcf_scores.extract_vcf_value(record: tuple[Any, ...], score_def: GenomicScoreDef) → str | int | float | bool | None[source]

Read one score off a VCF record: an INFO field, not a column.

VCF is the awkward backend and this function is where the whole of its awkwardness lives. A VCF score is addressed by INFO name – which is col_name, the string the config gave – looked up on the variant, typed by the header metadata, and for a per-allele field selected by the record’s allele index.

The five cases a TUPLE value falls into (a scalar value – the common shape, a well-formed Number=1 or a Flag – skips them all):

Number=A – one value per ALT allele: select this record’s allele. A record whose ALT is absent (‘.’) has no allele index and so no applicable value – under the VCF spec such a record has zero ALT alleles, so a Number=A field on it carries zero values and a row that supplies one anyway is malformed. It yields None, a null score, however many values the field carries and whatever the score def’s declared type (#256). Returning the null HERE also keeps the raw tuple from escaping as a score value. The check is a crash guard too: without it the tuple is indexed with None and the read dies with TypeError.
Number=R – one value per allele including the reference, which occupies offset 0: an ALT allele reads at allele_index + 1, and a record with no ALT reads the reference value at offset 0.
Number=1 or Number=0 – a declared scalar that arrived as a tuple (SC=2.5,3.5 under Number=1; htslib does not enforce Number on read). Refused: the row reads None and is reported by _check_number_arity() (gain#1257). It is decided on the tuple as pysam hands it over, ahead of the empty-element drop, so SS=a, is an over-arity row and not a one-value row with an empty element.
Number=. and Type=String – an unbounded string field, joined on ‘|’ into a single value (a VCF-local convention).
anything else – handed to parse_value, which joins it on ‘|’ through the converter parse_vcf_scoredefs installed.

An EMPTY element contributes nothing, whatever the shape (#630). ORIGIN=1, declares a value it does not carry, and pysam decodes that element as None. Joining one was the operation whose outcome depended on nothing but the field’s declared Type – "|".join raised TypeError here and took the whole fetch, while the converter’s map(str, ...) silently annotated the text '3|None' for every other type – so the non-per-allele shapes drop their empty elements through _drop_empty_elements() before either join sees one, and a tuple left with nothing reads null rather than the '' a join of nothing produces. The drop is done HERE, once, for both joins: this is the layer that still has the record, so it is the only one that can name the row it reports, and it leaves the converter a tuple that cannot contain a None.

The per-allele shapes need no drop – an empty element they select is already the null they give an allele with no value of its own – but they REPORT it, because the row is malformed either way and the arity check cannot see it: ALT=T,G S=d1, carries one value per allele and trips nothing, while its second allele reads null for want of a value the row said it had. Which fields can say that of a None at all is _reports_empty_elements().

Neither per-allele selection trusts the tuple to be the right length (#289). Nothing rejects a row whose value count does not match its ALT column – not pysam, not resource load – and indexing one bare aborted the whole fetch: the second allele of ALT=T,G S=d11 ran off the end with IndexError, and the try in parse_value that turns a bad cell into a logged null is deliberately not around this call (it guards the PARSE, and #256’s crash-guard test depends on that placement), so the crash escaped get_score and took the scan with it. An allele past the end of the tuple therefore reads None – the same null #256 gives the ALT-less record, by the same rule: no applicable per-ALT value, no score. The mismatch itself is reported by _check_number_arity(), which also catches the mirror shape (more values than alleles, the extras unreadable), once per table.

A key the header declares but this record does not carry yields None rather than raising: info.get returns None, None is not a tuple, so the number cases are skipped and parse_value turns it into a null score. For a key the header does NOT declare, pysam’s info.get raises ValueError: Invalid header – but nothing in this tree can ask for one, since a VCF table’s score defs are built FROM the header and a configured score naming an undeclared field is rejected when the score is opened (pinned by test_vcf_check_for_missing_score_columns).

The metadata lookup stays inside the tuple branch. INFO_META.get builds a fresh pysam VariantMetadata for the key, per score, per record. A Number=1 field decodes to a scalar, never reaches that branch, and must not pay for a metadata object it will never read; that is the common shape of a score-bearing INFO field, and hoisting the lookup out of it took a 50-score read of a 3000-row VCF from 26.65 to 19.83us/line. (Pinned by test_vcf_reads_the_info_metadata_only_for_a_tuple_value.)

The two pysam proxies this reads – INFO and INFO_META – are resolved once per record by the VCF backend and carried in the payload, because pysam allocates a fresh proxy on every variant.info access. See table_vcf for that measurement and why they live there.

gain.genomic_resources.vcf_scores.parse_vcf_scoredefs(vcf_header_info: dict[str, Any] | None, config_scoredefs: dict[str, GenomicScoreDef] | None, *, resource_id: str, merge: bool = False) → dict[str, GenomicScoreDef][source]

Build score definitions from a VCF header’s INFO metadata.

Every INFO field the header declares becomes a score, typed through VCF_TYPE_CONVERSION_MAP and described by the header’s own description. This is why a VCF resource needs no scores: block to be usable: the file documents its own scores.

value_parser is set to None for Number of 1, A or R, because pysam already decodes those to a scalar (or to a tuple that extract_vcf_value() indexes by allele). Every other shape keeps converter, which joins a tuple on ‘|’ – the VCF-local convention for a field whose arity the header does not fix.

converter joins with str – a Number=./Type=Integer field therefore reads as text, which is now what such a field DECLARES as well (gain#1259; it used to declare the header’s Type= and was the one place the definition and the value disagreed) – and which is what made it the SILENT half of #630: an empty element would render as the four-character string 'None'. It is not guarded here. The tuples that reach it have already had their empty elements dropped by extract_vcf_value(), the only route to it and the only layer holding the record a report has to name; this parser sees a value, not a row.

config_scoredefs is what the resource’s own scores: block declared, and overrides the header for the fields it names: description, aggregators and NA values all take the config’s value when it gives one, falling back to the header’s. The value TYPE and the value PARSER are overridable only together, and only for a field whose header declares a scalar Number (_SCALAR_VALUED_NUMBERS). An entry that leaves type: unstated takes neither, so it reads exactly what the header-only resource reads (gain#1221). A field the header declares MULTI-VALUED takes neither either, whatever type: says: it keeps the header’s converter, because that converter IS the field’s |-join and a value type cannot describe a tuple (gain#1233), and it keeps the str that join produces, because a type is not merely descriptive – it selects the histogram, and a joined field declaring int aborted its own statistics build in np.isnan (gain#1259). An entry stating such a type is REFUSED by _refuse_overridden_type() – it contradicts the header, and gain#1336 raises rather than discarding it. Column addressing is NOT overridable – a VCF score is its INFO key, so col_name/col_index always come from the header side – which is why an entry whose id the header does not declare is a contradiction too, REFUSED by _refuse_undeclared_id() before anything else is asked of it (gain#1489), and why an entry that states an address OTHER than its id – a differing column_name:, or any column_index: – is one as well, REFUSED by _refuse_overridden_address() (gain#1498); an address equal to the id is redundant and passes. The fourth refusal, _refuse_genotype_arity(), is the header’s own claim rather than the config’s (gain#1258). The order is id, arity, address, type: the arity check runs BEFORE the two config-side ones because a per-genotype field is unreadable whatever the entry states and neither of their advised edits can change that, and the address check precedes the type check because a wrong address is wrong whatever type is stated. resource_id is threaded in for those four messages alone.

merge decides what happens to header fields the config does not mention: False (the default) returns only the configured scores, so the config acts as a filter; True keeps the rest as the header defined them. It is the resource’s merge_vcf_scores setting.

Module contents

gain.genomic_resources.get_resource_implementation_builder(resource_type: str) → Callable[[GenomicResource], GenomicResourceImplementation] | None[source]

Return an implementation builder for a certain resource type.

If the builder is not registered, then it will search for an entry point in the found implementations list. If an entry point is found, it will be loaded and registered and returned.

gain.genomic_resources.register_implementation(resource_type: str, builder: Callable[[GenomicResource], GenomicResourceImplementation]) → None[source]

Register a resource type with a given builder function.

The builder has to be a builder function which takes a genomic resource and returns a ready to use implementation. The type is the type of resource to which this builder will be mapped. This is usually the “type” field in the resource’s config.