Repositories and resources

A Genomic Resource Repository (GRR) is a tree of directories, each of which may be a resource: a genomic_resource.yaml config plus the data files it names. In Python that tree is a GenomicResourceRepo, and each directory it serves is a GenomicResource.

A GenomicResource is deliberately generic: it can tell you the resource’s declared type through get_type(), but it has no behaviour specific to any of them — it reads config and files and stops there. Turning one into a reference genome or a score is the job of the typed builders described in the following pages, which is why every one of them takes a repository and an id rather than a bare path.

Opening a repository

build_genomic_resource_repository() called with no arguments builds the repository named by the user’s environment — in a standard installation, the public IossifovLab GRR. Passing a definition dictionary instead builds exactly the repository it describes:

from gain.genomic_resources.repository_factory import build_genomic_resource_repository

# the environment's default repository
grr = build_genomic_resource_repository()

# one specific directory, as a repository of its own
local = build_genomic_resource_repository({
    "id": "local_grr",
    "type": "directory",
    "directory": "/data/my-grr",
})

The definition dictionary is the same structure the .grr_definition.yaml file holds. Genomic resources and repositories documents its keys, the repository types available, and the cache_dir key — caching is a key any type can carry, not a type of its own.

Finding a resource

Two lookups differ only in what they do when nothing matches: get_resource() raises, and find_resource() returns None. Use get_resource when the id is a constant in your program and its absence is a bug; use find_resource when the id came from outside and absence is an ordinary outcome.

Both accept a version_constraint in the same spelling the annotation configuration uses, so a script can pin the resource it was written against:

res = grr.get_resource("hg38/scores/phastCons100way", version_constraint=">=0")
print(res.get_full_id(), res.get_type())

A constraint nothing satisfies is a lookup failure, not a silently older resource — get_resource raises and find_resource returns None, the same way they treat an unknown id. Check the resource’s actual version before pinning: many published resources are still at 0, so a plausible looking >=1.0 will simply fail to resolve.

To go the other way — from a property to the resources that have it — use search_resources(), which takes a full-text term, a resource type, or a wildcard query over resource ids and labels, and yields matches lazily:

for res in grr.search_resources(resource_type="position_score"):
    print(res.get_id())

get_all_resources() is the unfiltered form. Both are generators over a potentially large repository, so prefer to consume them lazily rather than materialising a list.

Reading a resource

A resource exposes its metadata through small accessors — get_config(), get_type(), get_labels(), get_description() — and its files through a family of open_* methods that hide where the resource actually lives. The same call works whether the repository is a local directory, an HTTP mirror or an S3 bucket:

with res.open_raw_file("statistics/histogram_phastCons100way.json") as infile:
    print(infile.read())

open_raw_file() is the general one; the typed openers (open_tabix_file(), open_vcf_file(), open_fasta_file(), open_bigwig_file()) return the corresponding pysam or pyBigWig object, fetching the index alongside the data file where the format needs one.

The manifest

Every resource carries a Manifest — the list of its files with their sizes and checksums. It is what lets a repository detect that a file has changed without reading it, and what a cached or mirrored repository compares against when it decides whether its copy is stale.

get_manifest() returns it, building one if the resource has none. The build reads every file, so where it is possible at all it is expensive — and it needs a read-write protocol, so on a read-only mount a resource with no stored manifest fails here rather than building one. get_loaded_manifest() is the cheap counterpart — it returns the stored manifest or None, and never builds one.

API

gain.genomic_resources.repository_factory.build_genomic_resource_repository(definition: dict | None = None, file_name: str | None = None) → GenomicResourceRepo

Build a GRR using a definition dict or yaml file.

class gain.genomic_resources.repository.GenomicResourceRepo(repo_id: str)

Abstract base class for genomic resource repositories.

A repository manages a collection of genomic resources, providing methods to discover, retrieve, and (for writable repos) create resources.

Repositories can be:

• Protocol-based: Direct access to a single storage backend

• Group: Aggregates multiple child repositories

• Cached: Wraps another repository with local caching

All repositories support resource lookup with optional version constraints:

repo.get_resource(“hg19/genome”) # Latest version repo.get_resource(“hg19/genome”, “>=2.0”) # Version 2.0 or higher repo.get_resource(“hg19/genome”, “=2.1”) # Exact version 2.1

Variables:

• repo_id – Unique identifier for this repository

• definition – Configuration dict used to create this repository

close() → None

Release any resources held by this repository.

property definition: dict[str, Any] | None

Get a copy of the repository configuration definition.

Returns:

Deep copy of definition dict, or None if not set

abstractmethod find_resource(resource_id: str, version_constraint: str | None = None, repository_id: str | None = None) → GenomicResource | None

Return one resource with id qual to resource_id.

If resource is not found, None is returned.

repository_id selects a repository by id under the same rule as get_resource() – a repository answers to its own id, and a falsy id is no filter.

abstractmethod get_all_resources() → Generator[GenomicResource, None, None]

Return a generator over all resource in the repository.

abstractmethod get_resource(resource_id: str, version_constraint: str | None = None, repository_id: str | None = None) → GenomicResource

Return one resource with id qual to resource_id.

If resource is not found, exception is raised.

repository_id restricts the lookup to the repository carrying that id, anywhere in this repository’s tree – including this repository itself: every repository answers to its own id, so passing repo.repo_id is equivalent to passing nothing (#447). A falsy repository_id is no filter at all.

abstractmethod invalidate() → None

Clear cached state and force reload on next access.

Implementations should clear any cached resource lists, metadata, or file contents to ensure fresh data is loaded.

property repo_id: str

Get the repository identifier.

Returns:

Repository ID string

abstractmethod search_resources(search_term: str | None = None, resource_type: str | None = None, resource_query: str | None = None) → Generator[GenomicResource, None, list[tuple[str, str]] | None]

Search resources by FTS term, type and/or wildcard query.

All supplied filters conjoin.

The generator’s return value carries the (repository id, reason) pairs of the children a group skipped while still answering (ADR 0012, gain#686); None and [] both mean nothing was skipped. A for loop discards it, which is exactly right for a caller that does not present totals.

search_resources_by_child(search_term: str | None = None, resource_type: str | None = None, resource_query: str | None = None) → Generator[tuple[GenomicResourceRepo, GenomicResource], None, list[tuple[str, str]] | None]

Search, pairing each hit with the repository that serves it.

For a repository that serves resources itself the answer is always this one, which is what this implementation says. A group overrides it to name the child the resource actually came from, so a caller that has to label a hit – grr_manage list prints the id beside every row – does not have to take a group apart to find out.

The filters mean exactly what they mean for search_resources(), which is the projection of this – and the return value carries the same skips.

class gain.genomic_resources.repository.GenomicResource(resource_id: str, version: tuple[int, ...], protocol: ReadOnlyRepositoryProtocol | ReadWriteRepositoryProtocol, config: dict[str, Any] | None = None, manifest: Manifest | None = None)

Represents a single genomic resource with metadata and file access.

A genomic resource is a versioned collection of data files with a configuration file (genomic_resource.yaml) that defines its type, description, and resource-specific settings.

Common resource types include:

• genome: Reference genome sequences

• gene_models: Gene annotations and transcript models

• position_score: Position-based genomic scores

• allele_score: Variant effect scores

• gene_score: Gene-level scores

Variables:

• resource_id – Unique identifier like “hg19/gene_models/refseq”

• version (tuple[int, ...]) – Version tuple like (1, 2, 3)

• config – Configuration dictionary from genomic_resource.yaml

• proto – Repository protocol for accessing resource files

file_exists(filename: str) → bool

Check if filename exists in this resource.

get_config() → dict[str, Any]

Return the resource configuration.

Raises ValueError if the resource carries no config. The return type is not optional and this never returns None, so a caller has nothing to re-check (gain#1010).

get_description() → str

Return resource description.

get_file_content(filename: str, *, uncompress: bool = True, mode: str = 't') → Any

Return the content of file in a resource.

get_file_url(filename: str) → str

The URL of filename in this resource, per its protocol.

A filesystem path for a directory repository, an http(s):// or s3:// URL otherwise. The name is validated; the file need not exist.

get_full_id() → str

Return a string combining resource ID and version.

Returns a string of the form aa/bb/cc(3.2) for a genomic resource with id aa/bb/cc and version 3.2. If the version is 0 the string will be aa/bb/cc.

This is also the resource’s path component under a repository’s url: a protocol addresses the resource’s directory by joining this string onto the repository root, so the suffix is part of where a versioned resource is stored, not merely of how it is displayed.

get_id() → str

Return genomic resource ID.

get_labels() → dict[str, Any]

Return resource labels.

meta and meta.labels are both free-form YAML, so what is in either is whatever the curator wrote – a scalar, a list and an int are all things a resource can declare, and only the resource types that run the base schema are refused for it. Both levels are narrowed rather than trusted: a non-mapping reads as no labels and is reported, so that every caller sees a mapping whatever the resource says (gain#654). The outer level is narrowed by get_meta(), which every meta reader shares (gain#1004); this is the inner half of the two-tier narrowing described there, and it holds to the same never-validates, never-raises contract.

The values are returned as written. How a value is read – a list as a set of alternatives, everything else as its str() – is label_alternatives().

get_loaded_manifest() → Manifest | None

Return the resource manifest without ever building one.

get_manifest() falls back to building the manifest on a read-write protocol – an md5 scan of every byte of the resource that also writes .grr/*.state files, and that fails outright on a read-only GRR mount. A pure read path that merely wants to consult the manifest uses this instead and copes with None.

get_manifest() → Manifest

Load resource manifest if it exists. Otherwise builds it.

get_memo_key() → tuple[str, str, str]

Return a key identifying everything this resource denotes.

For memoising an object built from a resource – gene models, a gene score, a gene set collection, a liftover chain. Two resources that would build different objects have different keys.

The config is part of the key because a resource at the repository root, spelled ".", takes its whole meaning from it: its id and repository url alone do not separate it from another root resource over the same directory. The repository url is part of the key too, so the same resource reached through two repositories is memoised once per repository.

The key is finer than value identity, never coarser, so a miss costs a rebuild rather than a wrong answer. Raises ValueError if the resource carries no config.

get_meta() → dict[str, Any]

Return the resource’s meta block, narrowed to a mapping.

meta is free-form YAML, so what is there is whatever the curator wrote – meta: | followed by prose parses to a string, and only the resource types that run the base schema are refused for it. It is narrowed rather than trusted, so that every reader of the block sees a mapping whatever the resource says: a non-mapping reads as absent metadata and is reported.

The narrowing is SHALLOW – it promises a mapping at the top level and says nothing about what any field inside holds, which is just as free-form. A reader of a field narrows that field itself, the way get_labels() narrows meta.labels on top of this and get_description settles for str() on whatever it finds.

This is the single seam through which the meta block is read – by get_description, get_summary, get_labels and the FTS index-row collector alike – because narrowing it in one reader and not the others is what gain#1004 was: get_labels coped while the two beside it raised a bare AttributeError, which aborted the repository-wide index build outright.

Narrowing the block is not on its own enough to make every derived value agree. The index row used to collect description and summary out of this mapping with its own .get calls, so it missed the description fall-back get_summary() applies and a resource carrying a description and no summary indexed an empty summary column (gain#1008). Both now derive through _description_in() and _summary_in(), which are the single spelling of what each field is: a reader holding this block derives from those rather than reaching into it, so a second spelling cannot drift from the accessors again.

Reading never validates (ADR 0008) and never raises: this is on the path of every repository-wide walk – a label search, the index build, grr_manage list – and one malformed resource must cost that walk only itself (gain#464, gain#503).

get_public_url() → str

Return this resource’s address on the GRR’s public mirror.

get_repo_public_url() → str

Return repository’s URL.

get_repo_url() → str

Return repository’s URL.

get_summary() → str | None

Return resource summary.

get_type() → str

Return resource type as defined in ‘genomic_resource.yaml’.

get_url() → str

Return this resource’s address on the repository’s own url.

get_version_str() → str

Return version string of the form ‘3.1’.

invalidate() → None

Clean up cached attributes like manifest, etc.

open_bigwig_file(filename: str) → Any

Open a bigwig file and return it.

open_fasta_file(filename: str, index_filename: str | None = None, compressed_index_filename: str | None = None) → FastaFile

Open a bgzipped fasta file and return a pysam.FastaFile.

open_raw_file(filename: str, mode: str = 'rt', **kwargs: str | bool | None) → IO

Open a file in the resource and returns a File-like object.

open_tabix_file(filename: str, index_filename: str | None = None) → TabixFile

Open a tabix file and returns a pysam.TabixFile.

open_vcf_file(filename: str, index_filename: str | None = None) → VariantFile

Open a vcf file and returns a pysam.VariantFile.

class gain.genomic_resources.repository.Manifest

Manages file listings and checksums for a genomic resource.

A manifest maintains a catalog of all files in a resource with their sizes and MD5 checksums. This enables data integrity verification, efficient caching, and incremental updates.

The manifest is typically stored in a .MANIFEST file within the resource directory and is automatically loaded when accessing the resource.

add(entry: ManifestEntry) → None

Add or update a manifest entry.

Parameters:

entry – ManifestEntry to add to the manifest

static from_file_content(file_content: str) → Manifest

Create a manifest from raw YAML file content.

Parameters:

file_content – YAML-formatted string containing manifest entries

Returns:

Manifest object with entries parsed from the content

static from_manifest_entries(manifest_entries: list[dict[str, Any]]) → Manifest

Create a manifest from parsed manifest entry dictionaries.

Parameters:

manifest_entries – List of dicts with ‘name’, ‘size’, ‘md5’ keys

Returns:

Manifest object populated with the provided entries

get_files() → list[tuple[str, int]]

Get list of all files with their sizes.

Returns:

List of (filename, size) tuples for all files in manifest

names() → set[str]

Get set of all filenames in the manifest.

Returns:

Set of filenames tracked by this manifest

to_manifest_entries() → list[dict[str, Any]]

Convert manifest to list of dictionaries for serialization.

Returns:

List of dictionaries with ‘name’, ‘size’, ‘md5’ keys, sorted by filename

update(entries: dict[str, ManifestEntry]) → None

Add or update multiple manifest entries.

Parameters:

entries – Dictionary mapping filenames to ManifestEntry objects