Repositories and resources

A Genomic Resource Repository (GRR) organizes genomic resources in a directory hierarchy. Each resource is defined by a genomic_resource.yaml file and may have associated data files. As described in Genomic resources and repositories, a repository can be stored locally or accessed remotely.

In Python, GAIn represents a repository with a GenomicResourceRepo object and an individual resource with a GenomicResource object. A repository object lets you find resources. A resource object provides access to its definition, metadata, and files.

A GenomicResource is not the same as an object that interprets a specific kind of genomic data. For example, it can identify a resource as a reference genome or a position score, but it does not itself retrieve DNA sequences or score values. Those operations belong to specialized Python objects, such as ReferenceGenome and score objects, which are introduced in the following chapters. This chapter covers the repository and generic resource objects used to locate and inspect the underlying resources.

Accessing a repository

The usual way to access a GRR from Python is with build_genomic_resource_repository(), available through gain.grr:

import gain.grr as grr

repo = grr.build_genomic_resource_repository()

With no arguments, this function uses the default GRR configuration for the user’s environment. If no custom configuration is available, a standard GAIn installation uses the public IossifovLab GRR.

You can also provide a repository definition to access a particular GRR. For example, the following code accesses a local directory:

local = grr.build_genomic_resource_repository({
    "id": "local_grr",
    "type": "directory",
    "directory": "/data/my-grr",
})

The dictionary uses the same structure as the .grr_definition.yaml file. The Genomic resources and repositories chapter explains the configuration fields and supported repository types, including cache_dir. Caching can be configured for any repository type.

The build_genomic_resource_repository() function creates a Python object that provides access to a repository’s resources. Multiple repositories can be combined into a group using build_genomic_resource_group_repository(). Within a genomic context, the associated repository can be retrieved using get_grr_from_context().

Finding a resource

Resources are identified by path-like resource IDs. Once you have a repository object, you can retrieve a resource by its ID using get_resource():

resource = repo.get_resource("hg38/scores/phastCons100way")
print(resource.get_full_id(), resource.get_type())

Two methods are available for looking up a specific resource:

get_resource() raises an exception if the resource cannot be found.
find_resource() returns None if no resource matches.

Use get_resource() when a missing resource should stop the program, and find_resource() when a resource may legitimately be absent.

Both methods accept an optional version_constraint, using the same syntax as version constraints in annotation configuration files. For example:

resource = repo.get_resource(
    "hg38/scores/phastCons100way", version_constraint=">=0"
)

If no resource satisfies the version constraint, the lookup fails just as it would for an unknown ID: get_resource() raises, while find_resource() returns None. Many published resources have version 0, so check the available version before requiring a higher one, such as >=1.0.

If you do not know the resource ID, use search_resources(). It supports full-text search terms, resource types, and wildcard queries over resource IDs and labels. For example, to find position-score resources:

for resource in repo.search_resources(resource_type="position_score"):
    print(resource.get_id())

get_all_resources() iterates over all resources without filtering. Both methods return generators, which produce resources as you iterate; for a large repository, this avoids loading the entire result set into a list at once.

Inspecting and reading a resource

A GenomicResource provides information about a resource and access to its files without requiring you to know whether the GRR is stored in a local directory, an HTTP mirror, or an S3 bucket.

Its metadata can be inspected through methods such as get_config(), get_type(), get_labels(), and get_description(). These describe the resource; they do not interpret the genomic data in its files.

To read a file, specify its path within the resource. For example:

with resource.open_raw_file("statistics/histogram_phastCons100way.json") as infile:
    print(infile.read())

open_raw_file() provides general file access. Format-specific methods are also available: open_tabix_file(), open_vcf_file(), open_fasta_file(), and open_bigwig_file(). These return the appropriate pysam or pyBigWig objects and retrieve index files along with the data when the format requires them.

For routine analyses, you will often use a specialized resource object rather than read files directly. For example, the reference-genome and genomic-score chapters explain how to create objects that provide methods for retrieving sequences or scores. The generic resource object is useful when you need to inspect resource metadata or work directly with its files.

Closing and refreshing a repository

A repository object is ready to use as soon as it is built. Unlike many specialized resource objects, it does not require a separate open() call.

close() releases resources held by the repository. After close() is called, the repository’s definition attribute is None.

invalidate() serves a different purpose: it clears the repository’s cached state so that subsequent access reloads the resource list, metadata, and file contents. This can be useful if the underlying repository has changed.

Closing a repository does not automatically close specialized Python objects created from its resources. For example, a reference genome or score object may hold its own open file handles and has its own close() method. The following chapters explain the opening and closing requirements for each resource type.

The resource manifest

Each resource can have a Manifest, which records its files, sizes, and checksums. GAIn uses this information to detect changed files and to determine whether a cached or mirrored copy is out of date, without having to read the full contents of each file during that comparison.

Two methods provide access to a resource’s manifest:

get_loaded_manifest() returns an existing, loaded manifest, or None if there is none. It does not attempt to construct a manifest.
get_manifest() returns the manifest, constructing one if necessary. Constructing a manifest requires reading every file and can therefore be expensive.

Building a missing manifest also requires a read-write protocol. If the repository is available only through a read-only mount, get_manifest() may fail when no stored manifest is available. Use get_loaded_manifest() when you only need to check an existing manifest without triggering that work.

API

The classes below are defined in the module gain.genomic_resources.repository. GenomicResourceRepo and GenomicResource are also exported by gain.grr, as grr.GenomicResourceRepo and grr.GenomicResource.

`GenomicResourceRepo`(repo_id)	Abstract base class for genomic resource repositories.
`GenomicResourceRepo.close`()	Release the resources that the repository holds.
`GenomicResourceRepo.definition`	The definition that built the repository, as a copy.
`GenomicResourceRepo.find_resource`(resource_id)	Return the resource with the given id, or `None`.
`GenomicResourceRepo.get_all_resources`()	Return a generator over all resources of the repository.
`GenomicResourceRepo.get_resource`(resource_id)	Return the resource with the given id.
`GenomicResourceRepo.invalidate`()	Drop the cached state of the repository.
`GenomicResourceRepo.repo_id`	The identifier of the repository.
`GenomicResourceRepo.search_resources`([...])	Search the resources by a full-text term, a type or a query.
`GenomicResourceRepo.search_resources_by_child`([...])	Search the resources, and pair each hit with its repository.
`GenomicResource`(resource_id, version, protocol)	Represents a single genomic resource with metadata and file access.
`GenomicResource.file_exists`(filename)	Check whether a file exists in the resource.
`GenomicResource.get_config`()	Return the configuration of the resource.
`GenomicResource.get_description`()	Return the description of the resource.
`GenomicResource.get_file_content`(filename, *)	Return the content of a file of the resource.
`GenomicResource.get_file_url`(filename)	Return the address of a file of the resource.
`GenomicResource.get_full_id`()	Return the id of the resource together with its version.
`GenomicResource.get_id`()	Return the id of the resource, without the version.
`GenomicResource.get_labels`()	Return the labels of the resource.
`GenomicResource.get_loaded_manifest`()	Return the stored manifest of the resource, and never build one.
`GenomicResource.get_manifest`()	Return the manifest of the resource, and build it if it is missing.
`GenomicResource.get_meta`()	Return the `meta` block of the resource.
`GenomicResource.get_public_url`()	Return the address of the resource on the public mirror.
`GenomicResource.get_repo_public_url`()	Return the public url of the repository that serves the resource.
`GenomicResource.get_repo_url`()	Return the url of the repository that serves the resource.
`GenomicResource.get_summary`()	Return the summary of the resource.
`GenomicResource.get_type`()	Return the type of the resource.
`GenomicResource.get_url`()	Return the address of the resource on its repository.
`GenomicResource.get_version_str`()	Return the version of the resource as a string.
`GenomicResource.open_bigwig_file`(filename)	Open a bigWig file of the resource.
`GenomicResource.open_fasta_file`(filename[, ...])	Open a bgzipped FASTA file of the resource.
`GenomicResource.open_raw_file`(filename[, mode])	Open a file of the resource.
`GenomicResource.open_tabix_file`(filename[, ...])	Open a tabix file of the resource.
`GenomicResource.open_vcf_file`(filename[, ...])	Open a VCF file of the resource.
`Manifest`()	Manages file listings and checksums for a genomic resource.
`Manifest.from_file_content`(file_content)	Create a manifest from the YAML content of a manifest file.
`Manifest.get_files`()	Return the name and size of each file in the manifest.
`Manifest.names`()	Return the names of the files in the manifest.

class GenomicResourceRepo(repo_id: str)[source]

Abstract base class for genomic resource repositories.

A repository manages a collection of genomic resources, providing methods to discover, retrieve, and (for writable repos) create resources.

Repositories can be:

Protocol-based: Direct access to a single storage backend
Group: Aggregates multiple child repositories
Cached: Wraps another repository with local caching

All repositories support resource lookup with optional version constraints:

repo.get_resource(“hg19/genome”) # Latest version repo.get_resource(“hg19/genome”, “>=2.0”) # Version 2.0 or higher repo.get_resource(“hg19/genome”, “=2.1”) # Exact version 2.1

Build a repository with grr.build_genomic_resource_repository. The properties repo_id and definition describe the repository.

close() → None[source]

Release the resources that the repository holds.

After the call, definition is None.

property definition: dict[str, Any] | None

The definition that built the repository, as a copy.

None means that nobody set a definition, or that the repository is closed.

abstractmethod find_resource(resource_id: str, version_constraint: str | None = None, repository_id: str | None = None) → GenomicResource | None[source]

Return the resource with the given id, or None.

Use this method when the id comes from outside and absence is an ordinary result. Use get_resource() when absence is a bug.

Parameters:

resource_id (str) – The id of the resource, without the version.
version_constraint (str or None, default None) – A constraint such as ">=2.0". None selects the latest version.
repository_id (str or None, default None) – Restricts the lookup to the repository with this id. The rule is the same as for get_resource().

Returns:

The matching resource. None means that no resource matches the id, the constraint and the repository id.

Return type:

GenomicResource or None

abstractmethod get_all_resources() → Generator[GenomicResource, None, None][source]

Return a generator over all resources of the repository.

Read the generator lazily. A repository can be large.

Yields:: GenomicResource – A resource of the repository. A repository without resources yields nothing.

abstractmethod get_resource(resource_id: str, version_constraint: str | None = None, repository_id: str | None = None) → GenomicResource[source]

Return the resource with the given id.

Use this method when the id is a constant and its absence is a bug. Use find_resource() when absence is an ordinary result.

Parameters:

resource_id (str) – The id of the resource, without the version.
version_constraint (str or None, default None) – A constraint such as ">=2.0" or "=2.1". None selects the latest version.
repository_id (str or None, default None) – Restricts the lookup to the repository with this id, anywhere in the tree of this repository. An empty string or None means no restriction.

Returns:

The matching resource. The method never returns None.

Return type:

GenomicResource

Raises:

FileNotFoundError – If no resource matches and the repository is backed by a protocol (directory, http, s3, in-memory).
ValueError – If no resource matches and the repository is a group repository.

abstractmethod invalidate() → None[source]

Drop the cached state of the repository.

The next access loads the resource list, the metadata and the file contents again.

property repo_id: str: The identifier of the repository.

abstractmethod search_resources(search_term: str | None = None, resource_type: str | None = None, resource_query: str | None = None) → Generator[GenomicResource, None, list[tuple[str, str]] | None][source]

Search the resources by a full-text term, a type or a query.

The method returns a generator, so it reads the repository lazily. A resource must match every filter that you supply.

Parameters:

search_term (str or None, default None) – A full-text term. None means no full-text filter.
resource_type (str or None, default None) – A resource type, for example position_score. None means no type filter.
resource_query (str or None, default None) – A wildcard query over the ids and the labels of the resources. None means no query filter.

Yields:

GenomicResource – A matching resource. With no filter, every resource matches.

Returns:

The generator returns, when it ends, a (repository id, reason) pair for each child that a group repository skipped while it still answered. None and an empty list both mean that nothing was skipped. A for loop discards this value.

Return type:

list of tuple of (str, str) or None

Raises:

SearchTermError – If the search term cannot be parsed.
SearchIndexUnavailableError – If the repository has no usable search index.

search_resources_by_child(search_term: str | None = None, resource_type: str | None = None, resource_query: str | None = None) → Generator[tuple[GenomicResourceRepo, GenomicResource], None, list[tuple[str, str]] | None][source]

Search the resources, and pair each hit with its repository.

The filters mean the same as for search_resources().

Parameters:

search_term (str or None, default None) – A full-text term. None means no full-text filter.
resource_type (str or None, default None) – A resource type. None means no type filter.
resource_query (str or None, default None) – A wildcard query over the ids and the labels. None means no query filter.

Yields:

tuple of (GenomicResourceRepo, GenomicResource) – A matching resource and the repository that serves it. For a group, this is the child that holds the resource.

Returns:

The skipped children, as for search_resources().

Return type:

list of tuple of (str, str) or None

class GenomicResource(resource_id: str, version: tuple[int, ...], protocol: ReadOnlyRepositoryProtocol | ReadWriteRepositoryProtocol, config: dict[str, Any] | None = None, manifest: Manifest | None = None)[source]

Represents a single genomic resource with metadata and file access.

A genomic resource is a versioned collection of data files with a configuration file (genomic_resource.yaml) that defines its type, description, and resource-specific settings.

Common resource types include:

genome: Reference genome sequences
gene_models: Gene annotations and transcript models
position_score: Position-based genomic scores
allele_score: Variant effect scores
gene_score: Gene-level scores

Variables:

resource_id – Unique identifier like “hg19/gene_models/refseq”
version (tuple[int, ...]) – Version tuple like (1, 2, 3)
config – Configuration dictionary from genomic_resource.yaml
proto – Repository protocol for accessing resource files

file_exists(filename: str) → bool[source]

Check whether a file exists in the resource.

Parameters:: filename (str) – The name of the file, relative to the resource directory.
Returns:: True if the file exists, else False.
Return type:: bool

get_config() → dict[str, Any][source]

Return the configuration of the resource.

Returns:: The content of genomic_resource.yaml. The method never returns None.
Return type:: dict
Raises:: ValueError – If the resource has no config.

get_description() → str[source]

Return the description of the resource.

Returns:: The description field of the meta block. The string is empty when the resource has no description.
Return type:: str

get_file_content(filename: str, *, uncompress: bool = True, mode: str = 't') → Any[source]

Return the content of a file of the resource.

Parameters:

filename (str) – The name of the file, relative to the resource directory.
uncompress (bool, default True) – If True, decompress a compressed file before the method returns the content.
mode (str, default "t") – "t" returns the content as str. "b" returns bytes.

Returns:

The content of the file. The value is empty when the file is empty.

Return type:

str or bytes

Raises:

FileNotFoundError – If the file does not exist.

get_file_url(filename: str) → str[source]

Return the address of a file of the resource.

The method does not check that the file exists.

Parameters:: filename (str) – The name of the file, relative to the resource directory.
Returns:: A filesystem path for a directory repository. Otherwise, an http(s):// or s3:// url.
Return type:: str
Raises:: ValueError – If the name is not a valid file name for a resource.

get_full_id() → str[source]

Return the id of the resource together with its version.

Returns:: aa/bb/cc(3.2) for the id aa/bb/cc and the version 3.2. The string is aa/bb/cc when the version is 0.
Return type:: str

get_id() → str[source]

Return the id of the resource, without the version.

Returns:: The id, for example hg38/gene_models/MANE/1.3.
Return type:: str

get_labels() → dict[str, Any][source]

Return the labels of the resource.

Returns:: The labels mapping of the meta block, with the values as the curator wrote them. The dictionary is empty when the resource has no labels or when labels is not a mapping. The second case is also reported in the log.
Return type:: dict

get_loaded_manifest() → Manifest | None[source]

Return the stored manifest of the resource, and never build one.

Returns:: The stored manifest. None means that the resource has no stored manifest.
Return type:: Manifest or None

get_manifest() → Manifest[source]

Return the manifest of the resource, and build it if it is missing.

The build reads every file of the resource. It needs a read-write protocol. Use get_loaded_manifest() to avoid the build.

Returns:: The stored manifest, or a new manifest when the resource has none. The method never returns None.
Return type:: Manifest
Raises:: Exception – If the resource has no stored manifest and the protocol is read-only, for example a read-only mount.

get_meta() → dict[str, Any][source]

Return the meta block of the resource.

Returns:: The meta block of the config. The dictionary is empty when the block is absent, is null or is not a mapping. A block that is not a mapping is also reported in the log. The method does not validate the fields inside the block.
Return type:: dict

get_public_url() → str[source]

Return the address of the resource on the public mirror.

Returns:: The public url of the repository joined with the full id of the resource.
Return type:: str

get_repo_public_url() → str[source]

Return the public url of the repository that serves the resource.

Returns:: The public address of the repository, as set in its definition.
Return type:: str

get_repo_url() → str[source]

Return the url of the repository that serves the resource.

Returns:: A filesystem path for a directory repository, or an http(s):// or s3:// url.
Return type:: str

get_summary() → str | None[source]

Return the summary of the resource.

Returns:: The summary field of the meta block. When the resource has no summary, the method returns the description. None means that the resource has neither.
Return type:: str or None

get_type() → str[source]

Return the type of the resource.

Returns:: The type field of genomic_resource.yaml, for example position_score. A resource without the field has the type basic.
Return type:: str
Raises:: ValueError – If the resource has no config.

get_url() → str[source]

Return the address of the resource on its repository.

Returns:: The url of the repository joined with the full id of the resource.
Return type:: str

get_version_str() → str[source]

Return the version of the resource as a string.

Returns:: The version in the form 3.1.
Return type:: str

open_bigwig_file(filename: str) → Any[source]

Open a bigWig file of the resource.

Parameters:: filename (str) – The name of the bigWig file.
Returns:: The open file. Close it after use.
Return type:: pyBigWig.pyBigWig
Raises:: FileNotFoundError – If the file does not exist.

open_fasta_file(filename: str, index_filename: str | None = None, compressed_index_filename: str | None = None) → FastaFile[source]

Open a bgzipped FASTA file of the resource.

Parameters:

filename (str) – The name of the bgzipped FASTA file.
index_filename (str or None, default None) – The name of the .fai index. None means the name of the data file with the suffix .fai.
compressed_index_filename (str or None, default None) – The name of the .gzi index of the compressed file. None means the name of the data file with the suffix .gzi.

Returns:

The open file. Close it after use.

Return type:

pysam.FastaFile

Raises:

ValueError – If the .gzi index of the compressed file does not exist.
FileNotFoundError – If the data file or the .fai index does not exist.

open_raw_file(filename: str, mode: str = 'rt', **kwargs: str | bool | None) → IO[source]

Open a file of the resource.

The same call works for a local directory, an HTTP mirror and an S3 bucket. Close the returned object, or use it in a with block.

Parameters:

filename (str) – The name of the file, relative to the resource directory.
mode (str, default "rt") – The open mode. Text and binary read modes work for every repository. Write modes work for a read-write repository.
**kwargs – Options that the protocol passes to the open call.

Returns:

A file-like object.

Return type:

IO

Raises:

FileNotFoundError – If the file does not exist and the mode reads.

open_tabix_file(filename: str, index_filename: str | None = None) → TabixFile[source]

Open a tabix file of the resource.

Parameters:

filename (str) – The name of the bgzipped data file.
index_filename (str or None, default None) – The name of the index file. None means that the method looks for the .tbi index and then for the .csi index next to the data file.

Returns:

The open file. Close it after use.

Return type:

pysam.TabixFile

Raises:

FileNotFoundError – If the data file or the index file does not exist.

open_vcf_file(filename: str, index_filename: str | None = None) → VariantFile[source]

Open a VCF file of the resource.

Parameters:

filename (str) – The name of the VCF file.
index_filename (str or None, default None) – The name of the index file. None means that the method looks for the index next to the data file.

Returns:

The open file. Close it after use.

Return type:

pysam.VariantFile

Raises:

FileNotFoundError – If the data file or the index file does not exist.

class Manifest[source]

Manages file listings and checksums for a genomic resource.

A manifest maintains a catalog of all files in a resource with their sizes and MD5 checksums. This enables data integrity verification, efficient caching, and incremental updates.

The manifest is typically stored in a .MANIFEST file within the resource directory and is automatically loaded when accessing the resource.

static from_file_content(file_content: str) → Manifest[source]

Create a manifest from the YAML content of a manifest file.

Parameters:

file_content (str) – YAML text that holds a list of entries with the keys name, size and md5. Empty YAML text gives an empty manifest.

Returns:

The manifest with one entry for each item of the list.

Return type:

Manifest

Raises:

KeyError – If an item of the list has no name, size or md5 key.
yaml.YAMLError – If the text is not valid YAML.

get_files() → list[tuple[str, int]][source]

Return the name and size of each file in the manifest.

Returns:: A (name, size) pair for each file. The list is empty when the manifest has no entries.
Return type:: list of tuple of (str, int)

names() → set[str][source]

Return the names of the files in the manifest.

Returns:: The file names. The set is empty when the manifest has no entries.
Return type:: set of str

The repository builders below are defined in the module gain.genomic_resources.repository_factory. They are also exported by gain.grr, as grr.build_genomic_resource_repository, grr.build_genomic_resource_group_repository, grr.get_default_grr_definition and grr.get_default_grr_definition_path.

`build_genomic_resource_repository`([...])	Build a repository from a definition dictionary or a YAML file.
`build_genomic_resource_group_repository`(...)	Build a repository that groups other repositories.
`get_default_grr_definition`()	Return the definition of the default repository.
`get_default_grr_definition_path`()	Return the path of the default repository definition file.

build_genomic_resource_repository(definition: dict | None = None, file_name: str | None = None) → GenomicResourceRepo[source]

Build a repository from a definition dictionary or a YAML file.

Call the function with no arguments to build the default repository. The environment decides which repository that is: see get_default_grr_definition().

Parameters:

definition (dict or None, default None) – The definition of the repository. The key type selects the kind of repository, for example directory, http, s3 or group. The key cache_dir adds a local cache to any kind. None or an empty dictionary means that the function takes the definition from file_name or from the default definition.
file_name (str or None, default None) – The path of a YAML file that holds the definition. Do not pass it together with a non-empty definition.

Returns:

The repository. Its definition property holds the definition that built it.

Return type:

GenomicResourceRepo

Raises:

ValueError – If both definition and file_name are given, if no definition is found, or if the definition is not valid. The message does not contain passwords.

build_genomic_resource_group_repository(repo_id: str, children: list[GenomicResourceRepo]) → GenomicResourceRepo[source]

Build a repository that groups other repositories.

A lookup in the group asks its children.

Parameters:

repo_id (str) – The identifier of the group repository.
children (list of GenomicResourceRepo) – The repositories to group. They must be built already.

Returns:

The group repository. It has no definition, because the function takes repositories and not a definition.

Return type:

GenomicResourceRepo

get_default_grr_definition() → dict[str, Any][source]

Return the definition of the default repository.

The function loads the file that get_default_grr_definition_path() finds. When there is no such file, it returns a copy of the built-in definition of the public IossifovLab GRR.

Returns:

A definition that build_genomic_resource_repository accepts. The caller can change the dictionary.

Return type:

dict

Raises:

OSError – If the definition file cannot be read.
yaml.YAMLError – If the definition file is not valid YAML.

get_default_grr_definition_path() → str | None[source]

Return the path of the default repository definition file.

The environment variable GRR_DEFINITION_FILE has priority. The function does not check that the file it names exists. Otherwise, the function looks for .grr_definition.yaml in the home directory.

Returns:: The path of the definition file. None means that the variable is not set and the home directory has no .grr_definition.yaml.
Return type:: str or None

The genomic-context functions below are defined in the module gain.genomic_resources.genomic_context. They are also exported by gain.grr, as grr.get_genomic_context and grr.get_grr_from_context.

`get_genomic_context`()	Return a genomic context that merges every registered context.
`get_grr_from_context`(context)	Return the repository that a genomic context exposes.

get_genomic_context() → GenomicContext[source]

Return a genomic context that merges every registered context.

When two contexts expose the same key, the context that was registered last wins.

Returns:: A PriorityGenomicContext over a snapshot of the registered contexts. A context registered later is not part of it.
Return type:: GenomicContext

get_grr_from_context(context: GenomicContext) → GenomicResourceRepo[source]

Return the repository that a genomic context exposes.

Parameters:: context (GenomicContext) – The context to ask, for example the result of get_genomic_context().
Returns:: The repository of the context. The function never returns None.
Return type:: GenomicResourceRepo
Raises:: ValueError – If the context exposes no repository.