Adding new resource types

GAIn supports different types of genomic resources, such as reference genomes, genomic scores, and gene set collections. New resource types can also be added without modifying GAIn itself. This allows external Python packages to extend the kinds of data that a Genomic Resource Repository (GRR) can manage.

Each resource type is handled by a resource implementation, a Python class that defines how GAIn works with resources of that type. In particular, the implementation describes the resource, generates its HTML summary pages, and defines how its statistics are computed.

New resource types are made available to GAIn through Python entry points. This is different from adding a new annotator, which uses a separate extension mechanism described in Annotator plugins.

Implementing a resource type

A resource implementation subclasses GenomicResourceImplementation. Its constructor receives the GenomicResource object representing the resource.

The subclass must implement five abstract methods, covering two aspects of resource management.

Describing the resource. These methods provide the information displayed in the repository browser and determine when it needs to be updated:
- get_info() generates the HTML describing the resource.
- get_statistics_info() generates the HTML presenting its statistics.
- calc_info_hash() calculates a hash of the inputs used to generate the resource information. GAIn uses this hash to determine whether the information needs to be regenerated.
Computing statistics. These methods define how resource statistics are calculated and determine when they need to be updated:
- create_statistics_build_tasks() defines the tasks needed to calculate statistics for the resource.
- calc_statistics_hash() calculates a hash of the inputs used to compute statistics. This allows grr_manage to skip recalculation when the statistics are already up to date.

The create_statistics_build_tasks() method must accept two keyword-only arguments, region_size and grr, even if the implementation does not use them. These arguments are passed consistently by grr_manage to different resource implementations. The method should not accept arbitrary additional keyword arguments, so unexpected arguments produce an error rather than being silently ignored.

A resource implementation must also provide the files property, which identifies the files it uses. GAIn relies on this information when retrieving files for caching or mirroring repositories. Omitting a file may cause it to be unavailable in a cached copy. Resource publishing uses the resource manifest instead.

Registering a resource type

GAIn discovers resource implementations through the Python entry-point group gain.genomic_resources.implementations. Each registration associates a resource type name with a builder that creates its implementation.

The resource type name corresponds to the type: field in a resource’s genomic_resource.yaml file. A builder is a callable that takes a GenomicResource object and returns the corresponding resource implementation. Since an implementation class can be instantiated directly from a resource, the class itself can serve as its builder.

GAIn registers its built-in resource types in core/pyproject.toml. For example, the genome resource type is registered as follows:

[project.entry-points."gain.genomic_resources.implementations"]
genome = "gain.genomic_resources.implementations.reference_genome_impl:ReferenceGenomeImplementation"

An external package can register a new resource type in its own pyproject.toml:

[project.entry-points."gain.genomic_resources.implementations"]
my_resource_type = "my_package.my_impl:MyResourceImplementation"

This registration tells GAIn to use the MyResourceImplementation class from my_package.my_impl for resources whose type is my_resource_type.

The following example illustrates the structure of a resource implementation:

from gain.genomic_resources import GenomicResource
from gain.genomic_resources.repository import GenomicResourceRepo
from gain.genomic_resources.resource_implementation import (
    DEFAULT_STATISTICS_REGION_SIZE,
    GenomicResourceImplementation,
)
from gain.task_graph.graph import TaskDesc


class MyResourceImplementation(GenomicResourceImplementation):
    """Resource implementation for my_resource_type."""

    def __init__(self, resource: GenomicResource):
        super().__init__(resource)

    @property
    def files(self) -> set[str]:
        return {self.config["filename"]}

    def get_info(self, **kwargs):
        ...

    def get_statistics_info(self, **kwargs):
        ...

    def calc_info_hash(self) -> bytes:
        ...

    def calc_statistics_hash(self) -> bytes:
        ...

    def create_statistics_build_tasks(
        self, *,
        region_size: int = DEFAULT_STATISTICS_REGION_SIZE,
        grr: GenomicResourceRepo | None = None,
    ) -> list[TaskDesc]:
        ...

The example shows the required interface rather than a complete implementation. The methods must be filled in with the logic appropriate for the new resource type.

Once the package is installed, GAIn can discover the registered implementation. Resources using the new type can then be handled by grr_manage and the repository browser without additional configuration.

Looking up and registering builders

GAIn also provides functions for accessing and registering resource implementations directly in Python.

get_resource_implementation_builder() looks up the builder associated with a resource type. It first checks the in-process registry and then searches the installed entry points. Discovered builders are cached for subsequent use. The function returns None if the resource type is not registered.

register_implementation() registers a builder directly in the current Python process, without requiring a package entry point. This is useful for testing or for implementations defined and used within the same script.

For example:

from gain.genomic_resources import (
    get_resource_implementation_builder,
    register_implementation,
)

register_implementation("my_resource_type", MyResourceImplementation)

builder = get_resource_implementation_builder("my_resource_type")
impl = builder(repo.get_resource("my/resource/id"))

Here, register_implementation() associates the resource type name with its implementation class. The lookup function retrieves that class as a builder, which is then used to create an implementation object for a resource.

Direct registration applies only to the current Python process and requires the registration code to have been executed. For resource types distributed as installable Python packages, entry-point registration is therefore preferred.

Multiple resource type names can also refer to the same implementation. For example, GAIn registers both fragment_score and the deprecated cnv_collection type with FragmentScoreImplementation. The deprecated name remains supported for compatibility with existing repositories, although using it produces a warning.

Because resource type names share a common registry, external packages should choose distinctive names to avoid conflicts with built-in or other third-party resource types.

API

The classes below are defined in the module gain.genomic_resources.resource_implementation. The functions are defined in the package gain.genomic_resources. gain.grr exports none of them.

`GenomicResourceImplementation`(genomic_resource)	Base class of the implementation of a resource type.
`GenomicResourceImplementation.calc_info_hash`()	Compute the digest of the inputs of the info page.
`GenomicResourceImplementation.calc_statistics_hash`()	Compute the digest of the inputs of the statistics build.
`GenomicResourceImplementation.create_statistics_build_tasks`(*)	Create the task graph tasks that compute the statistics.
`GenomicResourceImplementation.files`	The names of the resource files that the implementation reads.
`GenomicResourceImplementation.get_config`()	Return the configuration of the resource.
`GenomicResourceImplementation.get_info`(**kwargs)	Render the HTML of the info page of the resource.
`GenomicResourceImplementation.get_statistics`()	Load the statistics of the resource.
`GenomicResourceImplementation.get_statistics_info`(...)	Render the HTML of the statistics page of the resource.
`GenomicResourceImplementation.reload_statistics`()	Drop the cached statistics and load them again.
`GenomicResourceImplementation.resource_id`	The id of the resource that the implementation wraps.

class GenomicResourceImplementation(genomic_resource: GenomicResource)[source]

Base class of the implementation of a resource type.

A resource is a folder in a repository. An implementation knows how to describe the resource, how to compute its statistics and how to render its HTML pages. A subclass overrides the five abstract methods and, when it reads files, the files property.

Parameters:

genomic_resource (GenomicResource) – The resource that the implementation wraps.

Variables:

resource (GenomicResource) – The wrapped resource.
config (dict) – The configuration read from the resource when the implementation is constructed. A subclass that validates its configuration may replace it with the validated form.

abstractmethod calc_info_hash() → bytes[source]

Compute the digest of the inputs of the info page.

The page is rendered again only when the digest changes.

Returns:: The digest of everything that get_info depends on.
Return type:: bytes

abstractmethod calc_statistics_hash() → bytes[source]

Compute the digest of the inputs of the statistics build.

grr_manage compares the digest with the stored one to decide whether it must compute the statistics again.

Returns:: The digest of everything that the statistics tasks read.
Return type:: bytes

abstractmethod create_statistics_build_tasks(*, region_size: int = 3000000000, grr: GenomicResourceRepo | None = None) → list[TaskDesc][source]

Create the task graph tasks that compute the statistics.

grr_manage calls every kind with the same two keyword-only arguments. An implementation declares both, even when it ignores them, and declares no other argument.

Parameters:

region_size (int) – The longest region that one task covers. A kind that splits its build by genomic region reads it. 0 means do not split: a genomic score then scans the resource in one pass and a reference genome uses one region for each chromosome. A kind that does not split ignores it.
grr (GenomicResourceRepo | None) – The repository that holds the resource. None means that the caller names no repository.

Returns:

The tasks of the build. An empty list means that the kind has no statistics to compute.

Return type:

list[TaskDesc]

Raises:

ValueError – If region_size is negative.

property files: set[str]: The names of the resource files that the implementation reads.

get_config() → dict[source]

Return the configuration of the resource.

Returns:: The configuration read when the implementation was constructed. A subclass that validates its configuration returns the validated form.
Return type:: dict

abstractmethod get_info(**kwargs: Any) → str[source]

Render the HTML of the info page of the resource.

Parameters:: **kwargs (Any) – Options of the renderer. The caller defines them.
Returns:: The HTML of the page.
Return type:: str

get_statistics() → ResourceStatistics | None[source]

Load the statistics of the resource.

Returns:: The statistics. None means that the implementation has none: this base class always returns None.
Return type:: ResourceStatistics | None

abstractmethod get_statistics_info(**kwargs: Any) → str[source]

Render the HTML of the statistics page of the resource.

Parameters:: **kwargs (Any) – Options of the renderer. The caller defines them.
Returns:: The HTML of the page.
Return type:: str

reload_statistics() → ResourceStatistics | None[source]

Drop the cached statistics and load them again.

Call it after the statistics were rebuilt on disk.

Returns:: What get_statistics() returns. None means that the implementation has no statistics.
Return type:: ResourceStatistics | None

property resource_id: str: The id of the resource that the implementation wraps.

`register_implementation`(resource_type, builder)	Register the builder of a resource type in the process registry.
`get_resource_implementation_builder`(...)	Return the builder of the implementation of a resource type.

register_implementation(resource_type: str, builder: Callable[[GenomicResource], GenomicResourceImplementation]) → None[source]

Register the builder of a resource type in the process registry.

The registration lasts for the life of the process and replaces an earlier builder of the same type.

Parameters:

resource_type (str) – The resource type, the type field of genomic_resource.yaml.
builder (Callable[[GenomicResource], GenomicResourceImplementation]) – A callable that takes a resource and returns a ready to use implementation. An implementation class whose constructor takes only the resource is its own builder.

get_resource_implementation_builder(resource_type: str) → Callable[[GenomicResource], GenomicResourceImplementation] | None[source]

Return the builder of the implementation of a resource type.

The lookup checks the in-process registry first. It then falls back to the entry points of the group gain.genomic_resources.implementations and caches the builder that it loads.

Parameters:: resource_type (str) – The resource type, the type field of genomic_resource.yaml.
Returns:: A callable that takes a resource and returns its implementation. None means that nothing registered the type.
Return type:: Callable[[GenomicResource], GenomicResourceImplementation] | None