Genomic resources and repositories

A Genomic Resource Repository (GRR) is a collection of genomic resources (e.g., genomes, gene models, scores, and gene sets) stored either locally (on disk) or remotely (over the network). GAIn uses GRRs as the backing store for resources during annotation and analysis.

Repository discovery

A GRR configuration file, also called a GRR definition file, is a small YAML file that tells GAIn which Genomic Resource Repositories (GRRs) to use and in what order to search them. It does not contain genomic data itself. Instead, it points to the repositories where resources live, such as local directories or remote URLs, and can also describe how those repositories should be combined and cached.

GAIn command-line tools determine which GRR definition to use by checking several sources in order. A GRR definition passed directly with the -g or --grr command-line option has the highest priority and applies only to that command:

grr_browse -g /path/to/my_grr_definition.yaml
annotate_tabular -g /path/to/my_grr_definition.yaml input.tsv pipeline.yaml

If no command-line GRR definition is provided, GAIn next checks the GRR_DEFINITION_FILE environment variable,

export GRR_DEFINITION_FILE=/path/to/my_grr_definition.yaml
grr_browse

If GRR_DEFINITION_FILE is not set, GAIn then checks the default ~/.grr_definition.yaml file in the user’s home directory.

If none of these are available, GAIn falls back to the public IossifovLab GRR.

To configure which GRRs GAIn uses by default, create a file named .grr_definition.yaml in your home directory. The example below points GAIn to the public IossifovLab GRR (a remote repository accessed via URL):

id: development
type: group
children:
- id: GRR
  type: url
  url: https://grr.iossifovlab.com

If .grr_definition.yaml contains the next example, GAIn will resolve resources from your local directory-based GRR, such as one created in “Getting Started in GRR”. This overrides the default behavior, so the public IossifovLab GRR will no longer be used unless you add it explicitly.

id: development
type: group
children:
- id: grr_local
  type: directory
  directory: [path to my_grr]/my_grr

The configuration below defines two GRRs and searches them in order. When GAIn resolves a resource ID, it first queries the GRR with id GRR. If the resource is not found there, GAIn then queries the GRR with id grr_local.

id: development
type: group
children:
- id: GRR
  type: url
  url: https://grr.iossifovlab.com
- id: grr_local
  type: directory
  directory: [path to my_grr]/my_grr

Repository configuration

A repository configuration is a YAML mapping that describes a single repository. Every repository has a required id and type, plus additional fields depending on the repository type. A repository can be a concrete repository, such as a local directory or remote URL, or a group that combines several child repositories.

Common fields

Field	Requirement	Description
id	Required string	Identifier for the repository.
type	Required string	Repository type. Allowed values are directory, http, url, s3, embedded, and group.
cache_dir	Optional string	Path to a local filesystem directory used to cache downloaded resources. May be added to any repository type, including a group. It must be a plain path, not a URL – a remote cache target (s3://..., http://...) is not supported and is rejected.

Repository types

Type	Required field	Description
directory	directory	A GRR stored in a local directory on disk. The directory field must be an absolute path; relative paths are rejected. The aliases dir and file are accepted as synonyms of directory.
url	url	A general-purpose remote repository. The URL scheme selects the protocol; http, https, and s3 are supported.
http	url	A remote HTTP(S) repository. This is like url but restricted to http and https URLs.
s3	url	A remote S3 repository. This is like url but restricted to s3:// URLs.
embedded	content	An in-memory repository whose resources are defined inline in the configuration. The content field is a nested mapping describing files and directories. Directory values are nested mappings; file values are file contents. The alias memory is accepted as a synonym of embedded.
group	children	A collection of repositories. The children field is a list of repository configurations. Each child can be a concrete repository or another group. Groups can be nested.

Within a group, the first repository that contains the requested resource wins. Order children accordingly: list a local directory before a remote repository if local resources should take precedence, or after it if the remote repository should be authoritative.

Repository caching

When a repository is configured with a cache_dir option, GAIn caches resources locally before using them. This matters because many genomic resources are large (often hundreds of MB to many GB), and repeatedly downloading or streaming them from a remote GRR can be slow and network-dependent.

With caching enabled, the first use of a resource may take longer while GAIn downloads it into cache_dir. After that, GAIn reuses the cached copy, which is typically much faster and avoids repeated network transfers. This is especially useful for resources you access frequently (for example, common reference genomes, gene models, or widely used scores).

cache_dir can be attached to any repository, including a group. When attached to a group, it caches resources served through that group, which provides a convenient way to use a single cache in front of several repositories.

The tradeoff is disk usage: cached resources can occupy substantial space, so choose a cache_dir location with enough capacity (and keep in mind that the cache may grow over time as you use more resources).

The cache itself must live on a local filesystem. cache_dir is a plain directory path, never a URL: GAIn serialises concurrent downloads into the cache with a lockfile, which only provides mutual exclusion locally. A cache_dir that carries a URL scheme (for example s3://bucket/cache or http://host/cache) is rejected when the repository is built. The source repository may of course be remote – s3, http and url repositories are all cacheable, into a local cache_dir.

For example, to cache resources from a single remote repository, add cache_dir directly to that repository entry. Replace <path_to_grr_cache> with the full path to the directory where cached resources should be stored.

id: main-GRR
type: url
url: https://grr.iossifovlab.com
cache_dir: <path_to_grr_cache>/main-GRR

In this configuration, resources loaded from main-GRR are cached under <path_to_grr_cache>/main-GRR.

To use one cache for a group of remote repositories, add cache_dir to the group entry:

id: development
type: group
cache_dir: <path_to_grr_cache>/remote_grrs
children:

- id: main-GRR
  type: url
  url: https://grr.iossifovlab.com

- id: GRR-ENCODE
  type: url
  url: https://grr-encode.iossifovlab.com

In this configuration, resources resolved through the development group are cached under <path_to_grr_cache>/remote_grrs.

Complete GRR definition example

The configuration below combines several features described above. It defines a top-level group with two children: a nested group of remote repositories that share a cache directory, followed by a local directory-based GRR.

In this example, GAIn searches main-GRR first, then GRR-ENCODE, and finally My_First_GRR. The first repository that contains the requested resource is used.

type: group
id: my_GRRs
children:
- type: group
  id: remote_GRRs
  cache_dir: <path_to_cache>/remote_grr_cache
  children:
  - id: main-GRR
    type: url
    url: https://grr.iossifovlab.com

  - id: GRR-ENCODE
    type: url
    url: https://grr-encode.iossifovlab.com

- id: My_First_GRR
  type: directory
  directory: <path_to_My_First_GRR>/My_First_GRR

To use this configuration, save it as ~/.grr_definition.yaml, point GRR_DEFINITION_FILE to it, or pass it explicitly with -g:

grr_browse -g my_grr_definition.yaml

Repository management

GAIn provides two command-line tools for working with genomic resources and repositories. Their usage is outlined below.

Command	Description
grr_manage	Create, inspect, and maintain GRRs, including manifests, statistics, information pages, and repair operations.
grr_browse	Browse the resources available through a GRR definition file.

grr_manage

grr_manage is the Genomic Resource Repository Management Tool. It is used to create, inspect, and maintain GRRs.

Usage:

grr_manage [-h] [--version] [--verbose] [--logfile LOGFILE]
           {list,repo-init,repo-manifest,resource-manifest,repo-stats,resource-stats,repo-info,resource-info,repo-repair,resource-repair,repo-index} ...

Commands:

Command	Description
list	List a GR Repo.
repo-init	Initialize a directory to turn it into a GRR.
repo-manifest	Create/update manifests for whole GRR.
resource-manifest	Create/update manifests for a resource.
repo-stats	Build the statistics for a resource.
resource-stats	Build the statistics for a resource.
repo-info	Build the index.html for the whole GRR.
resource-info	Build the index.html for the specific resource.
repo-repair	Update/rebuild manifest and histograms whole GRR.
resource-repair	Update/rebuild manifest and histograms for a resource.
repo-index	Publish the repository index (.CONTENTS files, search index, repository index pages) from the manifests already on disk.

The two scopes differ in more than how resources are selected. A resource-* command writes inside the selected resources’ directories (plus task logs under .task-log); the repository-global artifacts — .CONTENTS.json.gz, the search index and the repository index pages — are left as they were, so after it writes anything it notes that the repository index is stale. The repo-* commands republish global artifacts at the end of their runs — repo-repair and repo-info all three, repo-stats the .CONTENTS files and the search index, repo-manifest the .CONTENTS files only — and repo-index does only that: it rebuilds all three groups from the manifests already on disk, verifying nothing and writing nothing inside any resource directory. A resource without a committed manifest is left out of the index, reported by id, and fails the repo-index run.

The repository index page’s search runs on SQLite compiled to WebAssembly, and the pages set their text in Roboto and draw their icons from a Material Symbols subset. Whichever command publishes the pages publishes all three beside them, under .static/ at the repository root (sqlite-wasm-<version>/ and fonts/) — so a published repository carries everything its pages need and loads nothing from a CDN or a font host at view time; on an intranet or behind an air gap the pages render as they do on the web. The files change only when gain’s vendored copies do; a rerun on an unchanged repository leaves them untouched. Publish (or commit, for a repository kept in git) the .static directory along with index.html.

A repo-* command runs on an empty repository too, and publishes there exactly the artifacts listed above — so after deleting the last resource of a repository, repo-repair (or repo-index) leaves none of the artifacts it publishes still advertising it. Each command still settles only its own, exactly as on a repository that has resources in it; repo-fix-histograms republishes nothing unless it fixed or failed on something. Note that a search index rebuilt with no resource in it holds no contents at all, so searching that repository reports the index as unavailable rather than answering that nothing matched.

Options:

Option	Description
-h, --help	Show this help message and exit.
--version	Prints the GAIn version and exits.
--verbose, -v, -V	Enable verbose output.
--logfile LOGFILE	File to log output to. If not set, logs to console.

Rebuilding on demand: --force and --dry-run

The *-manifest, *-stats, *-info and *-repair commands accept these two, in both their repository- and resource-scoped forms. They are mutually exclusive: given both, the tool warns and fails rather than guessing which one you meant. (repo-fix-histograms writes inside resource directories too but offers neither flag, and repo-init and repo-index rebuild nothing.)

Option	Description
-n, --dry-run	Report what would be rebuilt and write nothing — no manifest and no recorded file state, so the run leaves the repository byte-identical and seeds nothing for the next run to reuse. Exits with the number of resources needing an update.
-f, --force	Act regardless of whether anything looks out of date: save the manifest even when it is already current, and rebuild the statistics even when the stored stats_hash still matches. Note that this does not make GAIn distrust the file states it has recorded — a file whose size and timestamp are unchanged still keeps its recorded md5 sum. Re-hashing file content is what -D/--without-dvc does.

Why forcing is sometimes the only option. Statistics are rebuilt only when a resource’s stats_hash no longer matches what its configuration and data hash to — the gate that makes re-running these commands over a large repository cheap. It also means a resource is not rebuilt merely because a newer GAIn would compute more statistics for it than the version that last built it: such a resource has a perfectly current hash and is skipped, and its info page reports the newer statistics as not computed.

--force is the deliberate way out. To put a newer GAIn’s statistics onto one already-built resource:

grr_manage resource-stats -r <resource_id> -R <repository_path> -f

Mind the scope. Beyond the resource-* limits described above, resource-stats does not re-render the resource’s own info page either — so the statistics it writes are on disk but not yet displayed. To rebuild the statistics and the pages that show them, use the info command instead:

grr_manage resource-info -r <resource_id> -R <repository_path> -f

and follow up with grr_manage repo-index when the repository-global artifacts should reflect the change too. The repository-scoped repo-stats -f and repo-repair -f do the same for every resource at once, which on a large GRR is a full recomputation — reach for the per-resource form unless you mean it.

grr_browse

grr_browse is the Genomic Resource Repository Browse Tool. It is used to browse and filter the resources available through a GRR definition file.

Usage:

grr_browse [-h] [--version] [--verbose] [--logfile LOGFILE] [-g GRR] [-s SEARCH] [-t TYPE] [--summary] [--bytes]

Options:

Option	Description
-h, --help	Show this help message and exit.
--version	Prints the GAIn version and exits.
--verbose, -v, -V	Enable verbose output.
--logfile LOGFILE	File to log output to. If not set, logs to console.
--bytes	Print the resource size in bytes.

Repository/Resource options:

Option	Description
-g, --grr GRR	Path to GRR definition file.
-s, --search SEARCH	FTS search term to filter resources.
-t, --type TYPE	Filter resources by type.
--summary	Print a summary for each resource below its listing line.

Searching resources

Every GRR carries a full-text search (FTS) index of its resources. This index is what backs the -s/--search option of grr_browse and the resource search of the GAIn web interface. Because the index includes the meta.labels of each resource, labels are searchable: any label key you define becomes a field you can filter on.

The search index

The index is stored at the root of the repository as .CONTENTS.sqlite3.gz — a gzipped SQLite database holding a single FTS5 virtual table named contents.

The index is built and refreshed by grr_manage repo-index (and by repo-stats, repo-info and repo-repair at the end of their runs). The rebuild is skipped when the repository contents are unchanged (the index records the md5 of the .CONTENTS file it was built from), so re-running any of these on an untouched repository is cheap.

Indexed fields

Each resource contributes one row. Five fields are always present:

Field	Description
full_id	The fully qualified resource id, including version.
id	The resource id.
type	The resource type (position_score, genome, …).
description	The meta.description of the resource.
summary	The meta.summary of the resource, or its meta.description when that summary is absent or empty. This is the value the resource reports as its summary and the one the repository index page displays, so summary : <term> finds a description-only resource by a term from its description.

Resource implementations may add their own fields — score resources, for example, contribute score_ids and score_descriptions.

In addition, every key of ``meta.labels`` becomes a field of its own, holding that resource’s value for the label. A resource configured like this:

type: position_score

meta:
  summary: Example conservation score
  labels:
    reference_genome: hg38/genomes/GRCh38-hg38
    assay_term_name: ChIP-seq

contributes the fields reference_genome and assay_term_name alongside the fixed five.

The field set of a repository is the union of the fields of all its resources: a label defined by a single resource becomes a field of the whole index, empty for every other resource. This means the searchable fields differ from repository to repository, and a query naming a field that no resource in the target repository defines is an error rather than an empty result (see Limitations below).

Searching by label

The value of -s/--search is passed to FTS5 verbatim, so the full FTS5 query syntax is available. A label is queried with the <field> : <value> column filter:

# all resources whose reference_genome label mentions hg38
grr_browse -g grr_definition.yaml -s 'reference_genome : hg38'

# phrase values must be quoted
grr_browse -g grr_definition.yaml -s 'assay_term_name : "ChIP-seq"'

# labels combine with each other and with the other fields
grr_browse -g grr_definition.yaml -s 'target : CTCF AND assay_term_name : "ChIP-seq"'

# NOT, OR and prefix matching all work
grr_browse -g grr_definition.yaml -s 'assay_term_name : "ChIP-seq" NOT target : CTCF'
grr_browse -g grr_definition.yaml -s 'biosample_summary : liv*'

A search term with no field prefix is matched against all fields at once, labels included:

grr_browse -g grr_definition.yaml -s 'liver'

--search and --type are independent and combine with AND. Unlike --search, --type is an exact match on the resource type, not a full-text match:

grr_browse -g grr_definition.yaml -t position_score -s 'reference_genome : hg38'

List-valued labels

A label value may be a list as well as a scalar. A resource that carries more than one value for one label — a multiome h5ad measuring both gene expression and chromatin accessibility — is labelled by what it is, rather than forced into a single value:

meta:
  labels:
    protocol: 10x_Multiome
    modality:
      - RNA
      - ATAC
    file_format: h5ad

The list is read as a set of alternatives by every reader of the label:

• The full-text index stores the elements joined by a space, so -s 'modality : RNA' and -s 'modality : ATAC' both find the resource.

• A label clause of a -q resource query — [modality="RNA"], [modality="R*"], ["RNA" in modality] — holds if it holds for any element. Each element is compared in its rendered form exactly as a scalar value is, and a label the resource does not carry still reads as "". “Has both” is two clauses on the one key, since every clause of a query must hold: -q '*[modality="RNA" and modality="ATAC"]'. There is no operator for “only RNA, nothing else”.

• The resource info page lists the elements comma-separated: modality: RNA, ATAC.

An empty list reads as "", the same as an absent label. Only a list is split this way; a nested mapping is a single value and is compared and indexed as its rendered text. The labels GAIn itself reads — reference_genome, source_genome and target_genome — name one resource each and must stay non-empty strings (see Genomic resource configuration below).

Limitations

Label search inherits the constraints of the underlying FTS5 index. The following are worth knowing before relying on it.

Label keys must be valid SQLite identifiers. Each key is used directly as a column name of the contents table, so a key must be a string that matches [A-Za-z_][A-Za-z0-9_]*, must not be an SQL keyword, and must not be a name FTS5 reserves (rank, rowid, contents). A key containing a hyphen or a space (cell-type, cell type), or one spelled order, cannot name a field; use underscores (cell_type). Beware YAML’s unquoted scalars — 2024: release and true: yes are an integer and a boolean key, not strings, and are refused as well.

Label keys must not collide with a field of the index. The index reserves full_id, id, type, description and summary, which every resource contributes, and score_ids and score_descriptions, which score resources contribute. None of them may be a label key — on any resource, whether or not its own type contributes the field. On a resource that does contribute it the label would replace that field’s value, which then could no longer be found by it; on one that does not, the label would land in a column that means something else for every resource that does, since the index has one column per name for the whole repository.

Two resources must not spell one key two ways. The index has one set of columns for the whole repository — the union of the fields of all its resources — and SQLite compares column names case-insensitively. So assay in one resource and Assay in another are one column asked for under two spellings, and cannot both be in the index, even though each resource is perfectly fine on its own. The same holds for the total: a repository whose resources have more than 1994 distinct field names between them is past what an FTS5 table can hold.

A resource that breaks any of these rules is skipped: grr_manage repo-repair reports it by resource id — naming the offending key, and, for a clash of spellings, the resource that already holds the name — indexes the rest of the repository normally, and exits non-zero. Only the skipped resources are missing from the index; fix their meta.labels and repair again. Resources join the index in resource id order, so of two resources spelling a key differently it is always the one that is later by id that is skipped.

If every resource is skipped, the index is left with nothing in it: searches then return no results and log a warning, rather than failing.

Search terms are FTS5 expressions, not literal strings. Characters that are meaningful to the query parser — most commonly - and : — must be quoted, or the search fails with an error rather than returning no results:

grr_browse -g grr_definition.yaml -s 'ChIP-seq'      # error: no such column: seq
grr_browse -g grr_definition.yaml -s '"ChIP-seq"'    # correct

A query naming an unknown field is an error. assay_term_name : foo fails with no such column: assay_term_name against a repository in which no resource carries that label, rather than matching nothing.

Matching is by token, not by equality. Label values are tokenized like any other text, so reference_genome : hg38 matches the resource whose label value is hg38/genomes/GRCh38.p14. This is usually what you want, but it is not an exact-value filter; there is no faceted or exact-match search over label values.

Version control for GRRs

GRRs can be managed under version control using a combination of Git, DVC, and grr_manage. In this setup, small files such as genomic_resource.yaml, .MANIFEST files, histogram metadata, and .dvc tracking files are stored in Git, while large resource files are stored with DVC. The grr_manage tool is then used to generate or update GRR metadata, including manifests, resource statistics, histograms, and HTML info pages.

This organization makes it possible to track both the structure and content of a GRR while avoiding the need to store large genomic data files directly in Git.

Initializing version control

A version-controlled GRR starts as a directory-based GRR. In the Create your first GRR section of the “Getting started with GRR” page, we created a local GRR named my_GRR and initialized it with grr_manage repo-init. We can now place that GRR directory under Git and DVC control.

Git is used for small files, such as genomic_resource.yaml, .MANIFEST, histogram metadata, and .dvc pointer files. DVC is used for large genomic resource files.

From the directory that contains my_GRR, enter the GRR root directory and initialize Git and DVC:

cd my_GRR
git init
dvc init

After initializing DVC, configure a DVC remote where large files will be stored. The remote can be a shared filesystem, SSH server, cloud bucket, or another DVC-supported storage backend. For example:

dvc remote add -d myremote <remote_url>

After initializing Git locally, the GRR directory can optionally be connected to a remote Git repository, such as a private or public GitHub repository, so that the GRR structure and metadata can be shared with other users:

git remote add origin git@github.com:<organization>/<repository>.git
git add .
git commit -m "Initialize version-controlled GRR"
git push -u origin main

Only small files and DVC pointer files should be committed to Git. Large genomic resource files should be added with dvc add and stored in the configured DVC remote.

Once this is done, the GRR can be managed using the same pattern as other version-controlled data repositories: small files are committed to Git, large resource files are added with dvc add and pushed with dvc push, and grr_manage is used to regenerate manifests, statistics, histograms, and info pages after changes. The public IossifovLab GRR is managed using this same approach.

Adding a resource to version control

In the Add new resources to the local section of “Getting started with GRR” page, we added a large (>5Gb) position score resource (PhyloP7) named my_positionscore to my_GRR. The resource directory contains the score file and its genomic_resource.yaml configuration file:

my_GRR/
└── my_positionscore/
    ├── hg38.phyloP7way.bw
    └── genomic_resource.yaml

In a version-controlled GRR, the large resource file should be added to DVC, while the small configuration file should be added to Git. From the root of my_GRR:

cd my_positionscore
dvc add hg38.phyloP7way.bw

This creates a .dvc pointer file for the large resource file. The .dvc file and the resource configuration should be added to Git:

git add hg38.phyloP7way.bw.dvc
git add genomic_resource.yaml
git commit -m "Add my_positionscore resource"

The large resource file itself should be pushed to the configured DVC remote:

dvc push

After adding the resource, return to the GRR root directory and run

grr_manage resource-repair to generate or update the resource manifest, histograms, and other derived files:

cd ..
grr_manage resource-repair

then grr_manage repo-index to publish the updated resource in the repository index:

grr_manage repo-index

The generated files should also be added to Git:

git add my_positionscore/.MANIFEST
git add my_positionscore/histograms/
git add .CONTENTS.json.gz .CONTENTS.sqlite3.gz
git commit -m "Add my_positionscore generated metadata"
git push

In this workflow, Git tracks the resource structure, configuration, DVC pointer files, manifests, and histogram outputs, while DVC stores the large resource data file itself.

Metadata-only updates

Metadata-only changes are simpler than changes to the underlying resource data. For example, suppose we want to update the summary, description, labels, or score descriptions in the genomic_resource.yaml file for the my_score resource. Because the large score file itself is not changing, we do not need to download or modify the DVC-managed resource file.

Edit the resource configuration file:

cd my_GRR
vi my_positionscore/genomic_resource.yaml

After editing genomic_resource.yaml, run the repair command from the GRR root directory:

grr_manage resource-repair
grr_manage repo-index

This updates the resource manifest and any derived metadata that depend on the configuration, and republishes the repository index. Then add the changed files to Git:

git add my_positionscore/genomic_resource.yaml
git add my_positionscore/.MANIFEST
git add .CONTENTS.json.gz .CONTENTS.sqlite3.gz
git commit -m "Update my_positionscore metadata"
git push

This workflow is efficient because metadata files are small and stored directly in Git, while large genomic resource files remain in DVC and do not need to be downloaded or modified for metadata-only updates.

Genomic resource configuration

GAIn supports a large number of genomic resource types (for example, genomes, gene models, and position scores). Each resource lives in its own folder within a GRR and includes the resource files plus a genomic_resource.yaml configuration file. In the sections below, we describe the configuration options available for each resource type.

All genomic_resource.yaml files share the same top-level structure: the first line sets the resource type (a string that determines how GAIn interprets the resource), and an optional meta section can provide human-readable metadata via summary, description, and labels.

type: <genomic resource type>

# resource-specific configuration

meta:
  summary: <(string) Short summary of the resource>
  description: <(string) Longer description of the resource>
  labels: <(dictionary) Arbitrary key/value pairs>

Labels are not merely descriptive: every label key becomes a searchable field of the repository’s full-text index, so meta.labels is how a resource is made discoverable by properties that are not part of its id or type. See Searching resources for the query syntax and for the naming constraints that label keys must satisfy. A value may be a scalar or a list of scalars; a list is read as a set of alternatives, so a resource that is several things at once can say so (see List-valued labels). Some labels are also read by GAIn itself — gene models and scores use reference_genome to declare the assembly they are built against, and liftover chains use source_genome and target_genome.

Each of those three labels names another resource, so its value must be a non-empty string. Label values are free-form YAML and nothing validates them, so a value that cannot be a resource id — a number, a list, a nested mapping, or an empty string — is read as if the label were absent, and GAIn logs a warning naming the resource, the label and what it found instead. The read itself never fails; what follows is whatever that resource does when unlabelled. A score builds its info page and its statistics from what it can measure itself (see below). A liftover chain then has no genomes to offer, so a liftover_annotator that does not name source_genome and target_genome itself will fail to build, saying which parameter it is missing. Quote an id that YAML would otherwise read as a number:

meta:
  labels:
    reference_genome: "2019"   # unquoted, this is the integer 2019

Note that the value is used exactly as written — surrounding whitespace is not trimmed, so reference_genome: " hg38 " is a non-empty string that simply names no resource, and fails at resolution rather than being read as absent.

For a score, the reference_genome label does two things. It is the top rung of the ladder that gives every contig of the score a length, and it is what makes the score’s resource info page answer what part of the reference genome has values.

The length ladder is reference_genome label → bigWig header → tabix estimate, applied per contig of the score: a contig the labelled genome lists takes the genome’s exact length; a contig it does not list falls through, for that contig alone, to what the score’s own file can say — a bigWig header is exact, a tabix index yields only an upper bound found by probing it, and a plain table knows how far its rows reach. The ladder runs at repair, when grr_manage resource-stats (or repo-stats) splits the score’s contigs into the regions its statistics are scanned by; nothing is stored, and a repair whose statistics are already current never asks. The label is deliberately not part of the statistics hash: adding, removing or re-pointing it rebuilds no histogram, and the next full rebuild simply splits on the genome the label names then.

The Coverage section of the page reports covered positions as a percentage of the whole assembly — every contig of the labelled genome, including the ones the score never touches, which are rolled up into a single “N contigs with no values” row. Without a resolvable reference_genome label there is nothing to divide by, and a tabix-backed score renders raw covered-position counts and no percentages at all. A bigWig-backed score is the one exception: its header carries exact contig sizes, so it renders percentages against the header’s contig list even unlabelled. If a score’s Coverage section shows counts where you expected percentages, add the label:

meta:
  labels:
    reference_genome: hg38/genomes/GRCh38-hg38

The percentage is computed when the page is rendered, never stored, so adding the label needs no data rebuild — re-render the page (grr_manage resource-info -r <resource_id>) and the percentages appear. Do not add -f for this: it forces a full statistics rebuild, histograms included.

A label naming a genome that does not resolve degrades the whole section back to raw counts rather than rendering a wrong percentage. So does a single contig the genome gets wrong: if the score holds positions past the end of the contig the genome declares, or the genome declares it empty, that contig’s row and the global percentage both fall back to raw counts, while the remaining contigs keep theirs. Either way the page shows counts rather than a number you cannot trust.

Two properties of the percentage are worth knowing before reading one. It is measured against every contig of the labelled assembly, alternate haplotypes and decoys included, so a whole-genome score built against the primary assembly reads somewhat below 100%. And a bigWig-backed score without a label is measured against its own header, which is a statement about the file rather than about the genome — a chr21-only bigWig reads as nearly fully covered until you label it.

While describing genomic_resource.yaml configuration options, we will first cover the resource types whose genomic_resource.yaml files are relatively simple (genome, gene models, liftover chains, and annotation pipelines). Next, we will cover position score and allele score resources, whose configuration files are typically more complex because the underlying data files are large and often follow resource-specific conventions. To support these cases, we introduce additional options for table and column matching, histogram configuration, and annotation defaults. Finally, we cover gene scores (which are similar to position and allele scores) and gene sets, which have their own resource-specific configuration in genomic_resource.yaml.

Genomes

Genome resources use a reference assembly FASTA and (optionally) provide assembly-specific metadata such as chromosome naming conventions and pseudoautosomal regions.

Resource-specific fields in genomic_resource.yaml for genome resources (type: genome) are:

Field	Requirement	Description
filename	Required string	Path to the genome FASTA file, relative to the resource directory.
index_file	Optional string	Path to the FASTA .fai index, relative to the resource directory. Default: <filename>.fai.
chrom_prefix	Optional string	Prefix expected in contig names, for example chr. Default: no prefix.
PARS	Optional subsection	Pseudoautosomal regions for the assembly.

The genome FASTA may be either a plain .fa file or a bgzipped FASTA (.fa.gz or .bgz). GAIn selects how to read the sequence from the file extension — a bgzipped genome is read with random access via pysam.FastaFile — so no extra configuration is required. A plain .fa genome needs only its .fai index; a bgzipped genome must be accompanied by two index files in the resource directory: a .fai FASTA index and a .gzi bgzip block index. We use samtools to create these index files. If samtools is not already available in your environment, install it with:

mamba install -c bioconda -c conda-forge samtools

Both index files are produced together by samtools faidx:

samtools faidx GRCh38.p14.genome.fa.gz

which writes GRCh38.p14.genome.fa.gz.fai and GRCh38.p14.genome.fa.gz.gzi next to the FASTA.

A bgzipped genome is configured exactly like a plain one — only the filename extension differs:

type: genome
filename: GRCh38.p14.genome.fa.gz
chrom_prefix: "chr"

meta:
  summary: Nucleotide sequence of the GRCh38.p14 genome assembly (bgzipped)

Let’s revisit the example genomic_resource.yaml from the Getting started with GRR genome section. As before, filename points to the downloaded FASTA file and contig names use the chr prefix. We also include PARS, which defines the pseudoautosomal regions on chromosomes X and Y.

type: genome
filename: GRCh38.p14.genome.fa
chrom_prefix: "chr"

PARS:
  "X":
    - "chrX:10000-2781479"
    - "chrX:155701382-156030895"
  "Y":
    - "chrY:10000-2781479"
    - "chrY:56887902-57217415"

meta:
  summary: Nucleotide sequence of the GRCh38.p14 genome assembly

Gene models

For gene model resources, the genomic_resource.yaml file has a minimal resource-specific section with only filename and format.

Resource-specific fields (type: gene_models):

Field	Type	Description
filename	string	Path to the gene model file, relative to the resource directory.
format	string	Gene model format. Supported values include default, refflat, refseq, ccds, knowngene, gtf, and ucscgenepred.

In the Getting started with GRR gene models example, the gene model file is a GTF, so we set format: gtf.

type: gene_models

filename: MANE.GRCh38.v1.4.ensembl_genomic.gtf.gz
format: gtf

meta:
  summary: MANE gene model version 1.4

Liftover chains

For liftover chain resources, the genomic_resource.yaml file has a minimal resource-specific section with only filename.

Resource-specific fields (type: liftover_chain):

Field	Type	Description
filename	string	Path to the chain file, relative to the resource directory.

type: liftover_chain
filename: hg38-chm13v2.over.chain.gz
meta:
  summary: Liftover Chain hg38 to T2T

Annotation pipelines

For annotation pipeline resources, the genomic_resource.yaml file has a minimal resource-specific section with only filename.

Resource-specific fields (type: annotation_pipeline):

Field	Type	Description
filename	string	Path to the pipeline YAML file, relative to the resource directory.

type: annotation_pipeline
filename: Clinical_annotation.yaml
meta:
  summary: Clinical Annotation Pipeline

Position scores

Position score resources (type: position_score) use a genomic_resource.yaml file with three resource-specific sections: table, scores, and (optionally) default_annotation.

Note

This section describes the YAML keys. The Python objects behind them — the PositionScore class and the GenomicScoreDef each entry of the scores block becomes — are described in Scores.

table

The table section specifies the data file (filename), its format, and how GAIn should interpret the columns.

Currently supported formats are tabix, vcf_info, tsv, csv, and bw.

The header_mode setting controls how column names (the header) are determined:

Value	Description
file	Extract the header from the file (default).
list	Use the explicit header provided via header.
none	No header is used; columns can only be referenced by index.

The header field is used only when header_mode is set to list. Example:

header_mode: list
header: ["chrom", "start", "end", "score_value"]

The user must tell GAIn which columns correspond to chrom (chromosome), pos_begin (start position), and pos_end (end position). This can be done by column index or by column name.

If the resource file has no header, columns must be specified by index. For example:

table:
  filename: positionscore1.bedGraph.gz
  format: tabix
  header_mode: none
  chrom:
    index: 0
  pos_begin:
    index: 1
  pos_end:
    index: 2

If the resource file includes a header, columns can be specified by name. In the next example, positionscore2.bedGraph.gz has columns named chr and pos:

table:
  filename: positionscore2.bedGraph.gz
  format: tabix
  header_mode: file
  chrom:
    name: chr
  pos_begin:
    name: pos
  pos_end:
    name: pos

The table section also supports chrom_mapping, which can be used to reconcile chromosome naming differences between the resource file and the reference genome. This is useful, for example, when the resource uses contig names like chr1 but the genome uses only numbers.

Three options are available under chrom_mapping:

Option	Description
add_prefix	Takes a string value and adds it as a prefix.
del_prefix	Takes a string value and removes it from the start of each chromosome name.
filename	Takes a filepath, relative to the genomic resource directory, containing the chromosome mapping table.

When filename is used, the file must contain two whitespace-delimited columns. The first line must be a header with the column names chrom and file_chrom. Values in file_chrom are what appear in the resource file, and values in chrom are what they will be mapped to. For example:

chrom           file_chrom
Chromosome_1     1
Chromosome_22    22

An example of using chrom_mapping (useful when the resource uses a chr prefix but the genome does not) is shown below:

table:
...
  chrom_mapping:
    add_prefix: "chr"

scores

The table section configures how the data file is read. The scores section specifies which score columns to extract, how to name them in the GRR, and what data type they should have. For example, the minimal configuration below extracts a float score from column index 2 and stores it under the id my_positionscore1:

scores:
- id: my_positionscore1
  type: float
  index: 2

Alternatively, score columns can be specified by name. In the next example, the score column in the file is named positionscore2, and the extracted score is stored under the id my_positionscore2:

scores:
- id: my_positionscore2
  type: float
  name: positionscore2

Optionally, the user may also add human-readable descriptions. These fields are used on the HTML summary page for the resource. For example:

desc: "conservation score"
large_values_desc: "more conserved"
small_values_desc: "less conserved"

The HTML summary page displays a default histogram for each score. Optionally, the user may provide a histogram configuration to override the default and control how the score distribution is displayed. Histogram configuration options are covered here. The example below shows a custom histogram within a complete scores entry. If the resource includes multiple scores, add additional entries under scores with different id values.

scores:
- id: my_positionscore2
  type: float
  name: positionscore2

  desc: "conservation score"
  large_values_desc: "more conserved"
  small_values_desc: "less conserved"

  histogram:
    type: number
    number_of_bins: 100
    view_range:
      min: 0.0
      max: 1.0
    y_log_scale: True

default_annotation

Annotation pipelines can choose which scores from a resource to use. If a pipeline does not explicitly specify scores for this resource, GAIn falls back to the resource’s default_annotation list. If default_annotation is not provided, all scores in the resource are used by default. An example is shown below:

default_annotation:
- source: my_positionscore2
  name: my_positionscore2

Putting all the pieces together, the following is a complete genomic_resource.yaml example for a position score resource. The optional meta field is omitted for conciseness.

type: position_score                         # resource type

table:                                       # how to read the input table
  filename: positionscore2.bedGraph.gz       # input file (relative path)
  format: tabix                              # file format
  header_mode: file                          # read header from file
  chrom:                                     # chromosome column
    name: chr                                # column name
  pos_begin:                                 # start position column
    name: pos                                # column name
  pos_end:                                   # end position column
    name: pos                                # column name

scores:                                      # how to extract data columns as scores
  - id: my_positionscore2                    # score id stored in GRR
    type: float                              # data type of the score values
    name: positionscore2                     # column name containing the score

    desc: "a description"                    # shown on the HTML summary page
    large_values_desc: "more"                # meaning of larger values (HTML)
    small_values_desc: "less"                # meaning of smaller values (HTML)

    histogram:                               # optional histogram override (HTML)
      type: number                           # numeric histogram
      number_of_bins: 100                    # bin count used in the histogram
      view_range:                            # visible range shown on the x-axis
        min: 0.0                             # minimum visible range in the histogram
        max: 1.0                             # maximum visible range in the histogram
      y_log_scale: True                      # use log scale on the y-axis

default_annotation:                          # default scores used for annotation
  - source: my_positionscore2                # score id to annotate from
    name: my_positionscore2                  # name of the annotation field

Score value types

A score’s type is one of float, int, str or bool, and it decides how each cell of the score’s column is read.

bool accepts a closed set of eight spellings — True, true, TRUE and 1 for true; False, false, FALSE and 0 for false. Any other text is not a boolean: the cell is reported in the log and read as no value, so a column spelled yes/no or T/F annotates as empty rather than being guessed at. A VCF Flag INFO field is typed bool too, and there presence is what means true.

Unlike the numeric types, a bool score declares no na_values by default, so a missing cell (., or an empty one) is reported the same way rather than being silently skipped. A resource whose boolean column is sparse should say so explicitly:

scores:
  - id: my_flag
    type: bool
    name: flag_column
    na_values: ["", "."]

Allele scores

genomic_resource.yaml files for allele score resources are almost exactly the same as for position score resources, with three differences:

1. type: allele_score

2. allele_score_mode must be specified. Options are:

   substitutions: single nucleotide substitutions (for example, C>T)

   allele: covers all allele types (for example, insertions and deletions in addition to substitutions)

3. In the table section, the user must also specify which columns contain the reference and alternative alleles using reference and alternative.

The scores, default_annotation, and meta sections are the same as for position scores. The example below shows the beginning of a valid genomic_resource.yaml for an allele score resource:

type: allele_score
allele_score_mode: substitutions

table:
  filename: AlphaMissense_hg38_modified.tsv.gz
  format: tabix

  chrom:
    name: CHROM
  pos_begin:
    name: POS
  pos_end:
    name: POS
  reference:
    name: REF
  alternative:
    name: ALT

... (scores, default_annotation, and meta sections follow) ...

Fragment scores

genomic_resource.yaml files for fragment score resources are the same as for position score resources, except that the resource type is set to fragment_score.

Note

This resource type was previously called cnv_collection. type: cnv_collection is still accepted but is deprecated: it stops being accepted in GAIn 2027.1.0, and until then every resource declaring it logs a warning naming that resource. Declare type: fragment_score instead.

Fragment scores are coordinate-based, like position scores: they are queried by chromosome and interval and do not model allele changes. A fragment is simply an interval carrying attributes; copy-number variants are the most common thing to store as fragments, which is where the older name came from. Annotation consists of reporting overlapping fragments and the selected associated fields (for example, event class and frequency).

The example below shows a valid genomic_resource.yaml for a fragment score resource holding CNVs (my_CNVcollection.txt), which uses chrom, pos_begin and pos_end as column names for chromosome, beginning position and end position, respectively. It also has a column called deletion_duplication which describes the event type recorded.

type: fragment_score
table:
  filename: my_CNVcollection.txt

scores:
- id: CNV type
  name: deletion_duplication
  type: str
  desc: duplication or deletion

meta:
  summary: fragment score resource

Gene scores

Gene scores are gene-level annotations, such as constraint metrics, expression summaries, or intolerance scores. genomic_resource.yaml files for gene score resources are similar to position score resources, except that the resource type is set to gene_score and there is no table section. The underlying data file is a table with a gene identifier column and one or more score columns. By default the gene identifier column must be named gene; if the file uses a different column name, set gene_column to that name.

In the example genomic_resource.yaml file below, data file gene_scores.tsv contains a required column named gene, plus two score columns named constraint and intolerance. The scores section defines which columns are exposed as scores, and default_annotation works the same way as for position scores.

The HTML summary page displays a default histogram for each score. Optionally, the user may provide a histogram configuration to override the default and control how the score distribution is displayed, as shown for the constraint_score in this example. Histogram configuration options are covered here.

type: gene_score

filename: gene_scores.tsv

gene_column: gene  # optional; defaults to "gene"

scores:
- id: intolerance_score
  desc: Probability of Loss-of-Function Intolerance

- id: constraint_score
  desc: Gene conservation score
  histogram:
    type: number
    number_of_bins: 126
    view_range:
      min: 0
      max: 1
    x_min_log: 0.00001
    x_log_scale: false
    y_log_scale: true

default_annotation:
- source: constraint_score
  name: constraint_score

meta:
  summary: Gene score resource

Gene set collections

A gene_set_collection defines relationships between genes and gene sets. These relationships can be provided either directly as gene sets (gmt format) or as gene-set mappings (map format). In both cases, the underlying structure is the same: a many-to-many association between genes and sets.

In gmt format, each line of the file directly defines a gene set and its member genes. In this format, each row corresponds to a single gene set. The first column defines the set identifier, the second column typically provides a description, and the remaining columns list the genes belonging to that set. No additional processing is required to construct the gene sets.

example.gmt, an example gmt data file:

PATHWAY_A   Description of pathway A    GENE1    GENE2    GENE3
PATHWAY_B   Description of pathway B    GENE2    GENE4

Example genomic_resource.yaml file for a gmt gene set collection resource:

type: gene_set_collection
id: example_gmt
format: gmt
filename: example.gmt

meta:
  summary: Minimal GMT example

In map format, each row defines a relationship between a gene and a gene set. The first column contains the gene identifier, and the second column contains the set identifier. Gene sets are formed by grouping all rows with the same set identifier. A companion file may optionally be provided to associate each set identifier with a human-readable description.

example-map.txt, an example map file:

GENE1   SET_A
GENE2   SET_A
GENE3   SET_A
GENE2   SET_B
GENE4   SET_B

Optional companion file: example-mapnames.txt

SET_A   Pathway A description
SET_B   Pathway B description

Example genomic_resource.yaml file for a map gene set collection resource:

type: gene_set_collection
id: example_map
format: map
filename: example-map.txt

histograms:
  genes_per_gene_set:
    type: number
    y_log_scale: true

  gene_sets_per_gene:
    type: number
    y_log_scale: true

meta:
  summary: Example MAP-based gene set collection

For both gmt and map resources, the optional histograms section can be used to summarize the structure of the collection. For example, genes_per_gene_set describes the distribution of gene set sizes, while gene_sets_per_gene describes how many sets each gene belongs to.

Histogram configuration

Note

This section describes the histogram YAML keys, and the custom plot_function hook below is the one Python example that belongs on this page. The configuration and histogram objects these keys parse into — NumberHistogram, CategoricalHistogram, NullHistogram and their configs — are described in Histograms.

Histograms provide a quick visual summary of how a score is distributed across the genome or across observed variants. Seeing the distribution is often as important as seeing individual values, because it helps interpret what “large” or “small” values typically look like for a given score and whether the score has outliers, heavy tails, or distinct modes.

For each score, the HTML summary page shows a default histogram whenever it is possible to compute one from the underlying data. Histogram configuration is optional. If a score includes a histogram block under scores, GAIn uses it to override the default display and control how the distribution is visualized.

Histogram behavior is controlled by the type field, which selects the histogram implementation. GAIn supports three histogram types: number for numeric scores, categorical for string or discrete category scores, and null to explicitly disable histogram computation/display when a histogram is not meaningful. The value of type must be exactly one of number, categorical, or null.

Some options are shared across number and categorical histogram types. For example, y_log_scale controls whether the y-axis is displayed on a log scale (default: False), which can be helpful when counts vary widely across bins or categories. x_log_scale controls whether the x-axis is displayed on a log scale (default: False). When x_log_scale is set to True, x_min_log defines the minimum x-axis value used for the logarithmic scale. The example below shows a minimal histogram configuration that overrides the default by enabling log-scale display on the y-axis for a numeric score. Other options depend on the selected type and are described in the sections below.

scores:
  - id: myscore
    column_name: RS
    type: int
    desc: a genomic score

    histogram:
      type: number
      y_log_scale: True

Number histograms

Number histograms are used for numeric scores, including continuous-valued scores and integer-valued scores. They are supported for scores of type int and float. By default, the histogram is calculated with 100 bins and uses linear scaling on both axes. They summarize the distribution by grouping values into bins along the x-axis and showing the number of observations per bin.

A number histogram configuration supports two options.

Option	Description
number_of_bins	Number of bins used to partition the score values. Default: 100.
view_range	Visible range on the x-axis using min and max values. This is useful for bounded scores, for example 0-1, or for focusing on the region of interest without being dominated by extreme outliers. Default: show all values.

The example below shows a number histogram configuration with an explicit bin count and visible range.

histogram:
  type: number
  number_of_bins: 10
  view_range:
    min: 0.0
    max: 1.0

Categorical histograms

Here, each value represents a discrete label (e.g., ClinVar clinical significance categories or review-status labels). Categorical histograms are supported for scores of type str and int. This histogram type shows the distribution of unique values in the score and is supported only for scores with fewer than 100 unique values. They summarize the distribution by counting how many observations fall into each unique value and displaying those counts.

A categorical histogram configuration supports five options.

Option	Description
displayed_values_count	The number of unique values that will be displayed in the histogram. Default: 20. The remaining values are grouped into the Other category.
displayed_values_percent	The percentage of total mass of unique values that will be displayed. The remaining values are grouped into the Other category. Only one of displayed_values_count and displayed_values_percent can be set.
label_rotation	Rotation angle for x-axis category labels in degrees. Default: 0.
value_order	The order in which the unique values are displayed in the histogram.
plot_function	Optional custom plotting function used instead of the default categorical histogram rendering. This is useful when the default plot and the available options are not sufficient, for example to reorder, filter, or relabel categories. The value should be provided as <python module>:<python function>, where the Python module path is relative to the resource directory. When plot_function is set, GAIn uses the custom function to render the histogram and ignores built-in categorical histogram options such as displayed_values_count, displayed_values_percent, and label_rotation.

The examples below show two common categorical histogram setups. The first uses the built-in categorical histogram rendering with displayed_values_count and label_rotation. The second uses plot_function, which overrides the default categorical histogram rendering.

Example 1: built-in categorical histogram options (top 5 values + label rotation)

histogram:
  type: categorical
  displayed_values_count: 5
  label_rotation: 90

Example 2: custom categorical histogram rendering using plot_function

histogram:
  type: categorical
  plot_function: "customplot1.py:my_own_plot"

For GAIn to render the second histogram using a custom plotting function, place a Python module such as customplot1.py that contains the function my_own_plot in the resource directory. The custom function must render and write a plot to the provided output stream (outfile) so it can be embedded in the HTML summary output. A simple example that sorts categories by their counts, keeps the top 20, and renders a basic bar chart (with optional log-scaled y-axis) to the provided output stream is:

from typing import IO
from gain.genomic_resources.histogram import CategoricalHistogram
import matplotlib.pyplot as plt

def my_own_plot(outfile: IO, histogram: CategoricalHistogram, xlabel: str, *_args, **_kw) -> None:
    items = sorted(histogram.raw_values.items(), key=lambda x: -x[1])[:20]
    labels, counts = zip(*items) if items else ([], [])
    plt.figure()
    plt.bar(labels, counts, log=histogram.config.y_log_scale)
    plt.xlabel(xlabel); plt.ylabel("count")
    plt.savefig(outfile); plt.clf()

Null histograms

Null histograms are used when calculating a histogram is not possible or does not make sense for a score. In this case, the HTML summary page will not display a histogram for the score, and instead records the reason why histogram display is disabled.

A null histogram configuration supports one required field.

reason: a short explanation of why the histogram is disabled.

Example:

histogram:
  type: null
  reason: "Histogram is not available for this score."

VCF score auto-detection

VCF files already describe many score-like fields in their headers. In particular, each ##INFO line provides an ID, a type, and a human-readable description. GAIn uses this metadata to automatically create score definitions for INFO fields, which you can then reference in configuration just like manually defined scores.

Create the following file and save it as example.vcf, which contains a single INFO field A:

##fileformat=VCFv4.1
##INFO=<ID=A,Number=1,Type=Integer,Description="Score A">
#CHROM POS ID REF ALT QUAL FILTER  INFO
chr1   5   .  A   T   .    .       A=1

Create the following genomic_resource.yaml for this score which omits an explicit scores section.

type: position_score

table:
  filename: example.vcf
  format: vcf_info

When you run grr_manage resource-repair, the scores and their descriptions will be automatically generated from the INFO field in the vcf file.

The configuration above is equivalent to spelling out the generated score definition explicitly:

type: position_score

table:
  filename: example.vcf
  format: vcf_info

scores:
- id: A
  type: int
  column_name: A
  desc: Score A

Some fields cannot be automatically generated. To customize a generated definition, add a scores: entry with the same id and include only the fields you want to change or extend (for example, overriding type or adding a histogram block):

scores:
- id: A
  type: float
  histogram:
    type: categorical
    value_order: ["alpha", "beta"]

GAIn derives each score’s type from the VCF INFO field type: Integer maps to int, Float to float, String to str, and Flag to bool.

This mapping applies to a field whose header declares a single value per score – a Number of 0, 1, A or R. A field declared multi-valued (an unbounded Number=., or any fixed arity above one) has no single value to type: GAIn reads every element of it as one string, joined on |, so such a score is typed str whatever its Type= says. ##INFO=<ID=A,Number=.,Type=Integer> with the row A=1,2 reads "1|2".

A scores: entry cannot override that, and must not contradict it. Stating any type: other than str on a multi-valued field is a configuration error: the stated type describes a value the join cannot produce, and the declared type is not merely descriptive – it selects the histogram the statistics build computes. Such a resource fails to build, and repo-repair names it and the score:

Invalid configuration: <resource id>: score 'A' states 'type: int', but
its ##INFO line declares Number=.,Type=Integer: a field the header
declares multi-valued reads '|'-joined text, so its value type is 'str'.
State 'type: str' or leave 'type:' unstated.

State type: str, or leave type: unstated and let the field take str from the header. Overriding type: works as described below for the single-valued fields.

Configuring a number histogram over a score whose type is not int, float or bool is a configuration error for the same reason – no number histogram can accumulate text. This reaches a joined VCF field and equally a plain table score declaring type: str. Give such a score histogram: {type: categorical}, or no histogram: at all and let GAIn pick the default for its type.

A scores: entry whose id matches no ##INFO line is a configuration error too: a VCF score is its INFO key, and the entry cannot address a column by any other name. GAIn refuses the resource where the definitions are built – ahead of the other checks on this list, since none of them can be asked of a field the header does not have – and lists what the header does declare, so a typo reads off the line:

Invalid configuration: <resource id>: score 'NOPE' names no ##INFO
field of the VCF header, which declares: A, B, C. A VCF score's 'id'
is its INFO key; fix the 'scores:' entry or the header.

For the same reason a scores: entry needs no column_name: at all, and one that states an address other than its id is a configuration error: a column_name: (or legacy name:) that differs from the id, whether or not the header declares that other name, or any column_index: (index:), since an INFO field is not a column. The reader would ignore such a line while the author reads it as the score’s source. An address equal to the id – column_name: A on id: A, as in the spelled-out example above – is redundant and passes. column_name: B on id: A does not:

Invalid configuration: <resource id>: score 'A' states column_name
'B', but a VCF score reads the INFO field named by its 'id', so this
score reads 'A'. Drop the 'column_name:' (or legacy 'name:') line, or
make it 'A'.

A field declared with the genotype arity Number=G cannot be a score at all: pysam does not read a per-genotype INFO field, so any row carrying one would fail the read. GAIn refuses the resource when the field would become a score – with no scores: block every header field does, and so does an entry naming it, or merge_vcf_scores: true – and names it:

Invalid configuration: <resource id>: score 'A' is declared
Number=G,Type=Integer in its ##INFO line; pysam does not read a
per-genotype INFO field, so this score can never be read. List the
fields you want in a 'scores:' block that leaves it out
('merge_vcf_scores' unset or false), or change the header.

A scores: block that leaves the field out reads the rest of the file as usual, whatever the rows carry for it.

The two header rules – a stated type: the ##INFO line contradicts, and a Number=G field – are about a VCF and so reach only the genomic score types described in this section: position_score, np_score, allele_score and fragment_score. The number-histogram rule is about a score’s declared type: alone, so it reaches a gene_score as well: a gene score pairing type: str with histogram: {type: number} fails to build with the same error, naming the resource and the score, whether or not a view_range is given.

Tabix indexing

Many GAIn resource types are backed by on-disk tables, typically tab-delimited genomic files (for example TSV/BED-like tables, bedGraph, or VCF-derived tables). These files can be large, but GAIn still needs to look up the records that overlap a given genomic interval during annotation (for example, chr1:100000-101000). Scanning the full file for every query would be too slow, so GAIn supports Tabix-indexed tables for fast random access by genomic region. (Some resource formats such as bigWig are already indexed and do not use Tabix.)

When you set format: tabix under a resource’s table section, you are telling GAIn that the data file is bgzip-compressed, coordinate-sorted, and accompanied by a Tabix index (.tbi or .csi). With that index in place, GAIn can jump directly to the relevant file blocks, and your table: mapping tells it how to interpret each row (which columns provide chrom, pos_begin, and pos_end, plus any header handling you specify).

The main pitfall is coordinate conventions: BED-style files are typically 0-based, half-open, while many TSV tables and VCF positions are 1-based. Keep the tabix indexing flags (for example -0) consistent with the file, and set zero_based accordingly in the resource YAML to avoid subtle off-by-one overlaps.

Common options:

Option	Description
-p, --preset	Preset parser for common formats, for example vcf, bed, and gff. This sets the expected coordinate columns automatically.
-s, --sequence	1-based column index for the chromosome/contig sequence name column.
-b, --begin	1-based column index for the start/begin coordinate column.
-e, --end	1-based column index for the end/stop coordinate column. If the file has no end column, set -e to the same value as -b for single-position intervals.
-0, --zero-based	Interpret coordinates as 0-based, BED-style, instead of 1-based.
-C, --csi	Generate a CSI index instead of the default TBI index. This is useful for very large coordinates or contigs.
-f, --force	Overwrite an existing index file.

For a full list of options run tabix --help. The examples below show how to produce Tabix indexes for common file layouts.

example usage of tabix

For a VCF-format score (-p vcf: use the VCF preset):

$ tabix -p vcf score.vcf.gz

For a 1-based TSV score with a single position column (-s: chrom column, -b: pos column, -e: same as -b):

$ tabix -s 1 -b 2 -e 2 score.tsv.gz

For a 1-based TSV score with start and stop position columns (-s: chrom, -b: start, -e: end):

$ tabix -s 1 -b 2 -e 3 score.tsv.gz

For a 0-based TSV score with start and stop position columns (-0: 0-based coordinates, plus -s/-b/-e as above):

$ tabix -0 -s 1 -b 2 -e 3 score.tsv.gz