Pipelines

An AnnotationPipeline is an ordered list of annotators over one repository. It is what annotate_tabular and annotate_vcf build from the YAML they are given, and what a Python caller builds from the same YAML with load_pipeline_from_yaml().

Loading a pipeline

Four loaders build a pipeline and differ only in where the configuration comes from:

Each takes the repository the annotators resolve their resources through, and two keyword arguments:

  • work_dir — a pathlib.Path under which every annotator gets a subdirectory of its own, A<index>_<type>. The loader injects that subdirectory into the annotator’s parameters as work_dir, unless the YAML entry sets one. Pass a Path: the loader joins the subdirectory name with / for each annotator, so a str raises TypeError as soon as the configuration has an annotator. Left unset, a temporary directory is minted with a warning; pass one.

  • allow_repeated_attributes — what to do when two annotators emit the same attribute name. Off, the pipeline refuses to build and names the overlap. On, every attribute with that name, the first one included, is renamed to <name>_<annotator_id>.

import pathlib

import gain.grr as grr
from gain.annotation.annotation_factory import build_annotation_pipeline

repo = grr.build_genomic_resource_repository()
pipeline = build_annotation_pipeline(
    [{"position_score": {"resource_id": "hg38/scores/phyloP7way"}}],
    repo,
    work_dir=pathlib.Path("annotation_work"),
)

Loading does everything but open. For each entry of the configuration, in order, the loader resolves the entry’s type to an annotator factory (Annotator plugins), calls it with the pipeline and the entry’s AnnotatorInfo, wraps the result for input_annotatable and value_transform if the entry uses them, and refuses any parameter the annotator never read. A fault in the configuration is reported as an AnnotationConfigurationError in four ways:

  • The configuration parser raises it for most faults in the shape of the configuration.

  • The loader catches a ValueError or a FileNotFoundError that any of the steps above raises, and raises an AnnotationConfigurationError that names the annotator and carries the original error.

  • An annotator raises it directly for a parameter value it refuses: one read with get_number or get_integer (Writing an annotator), or, in the allele and fragment score annotators, a filter expression that does not parse or a value they do not accept.

  • After the last entry, the loader raises it when two annotators emit the same attribute name and allow_repeated_attributes is off.

The loader wraps no other exception: a TypeError or a KeyError from an annotator factory passes through unchanged.

The lifecycle

A loader returns the pipeline closed. open() opens every annotator, and close() closes every annotator, logging rather than propagating what any of them raises. The pipeline is a context manager whose exit closes it, and, as with the resource objects, entering it does not open it. The spelling that does both is with pipeline.open() as pipeline:.

Running

annotate() takes one annotatable (or None) and an optional context, runs every annotator in order, merging each one’s answer into the context before the next runs, and returns the context. So the result carries every attribute of every annotator, internal ones included — it is the writers, not the pipeline, that drop internal attributes from the output.

batch_annotate() does the same over a sequence, one context per input, calling each annotator’s batch_annotate once with the whole sequence. An annotator with a batched backend runs its tool once per call; every other annotator loops. The optional batch_work_dir is an opaque scratch path for the batched annotators. The pipeline passes it to each annotator unchanged and gives it no meaning of its own. GAIn’s core annotators ignore it, and the demo_annotator and VEP adapters treat it as a path under their own work_dir.

Both open the pipeline on first use if nothing has opened it; both leave it open.

Reading a pipeline

Once built, a pipeline answers questions about itself: get_attributes() lists every attribute in pipeline order, get_attribute_info() finds one by name (the first match, so a later annotator that reuses a name is shadowed), and get_annotator_by_attribute_info() finds the annotator that produces it. get_resource_ids() is the set of every resource any annotator uses — what a caching repository would need to fetch to run the pipeline offline. The AnnotatorInfo of each annotator is available through get_info(), and to_dict on it round-trips to the YAML shape it was parsed from.

Reannotation

A ReannotationPipeline is built from a new pipeline and the previous one, and runs only what changed:

  • the annotators new to the pipeline;

  • the annotators whose used_context_attributes name an attribute a new annotator now produces;

  • the annotators whose internal attributes a new annotator reads, since an internal attribute is not in the previous output and has to be computed again.

Everything else is copied from the previous output. That is the reason the tuple matters to an implementer: an annotator that reads the context without declaring what it reads is not rerun when its inputs change.

The keyword argument full_reannotation=True turns the selection off: the reannotation pipeline runs every annotator of the new pipeline, copies nothing, and marks every attribute of the previous pipeline as deleted. The --full-reannotation option of annotate_tabular and annotate_vcf sets it.

The attribute plan holds the decision as a ReannotationPlan: the attributes copied, added, computed again and deleted, each with the annotator that produces it. format_plan() renders the plan as text, and print_plan() prints it, to standard error by default:

from gain.annotation.annotation_pipeline import ReannotationPipeline

reannotation = ReannotationPipeline(pipeline_new, pipeline_previous)
reannotation.print_plan(reference="previous.yaml")

API

load_pipeline_from_yaml(raw: str, grr: GenomicResourceRepo, *, allow_repeated_attributes: bool = False, work_dir: Path | None = None) → AnnotationPipeline[source]

Load an annotation pipeline from a YAML-formatted string.

load_pipeline_from_file(raw_path: str, grr: GenomicResourceRepo, *, allow_repeated_attributes: bool = False, work_dir: Path | None = None) → AnnotationPipeline[source]

Load an annotation pipeline from a configuration file.

load_pipeline_from_file_or_resource(arg: str, grr: GenomicResourceRepo, *, allow_repeated_attributes: bool = False, work_dir: Path | None = None) → AnnotationPipeline[source]

Load a pipeline from a file path or a GRR resource id.

Tries to interpret arg as a filesystem path first; on miss, falls back to looking it up as a GRR resource of type annotation_pipeline.

build_annotation_pipeline(config: list[dict[str, Any]] | RawFullConfig, grr: GenomicResourceRepo, *, allow_repeated_attributes: bool = False, work_dir: Path | None = None) → AnnotationPipeline[source]

Build an annotation pipeline.

class AnnotationPipeline(repository: GenomicResourceRepo)[source]

Provides annotation pipeline abstraction.

add_annotator(annotator: Annotator) → None[source]

Append an annotator; it runs after every annotator already added.

Adding to an open pipeline does not open the annotator: the pipeline opens its annotators only in open().

annotate(annotatable: Annotatable | None, context: dict | None = None) → dict[source]

Apply all annotators to an annotatable.

batch_annotate(annotatables: Sequence[Annotatable | None], contexts: list[dict] | None = None, batch_work_dir: str | None = None) → list[dict][source]

Apply all annotators to a list of annotatables.

close() → None[source]

Close the annotation pipeline.

get_annotator_by_attribute_info(attribute_info: Attribute) → Annotator | None[source]

The annotator producing attribute_info, or None.

Matched by attribute equality, so pass an attribute obtained from this pipeline – get_attribute_info()’s answer, say.

get_attribute_info(attribute_name: str) → Attribute | None[source]

The attribute named attribute_name, or None.

The first match in pipeline order, so a later annotator that reuses a name is shadowed here.

get_attributes() → list[Attribute][source]

Every attribute every annotator produces, in pipeline order.

get_attributes_by_type(attribute_type: str) → list[Attribute][source]

The attributes of one attribute_type, in pipeline order.

Attributes without a spec are skipped.

get_info() → list[AnnotatorInfo][source]

The annotator info of every annotator, in pipeline order.

One AnnotatorInfo per annotator, as each was built from.

get_resource_ids() → set[str][source]

The ids of every resource any annotator uses, as one set.

open() → AnnotationPipeline[source]

Open all annotators in the pipeline and mark it as open.

print() → None[source]

Print the annotation pipeline.

resolve_attribute_parameter(info: AnnotatorInfo, parameter: str, *, expected_attribute_type: str) → str[source]

Resolve info’s parameter to the name of one of my attributes.

The parameter names an upstream attribute the annotator reads. Refused, as a ValueError, when info has no such parameter, when no annotator in the pipeline produces an attribute of that name, or when the attribute’s AttributeSpec.attribute_type is not expected_attribute_type.

One implementation rather than one per caller because the copies it replaces had drifted apart – a typo fixed twice (gain#1170), a misspelling fixed in one copy and left in the other (gain#1280), and the listing of available attributes in the refusal present in one copy and absent from the others (gain#1490).

class ReannotationPipeline(pipeline_new: AnnotationPipeline, pipeline_previous: AnnotationPipeline, *, full_reannotation: bool = False)[source]

Provides functionality for reannotation.

format_plan(reference: str | None = None) → str[source]

Render the reannotation plan as human-readable text.

print_plan(reference: str | None = None, file: IO[str] | None = None) → None[source]

Print the reannotation plan.

class AnnotatorInfo(_type: str, attributes: list[AttributeConfig], parameters: ParamsUsageMonitor | dict[str, Any], documentation: str = '', resources: list[GenomicResource] | None = None, annotator_id: str = 'N/A')[source]

Defines annotator configuration.

to_dict() → dict[str, Any][source]

Convert annotator info to a configuration dictionary.

class AnnotationConfigurationError(message: str | None, other_error: Exception | None = None, error_mark: ErrorMark | None = None)[source]

Exception raised for errors in the annotation configuration.