gain.binning package

Submodules

gain.binning.binners module

Binner kinds: how a run-definition entry becomes tracks and values.

Kinds are discovered through the gain.binning.binners entry-point group, so a kind – the position-score binner here, the fragment-score binner in gain.binning.fragment_binner, an external plugin – registers the way every other gain plugin does, without editing the tool.

class Binner(*args, **kwargs)

Bases: Protocol

What a registered binner kind provides.

static bind(job: BinningJob, grr: GenomicResourceRepo) → AbstractContextManager[BoundBinner]

Bind job to its resources for as long as the with lasts.

A task binds once and asks for every region of its bundle through the bound value, so a resource is opened once per task however many regions the bundle holds, and is released when the with ends, whether the task succeeded or not.

kind: ClassVar[str]

classmethod parse_entry(label: str, config: dict[str, Any], grr: GenomicResourceRepo, *, base_dir: str | None = None) → list[BinningJob]

Resolve one run-definition entry into jobs.

The entry’s tracks are the jobs’ tracks, in order. label names the entry in error messages (binners[2]). base_dir is the run definition’s directory, passed only when the run definition was read from a file; a kind that reads a local file resolves a relative path against it. Raises RunDefinitionError for an entry that cannot be resolved, an entry matching nothing included.

class BinnerDef(*args, **kwargs)

Bases: Protocol

The typed definition of a track: what the other columns do not say.

to_json() → str

The definition as canonical JSON.

class BinningJob(binner: str, tracks: tuple[Track, ...], entry: str = '')

Bases: object

What one task binds to: tracks read together from one binding.

binner names the kind that binds it, which is how the task graph finds the binner. tracks are the job’s columns, in order; the block a binding returns for a region has one column per track. entry labels the run-definition entry the job was resolved from (binners[2]), set when the run definition is parsed.

binner: str

entry: str = ''

tracks: tuple[Track, ...]

class BoundBinner(*args, **kwargs)

Bases: Protocol

A job bound to its resources: bins them one region at a time.

bin_region(region: BedRegion, bin_size: int) → ndarray[tuple[Any, ...], dtype[float64]]

Reduce the job’s tracks over region on the global grid.

Returns a float64 block of shape (n_bins, n_tracks): one row per grid bin of region, one column per track of the job, in the job’s order.

class PositionScoreBinding(track: Track, score: PositionScore, *, value_arrays_batch_size: int = 100000)

Bases: object

One track’s score, open for as long as the with lasts.

Entering opens the score once, whatever number of regions are then binned through it; leaving closes it, on success or failure alike. value_arrays_batch_size is the rows hint of each batch that the array fold of bin_region() reads.

bin_region(region: BedRegion, bin_size: int) → ndarray[tuple[Any, ...], dtype[float64]]

Reduce the track over region to one float64 per grid bin.

The bins are those of PositionScore.get_score_in_bins(), the semantic reference for the global grid, the boundary split and first-record-wins. A float score under count, sum, mean, max, min or median, on a backend that serves fetch_region_value_arrays(), is folded from those arrays by ArrayBinFold, with the same bins bit for bit. Every other track is folded from get_score_in_bins itself.

A bin no record covers comes back None and is stored as NaN, unless the track’s replacement made it count. The count aggregator is the exception: a count of nothing is 0, so an uncovered bin of a count track is stored as 0.0. An exact int past the float range (an int sum or product) is stored as a signed infinity: see to_float64().

A chromosome the score never mentions is one uncovered run (gain#1211), so the chromosome’s bins are uncovered like any other’s.

Returned as a block of one column, the job’s one track.

class PositionScoreBinner

Bases: object

Bins position_score resources matched by a resource_query.

ENTRY_KEYS: ClassVar[frozenset[str]] = frozenset({'aggregator', 'name', 'none_value_replacement', 'resource_query', 'search_term'})

static bind(job: BinningJob, grr: GenomicResourceRepo) → PositionScoreBinding

Bind the job’s one track to its score.

The score is opened when the binding is entered and closed when it is left.

kind: ClassVar[str] = 'position_score_binner'

static load_def(text: str) → PositionScoreDef

Read a binner_def of this kind.

classmethod parse_entry(label: str, config: dict[str, Any], grr: GenomicResourceRepo, *, base_dir: str | None = None) → list[BinningJob]

Resolve one entry’s resource_query into one job per track.

The matches are match_resources()’s, restricted to position scores: one track each, in resource-id order.

class PositionScoreDef(aggregator: str, none_value_replacement: float | None = None, name: str | None = None)

Bases: object

A position-score track’s definition.

aggregator and none_value_replacement are the resolved ones (None for no replacement); name is the entry’s optional name. The score id is the track’s selector, not part of the definition.

aggregator: str

classmethod from_json(text: str) → Self

The definition to_json wrote.

name: str | None = None

none_value_replacement: float | None = None

to_json() → str

The definition as canonical JSON.

exception RunDefinitionError

Bases: ValueError

A run definition that cannot be resolved into a run.

class Track(name: str, resource_ids: tuple[str, ...], group: str, selector: str, binner_type: str, definition: BinnerDef, binner_index: int = 0, track_index: int = 0, parameters: str = '')

Bases: object

One column of the output: a score of resources, reduced one way.

resource_ids are the resources the column is computed from, in resource-id order: one for a position-score track or an unpooled fragment track, every matched resource for a pooled one. group is the column’s group of a grouped kind – a fragment track’s cell class, or all – and empty for a position-score track. selector is what picks the track within its resources: the score id of a position-score track, the group of a fragment track.

binner_type names the kind that produces the column (binner_type_of()). binner_index is the entry’s position in the run definition’s binners list and track_index the track’s position among all tracks of that entry, in output order. definition holds what is specific to the kind; it is written to the file as binner_def.

name is not written to the file; it keeps the uniqueness check, the chunk file names and the plan output. parameters is whatever else a kind needs to say decides the column’s values – a constant contribution, the metadata table and filter that map cells to groups – as canonical text, empty when the other fields say it all. It is not written to the file either; it keys the column’s chunks, so two run definitions sharing a work directory never share a chunk they compute differently.

binner_index: int = 0

binner_type: str

definition: BinnerDef

group: str

name: str

parameters: str = ''

resource_ids: tuple[str, ...]

selector: str

track_index: int = 0

aggregator_of(track: Track) → str

The aggregator a track’s definition names, empty if it has none.

binner_type_of(kind: str) → str

The binner_type the file records for a registered kind.

The kind without its _binner suffix: position_score_binner is position_score. The one place the two names are mapped.

canonical_json(fields: dict[str, Any]) → str

Sorted keys, no extra whitespace; NaN and infinities are refused.

Raises ValueError for a NaN or an infinite number, which JSON cannot carry.

check_keys(label: str, config: Any, known: frozenset[str]) → None

Refuse a mapping with keys outside known.

A mistyped key is refused rather than dropped, so what the user wrote never silently changes what the run does.

discover_binner_kinds() → dict[str, type[Binner]]

Map every registered binner kind to its class, by the class’s kind.

entry_name(label: str, config: dict[str, Any]) → str | None

An entry’s name, the base of its tracks’ names (F30).

Absent, the entry has none and its kind names its tracks. Given, it is a non-empty string; anything else – null included – is refused rather than read as absent.

load_track_def(binner_type: str, text: str, kinds: dict[str, type[Binner]] | None = None) → BinnerDef | dict[str, Any]

Read a binner_def of the file back into its typed definition.

A binner_type with no installed kind, or a kind without load_def, gives the plain dict of json.loads, so a file stays readable without its plugin.

match_resources(label: str, config: dict[str, Any], grr: GenomicResourceRepo, resource_type: str) → list[GenomicResource]

Resolve an entry’s resource_query into its matches, by id.

The query is always a repository search – an exact id is the search that matches one resource – restricted to resource_type by the search’s own type filter, and ordered by resource id, so the track order is deterministic whatever the repository yields. That filter needs no index of its own (gain#1212). A search_term is the full-text index’s filter, conjoined with the query (D7), and the one key that needs the index. An entry matching nothing is an error, never an empty contribution.

named_base(name: str | None, resource_id: str) → str

The base of a per-resource track’s name: name/<resource id>.

An entry without name names the track by the resource id alone.

numeric_aggregators() → list[str]

The registered aggregators whose result is a number (D11).

Those that declare a numeric output type, and the numeric-only ones that answer in their input’s type (min, max, sum, product) – a binned score is numeric, so they answer a number too. Read off the registry, so a numeric aggregator added to it is accepted here without a list to keep in step.

gain.binning.cli module

binning_tool: bin genomic scores into a fixed genome grid.

One task per (job, bundle of consecutive regions) writes a column chunk per track and region as a .npy vector in the work directory; one serial writer task assembles the HDF5 file region by region. HDF5 has a single writer, so no task other than the writer touches the file, and a rerun with the same work directory reuses the finished chunks and reruns only the writer.

cli(argv: list[str] | None = None) → None

Entry point of binning_tool.

gain.binning.fragment_array_folds module

Array bin folds: the fragment_start, fragment_length and coverage_profile folds.

fold_fragment_starts() bins the fragments of one region from the batches of FragmentScore.get_fragment_arrays_starting_in_region, one read per resource, into a (bins, tracks) block. The block equals, bit for bit, the one that BinFragmentStartAggregator gives for the heapq.merge by start of the record reads, with two exceptions:

• The record read keeps an int score and an int constant value exact, so sum, mean, product and the median of an even count past 2**53 can differ.

• The array fold takes every NaN value as no value. The record read takes only a value in the na_values of the score as no value, so a NaN that na_values does not cover counts as a value there.

The reads are pooled by a frontier: the smallest last start over the pending batches of the reads that are not exhausted. Each window takes the fragments that start before the frontier from each read, joins them in read order and sorts them by start with a stable sort, which is the order of heapq.merge. The fold then adds a window to dense state arrays with ufunc.at, which applies the values in input order.

fold_fragment_lengths() bins the fragments that overlap a region, from the batches of FragmentScore.get_fragment_arrays_overlapping_region. It clips each fragment to the region and splits it into one piece for each bin that it reaches, weighted by the overlap in base pairs. The block equals the one that BinFragmentLengthAggregator gives, with the same two exceptions. The pieces of a window, the pieces of the uncovered_value fill included, are in feed order, so each bin gets its values in the order of the streaming fold.

fold_coverage_profiles() bins the coverage profile of the same fragments, for int values. It turns each fragment into a +v event at its start and a -v event after its end, and sums the events of each track with an int64 cumsum. The levels make runs as CoverageProfile makes them, and the length fold folds the runs. The block equals the one that BinFragmentCoverageAggregator gives, with the exceptions of the length fold.

ARRAY_AGGREGATORS = ('count', 'sum', 'mean', 'max', 'min', 'median', 'product')

The aggregators that the array fold answers.

class CoverageRead(batches: Generator[tuple[ndarray, ndarray, dict[str, ndarray]], None, None], value_score: str | None = None, constant: float = 1.0, group_score: str | None = None, track_of: Callable[[Any], int | None] | None = None, start: int = 1, end: int = 1)

Bases: LengthRead

The batches of one resource that overlap [start, end], as LengthRead.

A fragment with a NaN value is no coverage: the fold drops it, so it covers no base.

EXACT_LEVEL_BOUND = 9007199254740992

The running sum of the absolute values of a region must stay below this bound. Below it, each value and each level is an exact integer in float64 and in int64.

exception InexactLevelsError

Bases: ValueError

The values of a region can make a level that is not exact.

fold_coverage_profiles() raises it when the running sum of the absolute values of the region reaches EXACT_LEVEL_BOUND.

LENGTH_AGGREGATORS = ('max', 'mean', 'median', 'min', 'sum')

count would count base pairs, and product raise a value to the power of a length. The two length-weighted array folds, fragment_length and coverage_profile, answer each of them.

Type:

The aggregators a length-weighted mode accepts

class LengthRead(batches: Generator[tuple[ndarray, ndarray, dict[str, ndarray]], None, None], value_score: str | None = None, constant: float = 1.0, group_score: str | None = None, track_of: Callable[[Any], int | None] | None = None, start: int = 1, end: int = 1)

Bases: StartRead

The batches of one resource that overlap [start, end].

Each fragment is clipped to [start, end]. A fragment with a NaN value stays: it adds no value, but it covers its bases, so no uncovered_value goes under it.

end: int = 1

start: int = 1

class StartRead(batches: Generator[tuple[ndarray, ndarray, dict[str, ndarray]], None, None], value_score: str | None = None, constant: float = 1.0, group_score: str | None = None, track_of: Callable[[Any], int | None] | None = None)

Bases: object

The batches of one resource and how a fragment finds its track.

track_of maps a value of the group_score column to a track index, or to None for a fragment that reaches no track. Without track_of every fragment goes to track 0. value_score names the column of the values, and None gives each fragment the constant value. dropped counts the fragments that reach no track.

batches: Generator[tuple[ndarray, ndarray, dict[str, ndarray]], None, None]

begins: ndarray[tuple[Any, ...], dtype[int64]]

their starts, ends, tracks and values.

Type:

The pending fragments

constant: float = 1.0

dropped: int = 0

ends: ndarray[tuple[Any, ...], dtype[int64]]

exhausted: bool = False

group_score: str | None = None

refill() → None

Pull batches while the pending fragments all share one start.

take_before(frontier: int | None) → _Window

Take the pending fragments that start before frontier.

None takes all of them.

track_of: Callable[[Any], int | None] | None = None

tracks: ndarray[tuple[Any, ...], dtype[int64]]

value_score: str | None = None

values: ndarray[tuple[Any, ...], dtype[float64]]

fold_coverage_profiles(reads: Sequence[CoverageRead], *, start: int, end: int, bin_size: int, tracks: int, aggregator: str, uncovered_value: float | None = None) → ndarray[tuple[Any, ...], dtype[float64]]

Fold the coverage profile of reads into a (bins, tracks) block.

The values of the fragments must be integers. The level of a base in a track is the sum of the values of the fragments of the track that cover it. The runs of equal levels other than 0 are folded as fold_fragment_lengths() folds fragments. A base at level 0 is uncovered, and gets the uncovered_value.

Raises InexactLevelsError when the running sum of the absolute values reaches EXACT_LEVEL_BOUND, and ValueError when a value is not an integer.

fold_fragment_lengths(reads: Sequence[LengthRead], *, start: int, end: int, bin_size: int, tracks: int, aggregator: str, uncovered_value: float | None = None) → ndarray[tuple[Any, ...], dtype[float64]]

Fold the fragments of reads into a (bins, tracks) block.

The rows are the grid bins from the one that holds start to the one that holds end. A fragment adds its value to each bin that it reaches, weighted by the base pairs that it shares with the bin. With an uncovered_value, each base of [start, end] that no fragment of a track covers adds that value to the track with a weight of 1 bp. A bin that gets no value holds the value of EMPTY_BIN_VALUES for aggregator. The reads give the fragments that overlap [start, end], each read in start order.

fold_fragment_starts(reads: Sequence[StartRead], *, start: int, end: int, bin_size: int, tracks: int, aggregator: str) → ndarray[tuple[Any, ...], dtype[float64]]

Fold the fragments of reads into a (bins, tracks) block.

The rows are the grid bins from the one that holds start to the one that holds end. A bin that no fragment with a value reaches holds the value of EMPTY_BIN_VALUES for aggregator. The reads give the fragments that start in [start, end], each read in start order.

gain.binning.fragment_binner module

fragment_score_binner: fragments per bin, by group, pooled or not.

An entry names a query over fragment_score resources and produces, per grid bin, an aggregate over the fragments that START in that bin – the partition FragmentScore.get_fragment_scores_starting_in_region() reads, so adjacent regions see each fragment exactly once – or, in FRAGMENT_LENGTH mode, over the fragments that OVERLAP it, each weighted by its overlap in base pairs. That mode reads FragmentScore.get_fragment_scores_overlapping_region() and clips each fragment to the region, so a base pair belongs to one region and no fragment’s base pair is counted twice. In COVERAGE_PROFILE mode the clipped fragments of each track make its coverage profile – per base, the sum of the values of the fragments over it – and each bin reduces the profile, weighted by base pairs; a base the profile leaves at 0 is uncovered.

An entry’s keys:

• resource_query (required) and search_term: the matched fragment_score resources, ordered by resource id.

• pool (default true): every match is merged by position into one set of group tracks, named <resource_query>:<group>; with false each resource is its own job, its tracks named <resource id>:<group>.

• group: {group: NAME}, one track with a constant name, or {cell_score_id, cell_meta_column, group_meta_column}, one track per group of a metadata table, which meta names; each of the three keys is optional and defaults to the convention’s (below), so group: {} groups by the convention throughout; or {group_score_id: S}, one track per distinct value of the str score S, read when the entry is resolved from the score’s full categorical histogram (its statistics must be built and pulled). Pooled, a value’s group is <sample_id>:<value>, by the resource’s SAMPLE_ID_LABEL label, which every pooled resource must carry, each its own, since one barcode recurs in every sample; unpooled, the bare value. The forms do not mix, and meta does not go with the last. Omitted, the resource’s labels decide: a resource carrying CELL_META_RESOURCE_ID_LABEL is grouped as with group: {}, any other is the one group all.

• value: what one fragment adds, {score_id: S}, the value of the numeric score S, or {value: V}, a constant; omitted, {value: 1}.

• aggregate: how a bin reduces the values, {mode, aggregator, uncovered_value}; mode is FRAGMENT_START (the default), FRAGMENT_LENGTH or COVERAGE_PROFILE, and aggregator omitted means sum. The last two take only LENGTH_AGGREGATORS, and an uncovered_value: a number each uncovered base of the region adds, with a weight of 1 bp, or null, nothing – omitted, null in FRAGMENT_LENGTH mode and 0 in COVERAGE_PROFILE mode; FRAGMENT_START refuses it. The keys score and value are refused here, naming the value key. Without value and aggregate an entry counts the fragments that start in each bin, whatever its resources’ scores.

• meta: exactly one source of the table mapping barcodes to groups – resource_id: ID, a data_frame resource; resource_label: LABEL, the data_frame resource the fragment resource’s label LABEL names; or file_name (with optional file_format, one of csv, tsv or excel, and file_separator), a local table, a relative name read from the run definition’s directory – and filter: [{column, value | label}], the conjuncts selecting a sample’s rows: a table column equal to a literal (value) or to the fragment resource’s label of that name (label). No filter selects every row. Omitted under a metadata grouping, meta is the convention’s: the table the CELL_META_RESOURCE_ID_LABEL label names, its SAMPLE_ID_COLUMN rows equal to the SAMPLE_ID_LABEL label.

A pooled entry resolves to one job, so its resources must agree: on whether they carry CELL_META_RESOURCE_ID_LABEL (when group is omitted), and on the one metadata table their meta names.

A fragment whose barcode is not in the filtered rows, or whose row’s group is empty, or whose group_score_id value is missing or is one its resource’s histogram does not list, reaches no track; it is counted as dropped, per resource and region (in FRAGMENT_LENGTH and COVERAGE_PROFILE modes, in every region it reaches). An empty bin holds the fold’s empty value – 0 for count and sum, NaN for the others – and so does every bin of a contig a resource lacks; with an uncovered_value no bin is empty, and a contig a resource lacks is uncovered throughout.

class FragmentBinningJob(binner: str, tracks: tuple[Track, ...], entry: str = '', resource_ids: tuple[str, ...] = (), score_id: str | None = None, value: float = 1, grouping: CellGrouping | ValueGrouping | None = None, warnings: tuple[str, ...] = ())

Bases: BinningJob

A fragment job: its tracks, and how each fragment reaches one.

resource_ids are the resources read together, in resource-id order – every match of a pooled entry, or one resource of an unpooled one. A fragment contributes its score_id score’s value when the entry’s value is a score, else the constant value. With a grouping a fragment reaches the track of its barcode’s group, or none; without one, the job’s one track. warnings are what a dry run tells the user about how the job was resolved.

grouping: CellGrouping | ValueGrouping | None = None

property local_files: tuple[str, ...]

The local metadata files the job reads, as absolute paths.

resource_ids: tuple[str, ...] = ()

score_id: str | None = None

value: float = 1

warnings: tuple[str, ...] = ()

class FragmentScoreBinding(job: FragmentBinningJob, scores: list[FragmentScore], track_maps: list[dict[str, int]] | None)

Bases: object

A fragment job’s resources, open for as long as the with lasts.

Each resource is its own FragmentScore, so the one-live-read-per-score limit holds however many are pooled. Each resource’s barcode-to-track map is built once, when bound.

With a grouping, dropped maps each (resource id, region) binned so far – the region as chrom:start-stop – to the number of its fragments that reached no track: a barcode absent from the resource’s filtered rows, a row of no group, or a grouping-score value that is missing or that the score’s histogram does not list.

bin_region(region: BedRegion, bin_size: int) → ndarray[tuple[Any, ...], dtype[float64]]

Fold the fragments of region into the job’s tracks.

A FRAGMENT_START job on scores with column-array reads folds the array batches of each resource with fold_fragment_starts(), a FRAGMENT_LENGTH job with fold_fragment_lengths(), and a COVERAGE_PROFILE job of int values with fold_coverage_profiles(). The declared type of the score, or the type of the constant value, decides what is int. A region whose levels can be inexact in int64 takes the streaming fold. Otherwise one streaming fold per track, of the track’s mode, is fed the merge by start of one read per resource – starting-in, or overlapping with each fragment clipped to the region – and each bin a fold answers goes to its row of the block. A resource without the region’s contig contributes nothing. Every read is drained, or closed on failure, before this returns. With a grouping, each resource’s dropped fragments are counted into dropped and logged.

dropped: dict[tuple[str, str], int]

class FragmentScoreBinner

Bases: object

Bins fragment_score resources matched by a resource_query.

static bind(job: BinningJob, grr: GenomicResourceRepo) → FragmentScoreBinding

Bind the job to one FragmentScore per resource.

With a cell grouping, the metadata table is read once and each resource’s barcodes are mapped to the job’s track indexes.

kind: ClassVar[str] = 'fragment_score_binner'

static load_def(text: str) → FragmentScoreDef

Read a binner_def of this kind.

classmethod parse_entry(label: str, config: dict[str, Any], grr: GenomicResourceRepo, *, base_dir: str | None = None) → list[BinningJob]

Resolve one entry into a pooled job, or one job per resource.

A job’s tracks are its groups: the constant one, or every distinct non-empty group of its resources’ filtered rows, in sorted order. base_dir is the directory a relative local meta file is read from; unset, the current directory.

gain.binning.fragment_def module

The typed definition of a fragment-score track (binner_def).

class FragmentScoreDef(aggregator: str, mode: str, uncovered_value: float | None, score_id: str | None, resource_query: str, pool: bool, name: str | None = None)

Bases: object

A fragment-score track’s definition.

score_id is the score a fragment contributes, None for the constant value. resource_query, pool and name come from the entry. The definition keeps them so it tells entries apart. The definition does not store the track’s in-memory name.

aggregator: str

classmethod from_json(text: str) → Self

The definition to_json wrote.

mode: str

name: str | None = None

pool: bool

resource_query: str

score_id: str | None

to_json() → str

The definition as canonical JSON.

uncovered_value: float | None

gain.binning.fragment_folds module

Streaming bin folds: fragments in, the bins of one track out.

A fold reduces the fragments of one track over one region, one grid bin at a time. It is fed FragmentValue fragments in start order and answers BinValue bins, each as soon as no later fragment can reach it; BinFragmentAggregator.flush() answers the rest, so every bin of the region is answered exactly once, in order, whether or not any fragment arrived.

The bins are those of the global grid anchored at position 1: the first is the one holding the region’s start, the last the one holding its end, and each is a whole grid bin, calc_bin_begin to calc_bin_end, not clipped to the region. So the bins of adjacent regions split on a bin edge follow on from each other.

What an empty bin holds is EMPTY_BIN_VALUES, applied when the bin is answered: a bin whose aggregator received no non-null value answers 0 for count and sum and NaN for the others.

BinFragmentStartAggregator puts a fragment in the bin holding its start; BinFragmentLengthAggregator in every bin it overlaps, weighted by the overlap in base pairs. The length fold reads a fragment as a run (start, end, value) only, so any runs in start order can be binned with it. BinFragmentCoverageAggregator bins the fragments’ coverage profile, CoverageProfile: it feeds the profile’s runs to a length fold.

class BinFragmentAggregator

Bases: ABC

A streaming fold of one track’s fragments into grid bins.

abstractmethod feed(fragment: FragmentValue) → list[BinValue]

Fold fragment; answer the bins it shows are complete.

abstractmethod flush() → list[BinValue]

Answer every bin not answered yet, to the region’s last.

class BinFragmentCoverageAggregator(*, start: int, end: int, bin_size: int, aggregator: str, uncovered_value: float | None = None)

Bases: BinFragmentAggregator

Each bin reduces the coverage profile of the fragments over it.

The fed fragments make a CoverageProfile, and its runs are folded by a BinFragmentLengthAggregator: each bin reduces the profile’s values weighted by the base pairs they hold in it. A position the profile leaves at 0 is uncovered, so with an uncovered_value it adds that value with a weight of 1 bp, and with None nothing. Fed fragments must lie inside [start, end] – a caller clips them to the region – and arrive in start order; any other is refused with a ValueError.

feed(fragment: FragmentValue) → list[BinValue]

Fold fragment; answer the bins it shows are complete.

flush() → list[BinValue]

Answer every bin not answered yet, to the region’s last.

class BinFragmentLengthAggregator(*, start: int, end: int, bin_size: int, aggregator: str, uncovered_value: float | None = None)

Bases: BinFragmentAggregator

Each bin reduces the values of the fragments that OVERLAP it.

A fragment adds its value to every bin it reaches, weighted by the base pairs it shares with the bin. Fed fragments must lie inside [start, end] – a caller clips them to the region – and arrive in start order; any other is refused with a ValueError.

With an uncovered_value, each base of [start, end] that no fed fragment covers adds that value with a weight of 1 bp, so no bin is empty; the part of an edge bin outside the region is not filled. With None, an uncovered base adds nothing.

The fragments come in start order, but an early long fragment can end after a later short one: each bin keeps its own aggregator until a fragment starts past it, or until flush(), so a bin is answered only when nothing fed later can reach it.

covered_end

The last base a fed fragment covers, start - 1 before any.

feed(fragment: FragmentValue) → list[BinValue]

Fold fragment; answer the bins it shows are complete.

flush() → list[BinValue]

Answer every bin not answered yet, to the region’s last.

pending: deque[Aggregator]

The aggregators of the bins from current_bin on.

class BinFragmentStartAggregator(*, start: int, end: int, bin_size: int, aggregator: str)

Bases: BinFragmentAggregator

Each bin reduces the values of the fragments that START in it.

A fragment belongs to the bin holding its start and to no other, however far it runs; its end is not read. A fragment starting outside [start, end], or before the one fed ahead of it, is refused with a ValueError when it is fed. An exact int result past the float range (sum or product of an int score) saturates to an infinity of its sign.

feed(fragment: FragmentValue) → list[BinValue]

Fold fragment; answer the bins it shows are complete.

flush() → list[BinValue]

Answer every bin not answered yet, to the region’s last.

class BinValue(start: int, end: int, value: float)

Bases: NamedTuple

One grid bin, [start, end] closed, and its folded value.

end: int

Alias for field number 1

start: int

Alias for field number 0

value: float

Alias for field number 2

class CoverageProfile

Bases: object

The coverage profile of fragments fed in start order, as runs.

The profile’s value at a position is the sum of the values of the fed fragments that cover it. It is answered as runs (start, end, value), in order: each a maximal stretch of one non-zero value, so two adjacent runs never share a value, and a position the profile leaves at 0 – no fragment, or values cancelling to 0 – is in no run. A fragment with a None value adds nothing. The sum keeps the type of the values: int values sum to an int, and any float among them makes it a correctly rounded float sum.

feed() answers the runs a fragment shows are complete, and flush() the rest. A fragment starting before the one fed ahead of it is refused with a ValueError. Nothing is kept per base pair, so the cost is per fragment, however long a fragment or a gap between fragments is.

active: list[tuple[int, int, int | float]]

(end, order, value) of each fragment covering cursor.

cursor: int | None

The first position whose profile no run has answered yet.

feed(fragment: FragmentValue) → list[FragmentValue]

Add fragment; answer the runs ending before its start.

flush() → list[FragmentValue]

Answer every run not answered yet.

held: FragmentValue | None

The last run, held back until the next shows it does not continue it with the same value.

level: int | float

previous: int | None

class FragmentValue(start: int, end: int, value: ScoreValue)

Bases: NamedTuple

One fragment, [start, end] closed, and the value it adds.

end: int

Alias for field number 1

start: int

Alias for field number 0

value: str | int | float | bool | None

Alias for field number 2

coverage_profile(fragments: Iterable[FragmentValue]) → Iterator[FragmentValue]

The runs of the coverage profile of fragments, in start order.

See CoverageProfile.

gain.binning.fragment_grouping module

The configuration side of the fragment_score_binner binner.

The GRR convention’s constants for single-cell fragment resources, the parsers of an entry’s value, aggregate, group and meta blocks, and the resolution of a cell grouping from a metadata table and of a value grouping from a score’s histogram. The binding of a job to its resources and the fold of its fragments into bins are not here.

BARCODE_COLUMN = 'barcode'

The metadata table’s column holding a cell’s barcode.

CELL_GROUP_DEFAULTS = {'cell_meta_column': 'barcode', 'cell_score_id': 'cell', 'group_meta_column': 'class'}

The keys of a metadata grouping, each with its default.

CELL_META_RESOURCE_ID_LABEL = 'cell_meta_resource_id'

The fragment resource’s label naming its cell-metadata data_frame.

CELL_SCORE = 'cell'

The fragment score carrying a fragment’s barcode.

CLASS_COLUMN = 'class'

The metadata table’s column holding a cell’s group.

CONVENTION_META = _MetaSpec(filters=(MetaFilter(column='sample_id', value=None, label='sample_id'),), table=None, resource_label='cell_meta_resource_id')

the convention’s table.

Type:

meta omitted under a metadata grouping

COUNT_SCORE = 'count'

The fragment score holding a fragment’s read pairs.

COVERAGE_PROFILE = 'coverage_profile'

The fragment mode reducing, per bin, the coverage profile of the fragments, weighted by base pairs; a base at 0 is uncovered.

class CellGrouping(cell_score_id: str, cell_meta_column: str, group_meta_column: str, table: MetaTable, filters: tuple[MetaFilter, ...])

Bases: object

How a fragment’s barcode becomes a group: a filtered table lookup.

cell_score_id is the fragment score carrying the barcode; the metadata table, its rows selected per fragment resource by filters, maps the barcode in cell_meta_column to the group in group_meta_column.

cell_meta_column: str

cell_score_id: str

property dropped_reason: str

filters: tuple[MetaFilter, ...]

group_meta_column: str

groups_of(table: DataFrame, resource: GenomicResource, prepared: dict[str, _TextColumn] | None = None) → dict[str, str]

Map each barcode of resource’s rows to its group.

The rows are those every filter conjunct selects, as a boolean mask; a row whose group is empty names no group and is left out, so its barcode maps nowhere. A barcode on more than one row is a DuplicateBarcodeError. prepared is prepare_filters() of table, when the caller holds it.

property parameters: dict[str, Any]

prepare_filters(table: DataFrame) → dict[str, _TextColumn]

Each filter column of table as text, converted once.

property score_id: str

The fragment score a group is looked up by.

table: MetaTable

track_maps(grr: GenomicResourceRepo, resources: list[GenomicResource], index_of: dict[str, int]) → list[dict[str, int]]

Per resource, each barcode’s track index; the table read once.

DEFAULT_GROUP = 'all'

The group of an entry that names none.

exception DuplicateBarcodeError

Bases: ValueError

A barcode on more than one of a resource’s filtered rows.

FILE_FORMATS: dict[str, str | None] = {'csv': ',', 'excel': None, 'tsv': '\t'}

A local table’s formats, each with its separator (none for excel).

FRAGMENT_LENGTH = 'fragment_length'

The fragment mode reducing, per bin, the fragments that overlap it, each weighted by its overlap in base pairs.

FRAGMENT_START = 'fragment_start'

The fragment mode reducing, per bin, the fragments that start in it.

MODES = ('fragment_start', 'fragment_length', 'coverage_profile')

The accepted aggregate.mode values.

MOVED_TO_VALUE = {'score': 'score_id', 'value': 'value'}

The keys aggregate does not take, each with the value key that says what one fragment adds.

class MetaFilter(column: str, value: str | None = None, label: str | None = None)

Bases: object

One conjunct of a meta filter: column equals an operand.

The operand is the literal value, or – when label is set – the value of that label on the fragment resource whose rows are being selected.

column: str

label: str | None = None

operand(resource: GenomicResource) → str | None

The text this conjunct’s column must equal, for resource.

value: str | None = None

class MetaTable(resource_id: str | None = None, path: str | None = None, file_format: str = 'csv', file_separator: str | None = None)

Bases: object

Where a cell-metadata table is read from.

A data_frame resource, resource_id, or a local file at the absolute path, read as file_format with file_separator (the format’s own when unset). The parent resolving a run and a worker binding a job read it through the same load().

file_format: str = 'csv'

file_separator: str | None = None

property is_local: bool

load(grr: GenomicResourceRepo) → DataFrame

Read the table, from grr or from the local file.

property name: str

The resource id, or the local file’s absolute path.

property parameters: dict[str, Any]

path: str | None = None

resource_id: str | None = None

SAMPLE_ID_COLUMN = 'sample_id'

The metadata table’s column naming a row’s sample.

SAMPLE_ID_LABEL = 'sample_id'

The fragment resource’s label naming its sample.

VALUE_GROUP_KEY = 'group_score_id'

the str score whose values group.

Type:

The key of a raw-value grouping

class ValueGrouping(group_score_id: str, prefixes: tuple[str, ...], values: tuple[tuple[str, ...], ...])

Bases: object

How a fragment’s raw value of a str score becomes a group.

The group of a fragment of the job’s index-th resource is prefixes[index] followed by its group_score_id value – the resource’s <sample_id>: in a pooled job, nothing in an unpooled one – when values[index], the values its histogram lists, has that value, and none otherwise.

property dropped_reason: str

group_score_id: str

property parameters: dict[str, Any]

prefixes: tuple[str, ...]

property score_id: str

The fragment score whose value is the group.

track_maps(grr: GenomicResourceRepo, resources: list[GenomicResource], index_of: dict[str, int]) → list[dict[str, int]]

Per resource, each of its values’ track index.

values: tuple[tuple[str, ...], ...]

gain.binning.h5_reader module

Read a binning_tool HDF5 file as DataFrames and a numpy matrix.

type Source = str | PathLike[str] | File

read_bins(source: Source) → DataFrame

Read /bins of a binning_tool file.

read_tracks(source: Source) → DataFrame

Read /tracks of a tracks_layout = 2 file.

read_values(source: Source) → ndarray[tuple[Any, ...], dtype[float64]]

Read /values of a binning_tool file as a numpy matrix.

The matrix has the shape (n_bins, n_tracks): row i belongs to row i of /bins and column j to row j of /tracks. A bin with no data is NaN. The whole matrix is read into memory. For a large file, slice h5["values"][start:stop] with h5py instead.

Parameters:

source – A path or an open h5py.File. A path is opened and closed inside the call; an open file is only read, never closed.

Returns:

A float64 array of two dimensions.

Return type:

numpy.ndarray

Raises:

ValueError – If /values is missing, is not a dataset or is not two-dimensional. The message names the file and the dataset.

track_labels(tracks_df: DataFrame) → list[str]

Return one display label for each track, in row order.

The label is read from resource_ids, selector and the JSON binner_def alone. Equal labels get #<position> appended.

gain.binning.position_fold module

The array fold of a position score into grid bins.

ArrayBinFold folds the batches of fetch_region_value_arrays() into the bins of the global grid with numpy. It gives the bins that get_score_in_bins() gives, bit for bit, for a float score under the aggregators of ARRAY_FOLD_AGGREGATORS.

The fold keeps the rules of the per-run fold:

• Where records overlap, the first record answers. A later record covers only the positions that no earlier record claimed, also across the boundary between two batches.

• A record with a missing value claims its positions. It is the same as a stretch that no record covers, and the replacement stands in for both. The caller marks the missing values in a mask of their own: a NaN value of the score is a value, as it is to the per-run fold.

• A run that crosses a bin boundary splits at the boundary, and each part has the weight of its length.

• sum and mean add value * length for each part, in position order, from the start of the bin. The numpy reductions that use pairwise summation (np.add.reduceat, ndarray.sum) give other sums, so the fold adds one part at a time for each bin: across the bins in one numpy operation, or along one bin with np.add.accumulate, which adds in sequence. An open bin at the end of a batch continues its running sum in the next batch.

• median keeps one (value, weight) pair for each part with a value, as MedianAggregator does. The fold keeps the pairs of the open bin only: the bin that holds the cursor, the first position that no record claimed yet. The bins before the cursor are closed, and the fold computes their medians at the end of each batch. A closed bin without a NaN value gets a weighted median from a stable sort by value, which keeps two equal values (-0.0 and 0.0) in position order, as list.sort does. A closed bin with a NaN value goes to MedianAggregator itself, with its pairs in position order: Python sorts a list with a NaN in an order that depends on the input, and a numpy sort puts NaN last.

docs/adr/0038-the-position-binner-folds-value-arrays-with-numpy.md records why the fold exists and what it was measured against.

ARRAY_FOLD_AGGREGATORS = frozenset({'count', 'max', 'mean', 'median', 'min', 'sum'})

The aggregators the array fold serves. product keeps the per-run fold.

class ArrayBinFold(aggregator: str, replacement: float | None, start: int, end: int, bin_size: int)

Bases: object

Fold value-array batches of one region into its grid bins.

Give add_batch() the batches of the region in their read order, then call finish() once. start and end are the 1-based closed bounds of the region.

add_batch(pos_begin: ndarray[tuple[Any, ...], dtype[int64]], pos_end: ndarray[tuple[Any, ...], dtype[int64]], values: ndarray[tuple[Any, ...], dtype[float64]], missing: ndarray[tuple[Any, ...], dtype[bool]]) → None

Fold one batch of records, given in their read order.

missing marks the records without a value. A NaN in values that missing does not mark is a value.

property cursor: int

The first position that no record claimed yet.

finish() → ndarray[tuple[Any, ...], dtype[float64]]

The value of each bin, NaN for a bin without a value.

A count bin without a value is 0.0: a count of nothing is zero, as CountAggregator answers.

property held_pairs: list[tuple[int, float, int]]

The median pairs that the fold holds for the open bin.

Each pair is the grid bin index, the value and the weight of one part, in position order. Only median holds pairs.

replacement_is_exact(replacement: float | None, bin_size: int) → bool

Whether the array fold adds replacement as the aggregators do.

The aggregators keep an int replacement exact, and the fold adds it as a float. The two agree while every sum of the replacement in one bin stays within the integers that a float64 holds exactly.

gain.binning.run_definition module

The binning_tool run definition: parsing and resolution.

class RunDefinition(reference_genome: str, bin_size: int, regions: list[BedRegion], jobs: list[BinningJob])

Bases: object

A parsed run definition with every query resolved.

jobs are what the tasks bind to; tracks are the output’s columns, every job’s tracks in job order.

bin_size: int

jobs: list[BinningJob]

reference_genome: str

regions: list[BedRegion]

property tracks: list[Track]

parse_run_definition(config: dict[str, Any], grr: GenomicResourceRepo, genome: ReferenceGenome, base_dir: str | None = None) → RunDefinition

Resolve config against grr and genome.

Every key is checked: a mistyped key is an error, never a silently applied default. Raises RunDefinitionError naming the offending entry. base_dir is the run-definition file’s directory, which a relative path in an entry is read from; a config parsed from memory leaves it unset, and the kinds read such a path from the current directory.

refuse_renamed_genome_key(config: dict[str, Any]) → None

Refuse a run definition that still has the pre-rename genome key.

A definition with both keys is refused too: the old key is never read.

gain.binning.view_cli module

binning_tool_view: print the tables of a binning_tool file.

The tables are /tracks, /bins and /values, printed as tab-separated text on stdout. /values is read one block of rows at a time, so the memory for the values stays flat for a file of any size. The memory for /bins and for the selected rows grows with the number of bins.

exception ViewError

Bases: Exception

A user error: the message is the whole report.

cli(argv: Sequence[str] | None = None) → int

Run binning_tool_view; return the exit code.

Module contents

Bin position scores into a fixed genome grid (binning_tool).