binning_tool

binning_tool bins genomic scores into a fixed genome grid. It reads a run definition in YAML that names the reference genome, the bins and the position_score and fragment_score resources to bin. It writes one HDF5 file with a bins × tracks matrix. Binning infrastructure describes the run definition and the output file.

binning_tool run.yaml

Examples

binning_tool run.yaml --dry-run

Resolves every query of run.yaml against the GRR and prints the tracks, regions, bins and tasks of the run. Writes nothing.

binning_tool run.yaml -o run.h5 -j 8

Bins the scores of run.yaml with 8 parallel workers and writes the result to run.h5.

binning_tool run.yaml --force

Computes every chunk again, also the chunks that an earlier run left in the work directory.

Running the tool

The full list of options is in the Option reference at the end of this page.

The only positional argument is the run definition. -o names the HDF5 file to write; by default it is the run definition’s path with an .h5 suffix, beside it, so binning_tool run.yaml writes run.h5.

What the run describes — the genome, the grid and the tracks — is the run definition’s business; the command line only says where to find the GRR, where to put the output, and how to run the work.

Dry run

--dry-run reads the run definition, resolves every query against the GRR, checks every rule of The run definition, and exits without writing an output file or a work directory. Resolving a raw-value grouping (group: {group_score_id: ...}) still reads each resource’s full histogram, and a caching GRR stores those files in its cache. It prints the list of tracks the run would produce — name, resource ids (comma-separated), score id (empty for a fragment count) and aggregator — together with the number of regions, bins and tasks. For every fragment_score_binner entry it then reports the resources the entry matched, the metadata tables it reads (resource ids, or the absolute paths of local files), its group and track counts, every warning its resolution raised, and the mode and uncovered value of its tracks. For the two single-cell entries of Fragment-score entries, the convention entry and the explicit unpooled one, in one run definition binning a small two-chromosome genome at 100 bp:

tracks:
  sc/atac_fragments/*:ODC   sc/atac_fragments/donor1,sc/atac_fragments/donor2               sum
  sc/atac_fragments/*:PVALB sc/atac_fragments/donor1,sc/atac_fragments/donor2               sum
  sc/atac_fragments/donor1:ODC      sc/atac_fragments/donor1                sum
  sc/atac_fragments/donor1:PVALB    sc/atac_fragments/donor1                sum
  sc/atac_fragments/donor2:ODC      sc/atac_fragments/donor2                sum
  sc/atac_fragments/donor2:PVALB    sc/atac_fragments/donor2                sum
regions: 2
bins: 14
tasks: 3
binners[0]: fragment_score_binner
    resources: sc/atac_fragments/donor1, sc/atac_fragments/donor2
    tables: sc/cell_meta_data
    groups: 2
    tracks: 2
    mode: fragment_start
    uncovered_value: null
binners[1]: fragment_score_binner
    resources: sc/atac_fragments/donor1, sc/atac_fragments/donor2
    tables: sc/cell_meta_data
    groups: 2
    tracks: 4
    mode: fragment_start
    uncovered_value: null

and for the local-file entry of Local metadata files, the report carries the warning:

binners[0]: fragment_score_binner
    resources: sc/atac_fragments/donor1, sc/atac_fragments/donor2
    tables: /tmp/binning-demo/cells.csv
    groups: 2
    tracks: 2
    warning: the cell metadata table '/tmp/binning-demo/cells.csv' is a local file; the run is not reproducible elsewhere
    mode: fragment_start
    uncovered_value: null

Use it to see what a query matched, how large the matrix will be and how the work will be cut before committing cluster time.

Parallelism

The work is split into one task per job and bundle of regions, followed by one task that assembles the HDF5 file. A job is what one task binds to: one track of a position-score entry, one resource of an unpooled fragment entry with all its group tracks, or the whole of a pooled fragment entry, however many resources it pools. Consecutive regions of the run definition are packed, in order, into bundles of at most --task-budget bases (default 50,000,000); a region is never split, so a chromosome longer than the budget is a task of its own, while the hundreds of alternate and unplaced contigs of a human genome — under two percent of its bases — pack into a handful of tasks instead of one each. The budget only decides how many tasks there are: the file is the same whatever its value. The budget counts bases, not reads: a pooled job reads every one of its resources over each base, so its tasks are that many times heavier, and a pooled job is never split across its samples.

Both ends of the dial are reachable. A budget of 1 is one task per region — a region is never split, so the smallest budget that cuts at all cuts everywhere. A budget of 0 or less is no budget at all: the whole run goes into one task per job. A task opens its job’s resources once, whatever the budget, so the whole-run task opens them once for the run.

Which end helps depends on how many jobs a run has, because a job’s work is never split across tasks at budget 0: the achievable parallelism is then exactly the number of jobs. A run of hundreds of tracks on a few workers loses nothing and sheds tasks; a run of two or three tracks with -j 8 leaves most of the workers idle and should keep the default. Note too that a task is the unit a rerun repeats: at budget 0 a task that fails part-way recomputes its whole job next time, where at the default a rerun keeps every bundle that finished.

The tasks run through the same task graph as the annotation tools, so the same flags apply: -j N sets the number of workers, -N names a configured dask cluster, and --task-log-dir keeps a log per task. The genomic-context flags (-g, --grr-directory, -R) and the verbosity flags (--verbose, --logfile) are likewise the shared ones. Run binning_tool --help for the full list.

Work directory and reruns

Each task writes one column chunk per track and region of its bundle into a work directory; the final task assembles the file from those chunks, region by region, so at no point is the whole matrix in memory — a genome-wide run at a small bin size is possible on an ordinary node. The work directory is -w; by default it is a sibling of the output named after it (run_work next to run.h5), and the task-status directory lives inside it. A work directory the tool created is removed after a successful run; --keep-work-dir keeps it.

A run that was interrupted resumes from the tasks it had finished when it is started again with the same work directory and budget: a task whose chunks are all present is skipped, a task missing any of its chunks is computed again in full, and the file is assembled if it is missing. A run whose chunks and output are all present does nothing. The chunks are keyed by everything in the run definition that decides their values — resources, score, aggregator, replacement, group, the fragment entry’s grouping, metadata settings, mode and uncovered value, bin size and region — so two run definitions sharing a work directory share exactly the chunks they compute identically. A local metadata file (meta: {file_name: ...}) is an input of every task, so editing it recomputes the chunks on the next run. A rerun does not notice that a resource, or a metadata table read from a data_frame resource, has changed underneath it; pass --force to recompute every chunk, for example after a resource was updated in the GRR.

See also

• Binning infrastructure: the run definition, the output file and a worked example.

• Printing a binning file with binning_tool_view: print the tables of a binning_tool file as tab-separated text.

• Running in parallel: the number of workers, the named clusters and the reruns.

Option reference

Bin position and fragment scores into a fixed genome grid, writing one HDF5 file with a bins x tracks matrix.

usage: binning_tool [-h] [-o OUTPUT] [-w WORK_DIR] [--keep-work-dir]
                    [--dry-run] [--task-budget BP] [-g GRR_FILENAME]
                    [--grr-directory GRR_DIRECTORY] [-j JOBS] [--process-pool]
                    [-N DASK_CLUSTER_NAME] [-c DASK_CLUSTER_CONFIG_FILE]
                    [--task-log-dir TASK_LOG_DIR] [-t TASK_IDS [TASK_IDS ...]]
                    [--keep-going] [--fork-tasks] [--force]
                    [-d TASK_STATUS_DIR] [--verbose] [--logfile LOGFILE]
                    run_definition

Positional Arguments

run_definition

the run definition (YAML): bins and binner entries

Named Arguments

-o, --output

the HDF5 file to write; defaults to the run definition with an .h5 suffix, beside it

-w, --work-dir

directory for the per-chunk intermediate files; defaults to a sibling of the output

--keep-work-dir

keep the working directory after a successful run (by default a working directory the tool created is removed)

Default: False

--dry-run

resolve every query, print the track list, the region, bin and task counts and, per fragment entry, its resources, metadata tables, group and track counts and warnings, and write nothing

Default: False

--task-budget

how many bases of consecutive regions one task bins; a region is never split, so a chromosome longer than the budget is a task of its own; 0 or less is one task per track over the whole run, and 1 is one task per region (default: 50_000_000)

Default: 50000000

-g, --grr-filename, --grr

The GRR configuration file. If the argument is absent, the a GRR repository from the current genomic context (i.e. gpf_instance) will be used or, if that fails, the default GRR configuration will be used.

--grr-directory

Local GRR directory to use as repository.

--verbose, -v, -V

Default: 0

--logfile

File to log output to. If not set, logs to console.

Task Graph Executor

-j, --jobs

Number of jobs to run in parallel. Defaults to the number of processors on the machine

--process-pool, --pp

Use a process pool executor with the specified number of processes instead of a dask distributed executor.

Default: False

-N, --dask-cluster-name, --dcn

The name of a named cluster in the dask config, under ‘gain_named_cluster.clusters’. The tasks run on it.

-c, --dccf, --dask-cluster-config-file

dask cluster config file

--task-log-dir

Path to directory where to store tasks’ logs

Execution Mode

-t, --task-ids

--keep-going

Whether or not to keep executing in case of an error

Default: False

--fork-tasks, --fork-task, --fork

Whether to fork a new worker process for each task

Default: False

--force, -f

Ignore precomputed state and always rerun all tasks.

Default: False

-d, --task-status-dir, --tsd

Directory to store the task progress.