grr_manage

grr_manage builds and maintains a Genomic Resource Repository (GRR). It reads the resource directories of a GRR, each with its genomic_resource.yaml and data files. It writes the manifests, the statistics and the info pages of the resources, and the repository-global files: the .CONTENTS files, the search index and the repository index pages. Use it when you create a GRR, after you add or change a resource, and when you publish a GRR. The manifest, stats, info and repair subcommands each have a repo-* form for the whole GRR and a resource-* form for one resource. The list, repo-init and repo-index subcommands have only one form.

Examples

grr_manage repo-init

Turns the current directory into an empty GRR.

grr_manage resource-repair -r hg38/scores/phyloP7way

Builds the manifest, the statistics and the info page of one resource of the GRR that holds the current directory.

grr_manage repo-repair -R /data/my_GRR -j 8

Repairs every resource of the GRR in /data/my_GRR with 8 parallel jobs, then publishes the repository-global files.

See also

Option reference

Genomic Resource Repository Management Tool

usage: grr_manage [-h] [--version] [--verbose] [--logfile LOGFILE]
                  {list,repo-init,repo-manifest,resource-manifest,repo-stats,resource-stats,repo-info,resource-info,repo-repair,resource-repair,repo-index}
                  ...

Positional Arguments

command

Possible choices: list, repo-init, repo-manifest, resource-manifest, repo-stats, resource-stats, repo-info, resource-info, repo-repair, resource-repair, repo-index

Command to execute

Named Arguments

--version

Prints the GAIn version and exits.

Default: False

--verbose, -v, -V

Default: 0

--logfile

File to log output to. If not set, logs to console.

Sub-commands

list

List a GR Repo

grr_manage list [-h] [--hr] [-s SEARCH] [-t TYPE] [-q QUERY] [-R REPOSITORY]
                [--grr GRR] [--extra-args EXTRA_ARGS] [--verbose]
                [--logfile LOGFILE]
Named Arguments
--hr

Projects the size in human-readable format.

Default: False

-s, --search

FTS search term to filter resources. A term may name one index column, e.g. ‘assay_term_name: “ATAC-seq”’, but the columns are per-repository – across a group, a repository that does not index the column is skipped with a warning rather than searched. Use -q to select on labels independently of any index.

-t, --type

Filter resources by type.

-q, --query

Filter resources by a wildcard query over the resource id and labels, e.g. ‘hg38/scores/[phenotype=”autism”]’. A label a resource does not carry reads as empty, so [key=””] holds for every resource rather than selecting the ones that have the label.

--verbose, -v, -V

Default: 0

--logfile

File to log output to. If not set, logs to console.

Repository/Resource
-R, --repository

URL to the genomic resources repository. If not specified the tool assumes a local file system repository and starts looking for .CONTENTS.json.gz file from the current working directory up to the root directory. If found the directory is assumed for root repository directory; otherwise error is reported.

--grr, --definition, -g

Path to an extra GRR definition file. This GRR will be loadedin a group alongside the local one.

--extra-args

comma separated list of key=value pairs arguments needed for connection to the specific repository protocol. Ex: if you want to connect to an S3 repository it is often neccessary to pass additional endpoint-url argument.

repo-init

Initialize a directory to turn it into a GRR

grr_manage repo-init [-h] [-R REPOSITORY] [--grr GRR]
                     [--extra-args EXTRA_ARGS] [--verbose] [--logfile LOGFILE]
Named Arguments
--verbose, -v, -V

Default: 0

--logfile

File to log output to. If not set, logs to console.

Repository/Resource
-R, --repository

URL to the genomic resources repository. If not specified the tool assumes a local file system repository and starts looking for .CONTENTS.json.gz file from the current working directory up to the root directory. If found the directory is assumed for root repository directory; otherwise error is reported.

--grr, --definition, -g

Path to an extra GRR definition file. This GRR will be loadedin a group alongside the local one.

--extra-args

comma separated list of key=value pairs arguments needed for connection to the specific repository protocol. Ex: if you want to connect to an S3 repository it is often neccessary to pass additional endpoint-url argument.

repo-manifest

Create/update manifests for whole GRR

grr_manage repo-manifest [-h] [-R REPOSITORY] [--grr GRR]
                         [--extra-args EXTRA_ARGS] [-n] [-f] [--with-dvc] [-D]
                         [--verbose] [--logfile LOGFILE]
Named Arguments
--verbose, -v, -V

Default: 0

--logfile

File to log output to. If not set, logs to console.

Repository/Resource
-R, --repository

URL to the genomic resources repository. If not specified the tool assumes a local file system repository and starts looking for .CONTENTS.json.gz file from the current working directory up to the root directory. If found the directory is assumed for root repository directory; otherwise error is reported.

--grr, --definition, -g

Path to an extra GRR definition file. This GRR will be loadedin a group alongside the local one.

--extra-args

comma separated list of key=value pairs arguments needed for connection to the specific repository protocol. Ex: if you want to connect to an S3 repository it is often neccessary to pass additional endpoint-url argument.

Force/Dry run
-n, --dry-run

report whether the manifest needs updating and write nothing: no manifest, and no recorded file state either. The run still reads and hashes whatever answering the question takes, it just keeps no receipt - so it leaves the repository byte-identical, and seeds nothing for the next run to reuse

Default: False

-f, --force

ignore resource state and rebuild manifest

Default: False

DVC params
--with-dvc

trust a ‘.dvc’ sidecar as the md5 sum and size of the file it describes, and never hash a DVC-managed file (default). A file GAIn has already hashed keeps its recorded md5 sum while its size and timestamp are unchanged; a file with no sidecar and no usable state is hashed

Default: True

-D, --without-dvc

verify mode: ignore every recorded state, compute from its content the md5 sum of every resource file that is on disk, and check it against the file’s ‘.dvc’ sidecar. The run reports every file that disagrees and exits non-zero, writing no manifest for the resources they belong to; fix the drift with ‘dvc add’ / ‘dvc commit’. A file that is NOT on disk still takes its md5 sum and size from its sidecar - there is no content to hash, and its manifest entry is never dropped

Default: True

resource-manifest

Create/update manifests for a resource

grr_manage resource-manifest [-h] [-R REPOSITORY] [--grr GRR]
                             [--extra-args EXTRA_ARGS] [-r RESOURCE] [-n] [-f]
                             [--with-dvc] [-D] [--verbose] [--logfile LOGFILE]
Named Arguments
--verbose, -v, -V

Default: 0

--logfile

File to log output to. If not set, logs to console.

Repository/Resource
-R, --repository

URL to the genomic resources repository. If not specified the tool assumes a local file system repository and starts looking for .CONTENTS.json.gz file from the current working directory up to the root directory. If found the directory is assumed for root repository directory; otherwise error is reported.

--grr, --definition, -g

Path to an extra GRR definition file. This GRR will be loadedin a group alongside the local one.

--extra-args

comma separated list of key=value pairs arguments needed for connection to the specific repository protocol. Ex: if you want to connect to an S3 repository it is often neccessary to pass additional endpoint-url argument.

-r, --resource

Specifies the resource whose manifest we want to rebuild. If not specified the tool assumes local filesystem repository and starts looking for ‘genomic_resource.yaml’ file from current working directory up to the root directory. If found the directory is assumed for a resource directory; otherwise error is reported.

Force/Dry run
-n, --dry-run

report whether the manifest needs updating and write nothing: no manifest, and no recorded file state either. The run still reads and hashes whatever answering the question takes, it just keeps no receipt - so it leaves the repository byte-identical, and seeds nothing for the next run to reuse

Default: False

-f, --force

ignore resource state and rebuild manifest

Default: False

DVC params
--with-dvc

trust a ‘.dvc’ sidecar as the md5 sum and size of the file it describes, and never hash a DVC-managed file (default). A file GAIn has already hashed keeps its recorded md5 sum while its size and timestamp are unchanged; a file with no sidecar and no usable state is hashed

Default: True

-D, --without-dvc

verify mode: ignore every recorded state, compute from its content the md5 sum of every resource file that is on disk, and check it against the file’s ‘.dvc’ sidecar. The run reports every file that disagrees and exits non-zero, writing no manifest for the resources they belong to; fix the drift with ‘dvc add’ / ‘dvc commit’. A file that is NOT on disk still takes its md5 sum and size from its sidecar - there is no content to hash, and its manifest entry is never dropped

Default: True

repo-stats

Build the statistics for a resource

grr_manage repo-stats [-h] [-R REPOSITORY] [--grr GRR]
                      [--extra-args EXTRA_ARGS] [-n] [-f] [--with-dvc] [-D]
                      [--region-size REGION_SIZE] [--verbose]
                      [--logfile LOGFILE] [-j JOBS] [--process-pool]
                      [-N DASK_CLUSTER_NAME] [-c DASK_CLUSTER_CONFIG_FILE]
                      [--task-log-dir TASK_LOG_DIR]
                      [-t TASK_IDS [TASK_IDS ...]] [--keep-going]
                      [--fork-tasks]
Named Arguments
--verbose, -v, -V

Default: 0

--logfile

File to log output to. If not set, logs to console.

Repository/Resource
-R, --repository

URL to the genomic resources repository. If not specified the tool assumes a local file system repository and starts looking for .CONTENTS.json.gz file from the current working directory up to the root directory. If found the directory is assumed for root repository directory; otherwise error is reported.

--grr, --definition, -g

Path to an extra GRR definition file. This GRR will be loadedin a group alongside the local one.

--extra-args

comma separated list of key=value pairs arguments needed for connection to the specific repository protocol. Ex: if you want to connect to an S3 repository it is often neccessary to pass additional endpoint-url argument.

Force/Dry run
-n, --dry-run

report whether the manifest needs updating and write nothing: no manifest, and no recorded file state either. The run still reads and hashes whatever answering the question takes, it just keeps no receipt - so it leaves the repository byte-identical, and seeds nothing for the next run to reuse

Default: False

-f, --force

ignore resource state and rebuild manifest

Default: False

DVC params
--with-dvc

trust a ‘.dvc’ sidecar as the md5 sum and size of the file it describes, and never hash a DVC-managed file (default). A file GAIn has already hashed keeps its recorded md5 sum while its size and timestamp are unchanged; a file with no sidecar and no usable state is hashed

Default: True

-D, --without-dvc

verify mode: ignore every recorded state, compute from its content the md5 sum of every resource file that is on disk, and check it against the file’s ‘.dvc’ sidecar. The run reports every file that disagrees and exits non-zero, writing no manifest for the resources they belong to; fix the drift with ‘dvc add’ / ‘dvc commit’. A file that is NOT on disk still takes its md5 sum and size from its sidecar - there is no content to hash, and its manifest entry is never dropped

Default: True

Statistics
--region-size

Region size to use for splitting statistics calculation into tasks. The default is one region per contig; 0 does not split. A smaller size balances one large resource across -j workers: keep it at or below about genome / (4 x -j), e.g. –region-size 20000000 for hg38 at -j 32

Default: 3000000000

Task Graph Executor
-j, --jobs

Number of jobs to run in parallel. Defaults to the number of processors on the machine

--process-pool, --pp

Use a process pool executor with the specified number of processes instead of a dask distributed executor.

Default: False

-N, --dask-cluster-name, --dcn

The name of a named cluster in the dask config, under ‘gain_named_cluster.clusters’. The tasks run on it.

-c, --dccf, --dask-cluster-config-file

dask cluster config file

--task-log-dir

Path to directory where to store tasks’ logs

Execution Mode
-t, --task-ids
--keep-going

Whether or not to keep executing in case of an error

Default: False

--fork-tasks, --fork-task, --fork

Whether to fork a new worker process for each task

Default: False

resource-stats

Build the statistics for a resource

grr_manage resource-stats [-h] [-R REPOSITORY] [--grr GRR]
                          [--extra-args EXTRA_ARGS] [-r RESOURCE] [-n] [-f]
                          [--with-dvc] [-D] [--region-size REGION_SIZE]
                          [--verbose] [--logfile LOGFILE] [-j JOBS]
                          [--process-pool] [-N DASK_CLUSTER_NAME]
                          [-c DASK_CLUSTER_CONFIG_FILE]
                          [--task-log-dir TASK_LOG_DIR]
                          [-t TASK_IDS [TASK_IDS ...]] [--keep-going]
                          [--fork-tasks]
Named Arguments
--verbose, -v, -V

Default: 0

--logfile

File to log output to. If not set, logs to console.

Repository/Resource
-R, --repository

URL to the genomic resources repository. If not specified the tool assumes a local file system repository and starts looking for .CONTENTS.json.gz file from the current working directory up to the root directory. If found the directory is assumed for root repository directory; otherwise error is reported.

--grr, --definition, -g

Path to an extra GRR definition file. This GRR will be loadedin a group alongside the local one.

--extra-args

comma separated list of key=value pairs arguments needed for connection to the specific repository protocol. Ex: if you want to connect to an S3 repository it is often neccessary to pass additional endpoint-url argument.

-r, --resource

Specifies the resource whose manifest we want to rebuild. If not specified the tool assumes local filesystem repository and starts looking for ‘genomic_resource.yaml’ file from current working directory up to the root directory. If found the directory is assumed for a resource directory; otherwise error is reported.

Force/Dry run
-n, --dry-run

report whether the manifest needs updating and write nothing: no manifest, and no recorded file state either. The run still reads and hashes whatever answering the question takes, it just keeps no receipt - so it leaves the repository byte-identical, and seeds nothing for the next run to reuse

Default: False

-f, --force

ignore resource state and rebuild manifest

Default: False

DVC params
--with-dvc

trust a ‘.dvc’ sidecar as the md5 sum and size of the file it describes, and never hash a DVC-managed file (default). A file GAIn has already hashed keeps its recorded md5 sum while its size and timestamp are unchanged; a file with no sidecar and no usable state is hashed

Default: True

-D, --without-dvc

verify mode: ignore every recorded state, compute from its content the md5 sum of every resource file that is on disk, and check it against the file’s ‘.dvc’ sidecar. The run reports every file that disagrees and exits non-zero, writing no manifest for the resources they belong to; fix the drift with ‘dvc add’ / ‘dvc commit’. A file that is NOT on disk still takes its md5 sum and size from its sidecar - there is no content to hash, and its manifest entry is never dropped

Default: True

Statistics
--region-size

Region size to use for splitting statistics calculation into tasks. The default is one region per contig; 0 does not split. A smaller size balances one large resource across -j workers: keep it at or below about genome / (4 x -j), e.g. –region-size 20000000 for hg38 at -j 32

Default: 3000000000

Task Graph Executor
-j, --jobs

Number of jobs to run in parallel. Defaults to the number of processors on the machine

--process-pool, --pp

Use a process pool executor with the specified number of processes instead of a dask distributed executor.

Default: False

-N, --dask-cluster-name, --dcn

The name of a named cluster in the dask config, under ‘gain_named_cluster.clusters’. The tasks run on it.

-c, --dccf, --dask-cluster-config-file

dask cluster config file

--task-log-dir

Path to directory where to store tasks’ logs

Execution Mode
-t, --task-ids
--keep-going

Whether or not to keep executing in case of an error

Default: False

--fork-tasks, --fork-task, --fork

Whether to fork a new worker process for each task

Default: False

repo-info

Build the index.html for the whole GRR

grr_manage repo-info [-h] [-R REPOSITORY] [--grr GRR]
                     [--extra-args EXTRA_ARGS] [-r RESOURCE] [-n] [-f]
                     [--with-dvc] [-D] [--region-size REGION_SIZE] [--verbose]
                     [--logfile LOGFILE] [-j JOBS] [--process-pool]
                     [-N DASK_CLUSTER_NAME] [-c DASK_CLUSTER_CONFIG_FILE]
                     [--task-log-dir TASK_LOG_DIR]
                     [-t TASK_IDS [TASK_IDS ...]] [--keep-going]
                     [--fork-tasks]
Named Arguments
--verbose, -v, -V

Default: 0

--logfile

File to log output to. If not set, logs to console.

Repository/Resource
-R, --repository

URL to the genomic resources repository. If not specified the tool assumes a local file system repository and starts looking for .CONTENTS.json.gz file from the current working directory up to the root directory. If found the directory is assumed for root repository directory; otherwise error is reported.

--grr, --definition, -g

Path to an extra GRR definition file. This GRR will be loadedin a group alongside the local one.

--extra-args

comma separated list of key=value pairs arguments needed for connection to the specific repository protocol. Ex: if you want to connect to an S3 repository it is often neccessary to pass additional endpoint-url argument.

-r, --resource

Specifies the resource whose manifest we want to rebuild. If not specified the tool assumes local filesystem repository and starts looking for ‘genomic_resource.yaml’ file from current working directory up to the root directory. If found the directory is assumed for a resource directory; otherwise error is reported.

Force/Dry run
-n, --dry-run

report whether the manifest needs updating and write nothing: no manifest, and no recorded file state either. The run still reads and hashes whatever answering the question takes, it just keeps no receipt - so it leaves the repository byte-identical, and seeds nothing for the next run to reuse

Default: False

-f, --force

ignore resource state and rebuild manifest

Default: False

DVC params
--with-dvc

trust a ‘.dvc’ sidecar as the md5 sum and size of the file it describes, and never hash a DVC-managed file (default). A file GAIn has already hashed keeps its recorded md5 sum while its size and timestamp are unchanged; a file with no sidecar and no usable state is hashed

Default: True

-D, --without-dvc

verify mode: ignore every recorded state, compute from its content the md5 sum of every resource file that is on disk, and check it against the file’s ‘.dvc’ sidecar. The run reports every file that disagrees and exits non-zero, writing no manifest for the resources they belong to; fix the drift with ‘dvc add’ / ‘dvc commit’. A file that is NOT on disk still takes its md5 sum and size from its sidecar - there is no content to hash, and its manifest entry is never dropped

Default: True

Statistics
--region-size

Region size to use for splitting statistics calculation into tasks. The default is one region per contig; 0 does not split. A smaller size balances one large resource across -j workers: keep it at or below about genome / (4 x -j), e.g. –region-size 20000000 for hg38 at -j 32

Default: 3000000000

Task Graph Executor
-j, --jobs

Number of jobs to run in parallel. Defaults to the number of processors on the machine

--process-pool, --pp

Use a process pool executor with the specified number of processes instead of a dask distributed executor.

Default: False

-N, --dask-cluster-name, --dcn

The name of a named cluster in the dask config, under ‘gain_named_cluster.clusters’. The tasks run on it.

-c, --dccf, --dask-cluster-config-file

dask cluster config file

--task-log-dir

Path to directory where to store tasks’ logs

Execution Mode
-t, --task-ids
--keep-going

Whether or not to keep executing in case of an error

Default: False

--fork-tasks, --fork-task, --fork

Whether to fork a new worker process for each task

Default: False

resource-info

Build the index.html for the specific resource

grr_manage resource-info [-h] [-R REPOSITORY] [--grr GRR]
                         [--extra-args EXTRA_ARGS] [-r RESOURCE] [-n] [-f]
                         [--with-dvc] [-D] [--region-size REGION_SIZE]
                         [--verbose] [--logfile LOGFILE] [-j JOBS]
                         [--process-pool] [-N DASK_CLUSTER_NAME]
                         [-c DASK_CLUSTER_CONFIG_FILE]
                         [--task-log-dir TASK_LOG_DIR]
                         [-t TASK_IDS [TASK_IDS ...]] [--keep-going]
                         [--fork-tasks]
Named Arguments
--verbose, -v, -V

Default: 0

--logfile

File to log output to. If not set, logs to console.

Repository/Resource
-R, --repository

URL to the genomic resources repository. If not specified the tool assumes a local file system repository and starts looking for .CONTENTS.json.gz file from the current working directory up to the root directory. If found the directory is assumed for root repository directory; otherwise error is reported.

--grr, --definition, -g

Path to an extra GRR definition file. This GRR will be loadedin a group alongside the local one.

--extra-args

comma separated list of key=value pairs arguments needed for connection to the specific repository protocol. Ex: if you want to connect to an S3 repository it is often neccessary to pass additional endpoint-url argument.

-r, --resource

Specifies the resource whose manifest we want to rebuild. If not specified the tool assumes local filesystem repository and starts looking for ‘genomic_resource.yaml’ file from current working directory up to the root directory. If found the directory is assumed for a resource directory; otherwise error is reported.

Force/Dry run
-n, --dry-run

report whether the manifest needs updating and write nothing: no manifest, and no recorded file state either. The run still reads and hashes whatever answering the question takes, it just keeps no receipt - so it leaves the repository byte-identical, and seeds nothing for the next run to reuse

Default: False

-f, --force

ignore resource state and rebuild manifest

Default: False

DVC params
--with-dvc

trust a ‘.dvc’ sidecar as the md5 sum and size of the file it describes, and never hash a DVC-managed file (default). A file GAIn has already hashed keeps its recorded md5 sum while its size and timestamp are unchanged; a file with no sidecar and no usable state is hashed

Default: True

-D, --without-dvc

verify mode: ignore every recorded state, compute from its content the md5 sum of every resource file that is on disk, and check it against the file’s ‘.dvc’ sidecar. The run reports every file that disagrees and exits non-zero, writing no manifest for the resources they belong to; fix the drift with ‘dvc add’ / ‘dvc commit’. A file that is NOT on disk still takes its md5 sum and size from its sidecar - there is no content to hash, and its manifest entry is never dropped

Default: True

Statistics
--region-size

Region size to use for splitting statistics calculation into tasks. The default is one region per contig; 0 does not split. A smaller size balances one large resource across -j workers: keep it at or below about genome / (4 x -j), e.g. –region-size 20000000 for hg38 at -j 32

Default: 3000000000

Task Graph Executor
-j, --jobs

Number of jobs to run in parallel. Defaults to the number of processors on the machine

--process-pool, --pp

Use a process pool executor with the specified number of processes instead of a dask distributed executor.

Default: False

-N, --dask-cluster-name, --dcn

The name of a named cluster in the dask config, under ‘gain_named_cluster.clusters’. The tasks run on it.

-c, --dccf, --dask-cluster-config-file

dask cluster config file

--task-log-dir

Path to directory where to store tasks’ logs

Execution Mode
-t, --task-ids
--keep-going

Whether or not to keep executing in case of an error

Default: False

--fork-tasks, --fork-task, --fork

Whether to fork a new worker process for each task

Default: False

repo-repair

Update/rebuild manifest and histograms whole GRR

grr_manage repo-repair [-h] [-R REPOSITORY] [--grr GRR]
                       [--extra-args EXTRA_ARGS] [-n] [-f] [--with-dvc] [-D]
                       [--region-size REGION_SIZE] [--verbose]
                       [--logfile LOGFILE] [-j JOBS] [--process-pool]
                       [-N DASK_CLUSTER_NAME] [-c DASK_CLUSTER_CONFIG_FILE]
                       [--task-log-dir TASK_LOG_DIR]
                       [-t TASK_IDS [TASK_IDS ...]] [--keep-going]
                       [--fork-tasks]
Named Arguments
--verbose, -v, -V

Default: 0

--logfile

File to log output to. If not set, logs to console.

Repository/Resource
-R, --repository

URL to the genomic resources repository. If not specified the tool assumes a local file system repository and starts looking for .CONTENTS.json.gz file from the current working directory up to the root directory. If found the directory is assumed for root repository directory; otherwise error is reported.

--grr, --definition, -g

Path to an extra GRR definition file. This GRR will be loadedin a group alongside the local one.

--extra-args

comma separated list of key=value pairs arguments needed for connection to the specific repository protocol. Ex: if you want to connect to an S3 repository it is often neccessary to pass additional endpoint-url argument.

Force/Dry run
-n, --dry-run

report whether the manifest needs updating and write nothing: no manifest, and no recorded file state either. The run still reads and hashes whatever answering the question takes, it just keeps no receipt - so it leaves the repository byte-identical, and seeds nothing for the next run to reuse

Default: False

-f, --force

ignore resource state and rebuild manifest

Default: False

DVC params
--with-dvc

trust a ‘.dvc’ sidecar as the md5 sum and size of the file it describes, and never hash a DVC-managed file (default). A file GAIn has already hashed keeps its recorded md5 sum while its size and timestamp are unchanged; a file with no sidecar and no usable state is hashed

Default: True

-D, --without-dvc

verify mode: ignore every recorded state, compute from its content the md5 sum of every resource file that is on disk, and check it against the file’s ‘.dvc’ sidecar. The run reports every file that disagrees and exits non-zero, writing no manifest for the resources they belong to; fix the drift with ‘dvc add’ / ‘dvc commit’. A file that is NOT on disk still takes its md5 sum and size from its sidecar - there is no content to hash, and its manifest entry is never dropped

Default: True

Statistics
--region-size

Region size to use for splitting statistics calculation into tasks. The default is one region per contig; 0 does not split. A smaller size balances one large resource across -j workers: keep it at or below about genome / (4 x -j), e.g. –region-size 20000000 for hg38 at -j 32

Default: 3000000000

Task Graph Executor
-j, --jobs

Number of jobs to run in parallel. Defaults to the number of processors on the machine

--process-pool, --pp

Use a process pool executor with the specified number of processes instead of a dask distributed executor.

Default: False

-N, --dask-cluster-name, --dcn

The name of a named cluster in the dask config, under ‘gain_named_cluster.clusters’. The tasks run on it.

-c, --dccf, --dask-cluster-config-file

dask cluster config file

--task-log-dir

Path to directory where to store tasks’ logs

Execution Mode
-t, --task-ids
--keep-going

Whether or not to keep executing in case of an error

Default: False

--fork-tasks, --fork-task, --fork

Whether to fork a new worker process for each task

Default: False

resource-repair

Update/rebuild manifest and histograms for a resource

grr_manage resource-repair [-h] [-R REPOSITORY] [--grr GRR]
                           [--extra-args EXTRA_ARGS] [-r RESOURCE] [-n] [-f]
                           [--with-dvc] [-D] [--region-size REGION_SIZE]
                           [--verbose] [--logfile LOGFILE] [-j JOBS]
                           [--process-pool] [-N DASK_CLUSTER_NAME]
                           [-c DASK_CLUSTER_CONFIG_FILE]
                           [--task-log-dir TASK_LOG_DIR]
                           [-t TASK_IDS [TASK_IDS ...]] [--keep-going]
                           [--fork-tasks]
Named Arguments
--verbose, -v, -V

Default: 0

--logfile

File to log output to. If not set, logs to console.

Repository/Resource
-R, --repository

URL to the genomic resources repository. If not specified the tool assumes a local file system repository and starts looking for .CONTENTS.json.gz file from the current working directory up to the root directory. If found the directory is assumed for root repository directory; otherwise error is reported.

--grr, --definition, -g

Path to an extra GRR definition file. This GRR will be loadedin a group alongside the local one.

--extra-args

comma separated list of key=value pairs arguments needed for connection to the specific repository protocol. Ex: if you want to connect to an S3 repository it is often neccessary to pass additional endpoint-url argument.

-r, --resource

Specifies the resource whose manifest we want to rebuild. If not specified the tool assumes local filesystem repository and starts looking for ‘genomic_resource.yaml’ file from current working directory up to the root directory. If found the directory is assumed for a resource directory; otherwise error is reported.

Force/Dry run
-n, --dry-run

report whether the manifest needs updating and write nothing: no manifest, and no recorded file state either. The run still reads and hashes whatever answering the question takes, it just keeps no receipt - so it leaves the repository byte-identical, and seeds nothing for the next run to reuse

Default: False

-f, --force

ignore resource state and rebuild manifest

Default: False

DVC params
--with-dvc

trust a ‘.dvc’ sidecar as the md5 sum and size of the file it describes, and never hash a DVC-managed file (default). A file GAIn has already hashed keeps its recorded md5 sum while its size and timestamp are unchanged; a file with no sidecar and no usable state is hashed

Default: True

-D, --without-dvc

verify mode: ignore every recorded state, compute from its content the md5 sum of every resource file that is on disk, and check it against the file’s ‘.dvc’ sidecar. The run reports every file that disagrees and exits non-zero, writing no manifest for the resources they belong to; fix the drift with ‘dvc add’ / ‘dvc commit’. A file that is NOT on disk still takes its md5 sum and size from its sidecar - there is no content to hash, and its manifest entry is never dropped

Default: True

Statistics
--region-size

Region size to use for splitting statistics calculation into tasks. The default is one region per contig; 0 does not split. A smaller size balances one large resource across -j workers: keep it at or below about genome / (4 x -j), e.g. –region-size 20000000 for hg38 at -j 32

Default: 3000000000

Task Graph Executor
-j, --jobs

Number of jobs to run in parallel. Defaults to the number of processors on the machine

--process-pool, --pp

Use a process pool executor with the specified number of processes instead of a dask distributed executor.

Default: False

-N, --dask-cluster-name, --dcn

The name of a named cluster in the dask config, under ‘gain_named_cluster.clusters’. The tasks run on it.

-c, --dccf, --dask-cluster-config-file

dask cluster config file

--task-log-dir

Path to directory where to store tasks’ logs

Execution Mode
-t, --task-ids
--keep-going

Whether or not to keep executing in case of an error

Default: False

--fork-tasks, --fork-task, --fork

Whether to fork a new worker process for each task

Default: False

repo-index

Publish the repository index (.CONTENTS files, search index, repository index pages) from the manifests already on disk

grr_manage repo-index [-h] [-R REPOSITORY] [--grr GRR]
                      [--extra-args EXTRA_ARGS] [--verbose]
                      [--logfile LOGFILE]
Named Arguments
--verbose, -v, -V

Default: 0

--logfile

File to log output to. If not set, logs to console.

Repository/Resource
-R, --repository

URL to the genomic resources repository. If not specified the tool assumes a local file system repository and starts looking for .CONTENTS.json.gz file from the current working directory up to the root directory. If found the directory is assumed for root repository directory; otherwise error is reported.

--grr, --definition, -g

Path to an extra GRR definition file. This GRR will be loadedin a group alongside the local one.

--extra-args

comma separated list of key=value pairs arguments needed for connection to the specific repository protocol. Ex: if you want to connect to an S3 repository it is often neccessary to pass additional endpoint-url argument.