gain.dask package

Submodules

gain.dask.config module

LEGACY_NAMED_CLUSTER_KEY = 'dae_named_cluster'

The old config key of the named clusters. Its clusters value is a list of cluster definitions, each with a name key.

LEGACY_NAMED_CLUSTER_REMOVAL_RELEASE = '2026.11.0'

The GAIn release that stops reading LEGACY_NAMED_CLUSTER_KEY. Named in the warning: a notice that does not say when it bites cannot be scheduled against.

NAMED_CLUSTER_KEY = 'gain_named_cluster'

The config key that holds the named clusters.

check_named_cluster_config() → None

Raise the refusal that reconfigure() found, if any.

Raises:

ValueError – When the named cluster config that dask loaded has the wrong shape.

clusters_shape_error(clusters: Any) → ValueError

Return the refusal of a clusters value that is not a mapping.

reconfigure() → None

Load default config into dask.

translate_legacy_named_cluster() → ValueError | None

Translate the old named cluster key in the dask config.

When the dask config holds dae_named_cluster, the function converts its clusters list to a mapping keyed by each name and merges the result over gain_named_cluster: the values of the old key win, and a key that the old key lacks keeps its value. An old cluster definition replaces the cluster definition of the same name whole. The default and distributed values of the old key come across too. The function removes the old key from the dask config and logs one warning. It does not change any file.

The function does not raise. When the old key, the new key or the old clusters value has the wrong shape, the function translates the other values and returns the first refusal. reconfigure() defers the refusal until GAIn reads the named clusters.

Returns:

The refusal of a malformed named cluster value, or None.

Return type:

ValueError | None

gain.dask.named_cluster module

setup_client(cluster_name: str | None = None, number_of_workers: int | None = None) → tuple[Client, dict[str, Any]]

Create a dask client for a named cluster.

Parameters:

• cluster_name – The name of a named cluster under gain_named_cluster.clusters. None selects the named cluster in gain_named_cluster.default.

• number_of_workers – The number of workers. None keeps the number in the cluster definition.

Returns:

The dask client and a copy of the cluster definition that built it.

Return type:

tuple[Client, dict[str, Any]]

Raises:

ValueError – When gain_named_cluster.clusters is not a mapping from names to cluster definitions, when a cluster definition holds a name key, or when no named cluster has the name cluster_name. Also when setup_client_from_config() refuses the cluster definition.

Notes

The function does not change the cluster definition in the dask config. Two calls for the same named cluster build two clients with the same settings.

setup_client_from_config(cluster_config: dict[str, Any], *, number_of_workers: int | None = None) → tuple[Client, dict[str, Any]]

Create a dask client from one cluster definition.

Parameters:

• cluster_config – A cluster definition: a mapping with the keys type, params and, optionally, number_of_workers or adapt_params. type is local, sge, slurm or manual. params holds the arguments of the dask cluster class. For manual, params holds the address of a running dask scheduler.

• number_of_workers – The number of workers. A number here wins over the number_of_workers of the cluster definition. None keeps the number of the cluster definition. When the cluster definition has no number_of_workers, the deprecated number_of_threads key gives the number. When no key gives a number, the cluster adapts by adapt_params, if the cluster definition has that key.

Returns:

The dask client and a copy of cluster_config. The copy is equal to cluster_config. It is not the same object.

Return type:

tuple[Client, dict[str, Any]]

Raises:

ValueError – When type is not a known cluster type, or when a manual cluster definition has no address under params.

Notes

For the local type, each worker runs one thread. When params holds threads_per_worker, the function logs a warning, also for the value 1, and uses 1. For sge and slurm, the jobqueue parameters cores and processes in params set the threads of a worker.

The function works on its own copy of cluster_config and does not change the mapping that it gets. Two calls with the same mapping build two clients with the same settings.

Module contents