DaskClusterFactory

Purpose

This is the recommended API for most users.

The DaskClusterFactory chooses the right cluster implementation at runtime. It checks SLURM_JOB_ID to detect whether batchlet is running inside an existing SLURM allocation.

If SLURM_JOB_ID is set: it creates DaskSlurmCluster and waits for all workers.
Otherwise: it creates a local dask cluster via DaskLocalCluster.

This makes application code portable across local development and SLURM-backed execution without changing cluster-creation logic.

Usage

from ska_sdp_batchlet import DaskClusterFactory

dask_params = {
    "workers_per_node": 2,
    "threads_per_worker": 2,
    "memory_per_worker": "4GB",
    "resources_per_worker": "GPU=1,CPU=2",
}

with DaskClusterFactory.get_cluster(dask_params) as cluster:
    client = cluster.get_client()
    # run dask code using client

API

class ska_sdp_batchlet.DaskClusterFactory[source]

A factory class which returns a new instance of a SpecCluster based dask cluster instance.

This factory current supports 2 kinds of clusters: 1. Slurm-based cluster using via DaskSlurmCluster 2. Local cluster using via DaskLocalCluster()

classmethod get_cluster(dask_cluster_params)[source]

Create and return a dask cluster based on the execution environment.

Parameters:: dask_cluster_params (dict) -- User provided dask cluster parameters. These are forwarded as is to the constructor of the dask cluster classes.
Return type:: SpecCluster
Returns:: Cluster instance which inherits SpecCluster.