DaskClusterFactory

Purpose

This is the recommended API for most users.

The DaskClusterFactory chooses the right cluster implementation at runtime. It checks SLURM_JOB_ID to detect whether batchlet is running inside an existing SLURM allocation.

• If SLURM_JOB_ID is set: it creates DaskSlurmCluster and waits for all workers.

• Otherwise: it creates a local dask cluster via DaskLocalCluster.

This makes application code portable across local development and SLURM-backed execution without changing cluster-creation logic.

Usage

from ska_sdp_batchlet.utils.dask_cluster import DaskClusterFactory

dask_params = {
    "workers_per_node": 2,
    "threads_per_worker": 2,
    "memory_per_worker": "4GB",
    "resources_per_worker": "GPU=1,CPU=2",
}

with DaskClusterFactory.get_cluster(dask_params) as cluster:
    client = cluster.get_client()
    # run dask code using client

API

class ska_sdp_batchlet.utils.dask_cluster.factory.DaskClusterFactory

A factory class which returns a new instance of a SpecCluster based dask cluster instance.

static get_cluster(dask_params)

Create and return a dask cluster based on the execution environment.

The selection logic is as follows:

1. If inside a slurm job allocation, return DaskSlurmCluster

2. Else, return a DaskLocalCluster.

Parameters:

dask_params (dict) -- User provided dask parameters. Given to batchlet as json input, in the key "dask_params".

Return type:

SpecCluster

Returns:

Cluster instance which inherits SpecCluster.