How to run pipelines using SLURM on AWS
=======================================

This guide provides general instructions for running pipelines on the AWS DP HPC cluster using
SLURM. These steps are applicable to most pipelines.

Related
-------

- :doc:`How to load Spack modules on AWS <aws-load-spack-modules>`
- :doc:`How to start an interactive compute node on AWS <aws-interactive-compute-node>`
- :doc:`How to monitor a pipeline job on AWS <aws-monitor-pipeline>`
- `Accessing the clusters (Confluence)
  <https://confluence.skatelescope.org/display/SE/Accessing+the+clusters>`_
- `How to run ICAL on AWS
  <https://developer.skao.int/projects/ska-sdp-ical/en/latest/running.html#starting-a-rapthor-run-interactively>`_
- `How to run CIMG on AWS
  <https://developer.skao.int/projects/ska-sdp-cimg/en/latest/usage/aws_slurm.html#running>`_

Prerequisites
-------------

- An account on the AWS DP HPC cluster
- The pipeline repository cloned to a directory on the AWS DP HPC cluster

Steps
-----

1. Log into the DP HPC headnode.
2. Change directory to the repository root folder:

       .. code-block:: bash

           cd ~/path/to/repo/<pipeline-repo>

3. Edit the SLURM script as needed.

       - Locate the SLURM script for your pipeline (for example,
         ``scripts/prod/aws-run-<pipeline>.sbatch``).
       - Adjust paths, job parameters, number of nodes, and other settings as required.

4. Submit the SLURM job:

       .. code-block:: bash

           sbatch path/to/pipeline/script.sbatch

5. To run on multiple nodes, override the SLURM directives when submitting the job. For example:

       .. code-block:: bash

           sbatch --nodes=3 --ntasks=3 --cpus-per-task=96 scripts/prod/aws-run-cimg.sbatch

6. Check job status:

       .. code-block:: bash

           squeue
           sacct

Finishing up
------------

Once the SLURM job has finished, check your pipeline's documentation for the expected output
location and log files, as these vary between pipelines.

If you need help locating job output or checking whether the run completed successfully, see
:doc:`How to monitor a pipeline job on AWS <aws-monitor-pipeline>`.

Verification
------------

Confirm that the job completed successfully in SLURM, then verify the expected data products and
logs using your pipeline's documentation.