AWS DP HPC cluster

This page provides an overview of the storage areas available on the AWS DP HPC cluster and their intended usage.

Storage areas

The AWS DP HPC cluster provides three types of storage:

• Home Directory (``~/``)

  • Path: /home/<username>

  • Space: total 1968 GB

  • Purpose: scripts, configuration files, small personal files

  • Note: limited space; do not store large datasets

• Shared Storage (``/shared/fsx1/``)

  • Space: total 28970 GB

  • Purpose: high-performance storage accessible by all compute nodes

  • Use for: pipeline inputs, temporary working data, outputs

  • Note: short-term storage; may be cleared during cluster maintenance

• S3 (``s3://<skao-sdp-testdata>/``)

  • Purpose: long-term cloud storage

  • Use for: test datasets and archiving results

  • Note: persists beyond cluster lifetime