Getting Started

Context

This repository contains the SDP benchmark tests for various levels of benchmarking. ReFrame is used as a framework to package these benchmarks to perform automated tests on various HPC systems. It is designed to easily compare results from different systems, system configurations and/or environments. There are three levels of benchmarks defined in the package namely,

• Level 0: Kernel benchmarks that characterise various components of the system like FLOPS performance, memory bandwidth, network performance, etc.

• Level 1: These are representative pieces of real workflows that can be used to characterise the workload of the pipelines. These include processing functions of radio astronomy pipelines like FFTs, gridding, prototype codes, etc.

• Level 2: These will be the entire pipelines or workflows of the radio-astronomy softwares like WSClean, RASCIL, ASKAP, etc.

Note

For level 1 and 2 benchmarks, runtime configurations should be chosen in such a way to model the intended computational workload. Whereas level 0 benchmarks can be used to characterise various existing HPC systems in terms of raw performance.

Some part of this work has been inspired from hpc-tests work from StackHPC who used ReFrame to build performance tests for HPC platforms.

Currently supported benchmarks

Level 0

• Sample numpy operations benchmark is included in the tests.

• Babel Stream

• GPUDirect RDMA (GDR) test.

• HPCG

• HPL

• IMB

• IOR

• NCCL tests

• STREAM

• numpy / cupy FFTs

• Part of HIPPO Team’s Funclib

Level 1

• CUDA NIFTY gridder

• Image Domain Gridder (IDG) Test

• Imaging IO Test

Level 2

• RASCIL

All the tests are defined in a portable fashion in reframe_<application>.py file in each application directory. Results are compared and plotted using Juypter notebooks, with a <application>.ipynb file in each application directory.

Software stack

The benchmark tests can be run using either platform provided packages or it is possible to deploy our own software stack in the user space using Spack. Moreover, ReFrame gives us a framework to test both platform provided and user deployed software stacks using the notion of partitions. There is no recommendation when it comes to which software stack to use and it depends on the final objective. For example, if we want to compare two different types of architectures, it is advisable to deploy the same software stack on both platforms and then compare the results. More details on how to use Spack and build packages are presented in Installing packages.