Running PSS product tests with ProTest
ProTest is a command line application. It is essentially a wrapper around pytest, that provides everything pytest needs to run tests of the PSS (see ProTest - the PSS Product Testing Framework). If you’ve followed the steps in Installing ProTest then a protest executable should be in your system path. Test this running
protest -h
This should produce the following output.
usage: protest [-h] [-H] [-p PATH] [-i INCLUDE [INCLUDE ...]] [-e EXCLUDE [EXCLUDE ...]]
[--cache CACHE] [--outdir OUTDIR] [--keep] [--reduce]
Run PSS Product Tests
options:
-h, --help show this help message and exit
-H, --show_help Show detailed help on test options
-p PATH, --path PATH Path to cheetah build tree
-i INCLUDE [INCLUDE ...], --include INCLUDE [INCLUDE ...]
Include the following test types (def=product)
-e EXCLUDE [EXCLUDE ...], --exclude EXCLUDE [EXCLUDE ...]
Exclude the following test types
--cache CACHE Directory containing locally stored test vectors
--outdir OUTDIR Directory to store candidate data products
--keep Preserve the post-test data products (e.g, candidates, cheetah logs, configs, etc)
--reduce Store only header information from SPS candidate filterbanks
ProTest takes a number of optional arguments. The first (-p PATH) is the path to a cheetah build. This can be either of the following
a build tree (generated by running make)
a bin directory (which contains the cheetah executables in one place, generated by running make install).
ProTest will determine which of these options you are providing under the hood.
If -p is not provided, ProTest will assume you have the cheetah executable(s) in your system $PATH and will look there for the relevant one for each of its tests. If it cannot find cheetah in your $PATH, then an exception will be raised and ProTest will exit.
The -i INCLUDE option tells ProTest which group(s) of tests you wish to run according to how they are marked. ProTest tests PSS pipelines which rely on different processing resources that may not necessarily be available on all hardware that it runs on. For example, tests of PSS pipelines which required GPUs will not execute successfully on a machine which only has CPUs available. Furthermore, as a tester, you may only wish to test one pipeline type (e.g., Single Pulse Search pipelines but not Acceleration Search pipelines). You may wish to run tests only associated with SKA-LOW but not SKA-MID. For this reason we utilise the marker functionality provided by pytest. Each test is decorated with a set of markers which classify tests into groups, thereby allowing the tester to run only the subset of tests that are of interest to them. The user can specify as many markers as required to enable a specific combination of test types.
If -i is not provided, ProTest will execute all of the tests that it has (as specified by the product marker). Available markers can be found in pytest.ini or by running protest -H
The -e EXCLUDE option tells ProTest which group(s) of tests should not be executed. The user can specify as many markers as required to exclude a specific combination of test types.
--cache is a path to a local repository of PSS test vectors. This may be an existing cache, or a new cache into which new vectors will be pulled and stored. If the vectors required by the tests are not found in the cache directory provided on the command line, then they will be downloaded from the PSS test vector server. –cache is an optional argument. If it is not provided, ProTest will create and use a directory in the user’s $HOME area. ProTest will check there is sufficient space on the drive to download a test vector before doing so.
--outdir is the path to a directory into which data products which result from the execution of cheetah are written. This could include lists of candidate metadata, filterbank files, folded archives, etc. For each test, a directory with a randomly generated name is created under the directory specified by –outdir, and the results of that test are written to that directory. If –outdir is not provided, then ProTest will use “/tmp” to store cheetah data products.
--keep allows the output data products that are created by test runs (written to <outdir>) to be preserved. By default ProTest completely cleans up after itself leaving no files behind. However, for diagnostic puporses, one may wish to keep configuration files, candidates, etc.
--reduce is used to prevent candidate files (e.g., filterbanks) from occupying large amounts of disk space (i.e., in CI pipelines). If --reduce is enabled, SPS candidate filterbanks are removed, even if --keep is enabled, and are replaced by a json file (placed in <outdir>) that contains information from the header of each filterbank.
To demonstrate, let’s run all of the available SPS (Single Pulse Search) tests. We’ll use a build tree in this case, and we will leave –outdir and –cache empty.
protest -i sps -p <path/to/cheetah/build_dir>/cheetah/
For clarity the <build_dir> refers to the directory in which make was executed, whereafter a subdirectory called cheetah is generated. At the time of writing, this command will produce the following output
Running pytest -m sps -c /home/bshaw/.venvs/protest/lib/python3.8/site-packages/ska_pss_protest/pytest.ini --path=/raid/bshaw/cheetah_builds/release_basic_cheetah_panda/cheetah /home/bshaw/.venvs/protest/lib/python3.8/site-packages/ska_pss_protest
=============================================================================================== test session starts ===============================================================================================
platform linux -- Python 3.8.13, pytest-7.4.0, pluggy-1.2.0
rootdir: /home/bshaw/.venvs/protest/lib/python3.8/site-packages/ska_pss_protest
configfile: pytest.ini
plugins: repeat-0.9.1, mock-3.11.1, metadata-3.0.0, bdd-6.1.1, html-3.2.0
collected 3 items / 2 deselected / 1 selected
../../../../home/bshaw/.venvs/protest/lib/python3.8/site-packages/ska_pss_protest/test_sps_emulator.py::test_detecting_fake_single_pulses
-------------------------------------------------------------------------------------------------- live log call --------------------------------------------------------------------------------------------------
INFO root:requester.py:132 Cache location: /home/bshaw/.cache/SKA/test_vectors
INFO root:requester.py:195 SPS-MID_747e95f_0.2_0.0002_2950.0_0.0_Gaussian_50.0_123123123.fil in local cache
INFO root:_config.py:138 Located cheetah executable: /raid/bshaw/cheetah_builds/release_basic_cheetah_panda/cheetah/pipelines/search_pipeline/cheetah_pipeline
INFO root:pipeline.py:144 Command is: /raid/bshaw/cheetah_builds/release_basic_cheetah_panda/cheetah/pipelines/search_pipeline/cheetah_pipeline --config=/tmp/yrkajb0u -p SinglePulse -s sigproc
INFO root:pipeline.py:171 Return code is: 0
INFO root:candlist.py:158 Detected candidates found at: /tmp/tmprr0_aij7/2012_03_14_00:00:00.spccl
INFO root:candlist.py:173 Located 60 candidates
PASSED [100%]
======================================================================================== 1 passed, 2 deselected in 49.42s =========================================================================================