SKA PST STAT Applications

ska_pst_stat_info

Prints the version of the ska-pst-stat library to stdout.

ska_pst_stat_core

The implementation of the STAT.CORE software component for the Pulsar Timing product. This application regularly (every 5 seconds) processes short (100ms) segments of the voltage data streams received from the Correlator Beam Former. Each segment of data is analysed and a range of statistics metrics are computed. Simple scalar metrics (mean, variance, etc) are provided to a TANGO device and a richer set of vector metrics (bandpasses, histograms, etc) are made available to signal displays. A small HDF5 data file (< 10 MB) is generated from each segment that is processed and is written to the configured output path of <path>/<eb_id>/<subsystem_id>/<scan_id>/monitoring_stats/.

Usage: ska_pst_stat_core

  -c port     port on which to accept control commands
  -d path     write output files to the path [default /tmp]
  -f config   ascii file containing observation configuration
  -h          print this help text
  -t timeout  wait for the specified number of seconds for exiting
  -v          verbose output

ska_pst_stat_file_proc

Computes data quality statistics for every sample in the specified pair of data and weights files, as output by the AA0.5 voltage recorder, producing a single HDF5 output file. By default, the amount of data processed is limited to the first 100 milliseconds in the file, this can be increased through the command line options.

Usage: ska_pst_stat_file_proc -d data -w weights

  -d data     name of data file
  -t length   maximum number of seconds to process [default 0.1]
  -w weights  name of weights file
  -h          print this help text
  -v          verbose output

The path of the output file is created according to the following rules:

The output filename has the same stem as the data filename with the extension replaced by h5; e.g. if the data filename is something.data then the output filename is something.h5

If the data file is in a parent folder, then the output file is written to the same parent folder with the last sub-folder replaced by stat (if necessary, the stat sub-folder is created); e.g.

if the data filename is parent/something.dada then the output filename is stat/something.h5

if the data filename is /absolute/path/to/something.data then the output filename is /absolute/path/stat/something.h5

plot_stat_file

A python script is provided as an example of how to examine the contents of the HDF5 file that is generated by ska_pst_stat_file_proc. This python script is found in python/src/ska_pst/stat.

usage: plot_stat_file.py [-h] [-c] [-o OUTPUT] stat_filename

positional arguments:
  stat_filename         PST Statistics File in HDF5 format.

optional arguments:
  -h, --help            show this help message and exit
  -c, --contour         plot contoured histogram instead of heat map
  -o OUTPUT, --output OUTPUT
                          save plot to specified output filename

To generate a stats file that can be used with this script, use the ska_pst_generate_file application produced in ska-pst-common and included in the ska-pst builder image.

# Launch an instance of the builder image
make local-dev-env

# Build the release
make local-cpp-build-release

# Generate data and weights files in /tmp/data and /tmp/weights
cd /tmp
testdir=/mnt/ska-pst/src/ska_pst/stat/tests/data
ska_pst_generate_file -d $testdir/data_header_LowAA0.5.txt -w $testdir/weights_header_LowAA0.5.txt -s GaussianNoise -T 0.1

# Generate the HDF5 statistics file
ska_pst_stat_file_proc -d data/2023-02-14-22:15:37_0000000000000000_000000.dada -w weights/2023-02-14-22:15:37_0000000000000000_000000.dada

# Plot the HDF5 statistics using a contour plot for the 2D histogram and output /tmp/stats.png
python /mnt/ska-pst/python/src/ska_pst/stat/plot_stat_file.py -c -o /tmp/stats.png stat/2023-02-14-22:15:37_0000000000000000_000000.h5