imaging_qa

imaging_qa is a command line app written using RASCIL. It uses the python package PyBDSF to find sources in an image and check with the original inputs. Currently it features the following:

• Reads FITS images.

• Finds sources above a certain threshold and outputs the catalogue (in CSV, FITS and skycomponents format). For multi-frequency images, the source detection can be performed on the central channel or average over all channels.

• Produces image statistics and diagnostic plots including: running mean plots of the residual, restored, background and sources and a histogram with fitted Gaussian and power spectrum of the residual are also plotted.

• Optional: Read in the sensitivity image and apply a primary beam correction to the fluxes.

• Optional: Estimate the spectral index by reading in frequency moment images (in FITS format) containing higher order Taylor terms.

• Optional: compares with input source catalogue : takes hdf5 and txt format. The source input should has columns of “RA(deg), Dec(deg), FluxI(Jy), FluxQ(Jy), FluxU(Jy), FluxV(Jy), Ref. Freq.(Hz), Spectral Index”.

• Optional: plot the comparison and error of positions and fluxes for input and output source catalogue.

Example:

The following runs the a data set from the RASCIL test:

#!/bin/bash
# Run this in the directory containing both the
# restored and residual fits files:
python $RASCIL/rascil/apps/imaging_qa_main.py \
--ingest_fitsname_restored test-imaging-pipeline-dask_continuum_imaging_restored.fits \
--ingest_fitsname_residual test-imaging-pipeline-dask_continuum_imaging_residual.fits

If a source check is required:

#!/bin/bash
# This example deals with the multi-frequency image
python $RASCIL/rascil/apps/imaging_qa_main.py \
--ingest_fitsname_restored test-imaging-pipeline-dask_continuum_imaging_restored_cube.fits \
--check_source True --plot_source True \
 --input_source_filename test-imaging-pipeline-dask_continuum_imaging_components.hdf

If primary beam correction is required:

#!/bin/bash
# This example deals with the multi-frequency image
python $RASCIL/rascil/apps/imaging_qa_main.py \
--ingest_fitsname_restored test-imaging-pipeline-dask_continuum_imaging_restored_cube.fits \
--check_source True --plot_source True --apply_primary True\
--ingest_fitsname_residual test-imaging-pipeline-dask_continuum_imaging_sensitivity.fits \
--input_source_filename test-imaging-pipeline-dask_continuum_imaging_components.hdf

Supplying arguments from a file:

You can also load arguments into the app from a file.

Example arguments file, called args.txt:

--ingest_fitsname_restored=test-imaging-pipeline-dask_continuum_imaging_restored.fits
--ingest_fitsname_residual=test-imaging-pipeline-dask_continuum_imaging_residual.fits
--check_source=True
--plot_source=True

Make sure each line contains one argument, there is an equal sign between arg and its value, and that there aren’t any trailing white spaces in the lines.

Then run the imaging_qa code as follows:

python imaging_qa_main.py @args.txt

Specifying the @ sign in front of the file name will let the code know that you want to ready the arguments from a file instead of directly from the command line.

What happens when the image files, the argument file, and the imaging_qa code are not all in the same directory? Let’s take the following directory structure as an example:

- rascil # this is the root directory of the RASCIL git repository
    - rascil
        - apps
            imaging_qa_main.py
        - my_data
            my_restored_file.fits
            my_residual_file.fits
        args.txt

With such a setup, the best way to run the imaging_qa code is from the top-level rascil directory (the git root directory). Your args.txt file will need to contain either the relative or absolute path to your FITS files. E.g.:

--ingest_fitsname_restored=rascil/my_data/test-imaging-pipeline-dask_continuum_imaging_restored.fits
--ingest_fitsname_residual=rascil/my_data/test-imaging-pipeline-dask_continuum_imaging_residual.fits
--check_source=True
--plot_source=True

And you need to provide similarily the relative or absolute path both to the args file and the code you are running:

python rascil/apps/imaging_qa_main.py @rascil/args.txt

Docker image

A Docker image is available at artefact.skao.int/rascil-imaging-qa which can be run with either Docker or Singularity. Instructions can be found at

• Dockerfiles for RASCIL

under Running the imaging_qa section.

Output plots

A list of plots are generated to analyze the image as well as comparing the input and output source catelogues.

Plots for restored image:

..._restored_plot.png  # Running mean of restored image
..._sources_plot.png  # Running mean of the sources
..._background_plot.png  # Running mean of background
..._restored_power_spectrum.png # Power spectrum of restored image

Plots for residual image:

..._residual_hist.png  # Histogram and Gaussian fit of residual image
..._residual_power_spectrum.png  # Power spectrum of residual image

Plots for position matching:

..._position_value.png  # RA, Dec values of input and output sources
..._position_error.png  # RA, Dec error (output-input)
..._position_distance.png # RA, Dec error with respect to distance from the centre

Plots for wide field accuracy:

..._position_quiver.png  # Quiver plot of the movement of source positions
..._gaussian_beam_position.png  # Gaussian fitted beam sizes for output sources

Plots for flux matching:

..._flux_value.png  # Values of output flux vs. input flux of sources
..._flux_ratio.png  # Ratio of flux out/flux in
..._flux_histogram.png  # Histogram of flux comparison
..._flux_position.png # Flux vs. RA and Dec of the sources

Plots for spectral index:

..._spec_index.png # Spectral index of input vs output fluxes over frequency.
..._spec_index_diagnostics_dist.png # Spectral index out/in vs. distance to centre
..._spec_index_diagnostics_flux.png # Spectral index out/in vs. input sources flux

Command line arguments

RASCIL continuum imaging checker

usage: imaging_qa_main [-h]
                       [--ingest_fitsname_restored INGEST_FITSNAME_RESTORED]
                       [--ingest_fitsname_residual INGEST_FITSNAME_RESIDUAL]
                       [--ingest_fitsname_sensitivity INGEST_FITSNAME_SENSITIVITY]
                       [--ingest_fitsname_moment INGEST_FITSNAME_MOMENT]
                       [--finder_beam_maj FINDER_BEAM_MAJ]
                       [--finder_beam_min FINDER_BEAM_MIN]
                       [--finder_beam_pos_angle FINDER_BEAM_POS_ANGLE]
                       [--finder_thresh_isl FINDER_THRESH_ISL]
                       [--finder_thresh_pix FINDER_THRESH_PIX]
                       [--finder_multichan_option FINDER_MULTICHAN_OPTION]
                       [--perform_diagnostics PERFORM_DIAGNOSTICS]
                       [--apply_primary APPLY_PRIMARY]
                       [--use_frequency_moment USE_FREQUENCY_MOMENT]
                       [--telescope_model TELESCOPE_MODEL]
                       [--check_source CHECK_SOURCE]
                       [--plot_source PLOT_SOURCE]
                       [--input_source_filename INPUT_SOURCE_FILENAME]
                       [--match_sep MATCH_SEP] [--flux_limit FLUX_LIMIT]
                       [--trim_image TRIM_IMAGE] [--trim_box TRIM_BOX]
                       [--quiet_bdsf QUIET_BDSF] [--source_file SOURCE_FILE]
                       [--rascil_source_file RASCIL_SOURCE_FILE]
                       [--logfile LOGFILE] [--savefits_rmsim SAVEFITS_RMSIM]
                       [--restart RESTART] [--use_dask USE_DASK]
                       [--dask_scheduler DASK_SCHEDULER]
                       [--dask_memory DASK_MEMORY]
                       [--dask_nworkers DASK_NWORKERS]
                       [--dask_nthreads DASK_NTHREADS]

Named Arguments

--ingest_fitsname_restored

FITS file of the restored image to be read

--ingest_fitsname_residual

FITS file of the residual image to be read

--ingest_fitsname_sensitivity

FITS file of the sensitivity image to be read

--ingest_fitsname_moment

FITS file of the frequency moment images to be read (Note: Use the prefix of the fits files, e.g. if the restored image is test_image_restored.fits here should input test_image)

--finder_beam_maj

Major axis of the restoring beam (degrees) (usually not needed, passed in restored image)

Default: 1.0

--finder_beam_min

Minor axis of the restoring beam (degrees) (usually not needed, passed in restored image)

Default: 1.0

--finder_beam_pos_angle

Positioning angle of the restoring beam (degrees) (usually not needed, passed in restored image)

Default: 0.0

--finder_thresh_isl

Threshold to determine the size of the islands used in BDSF (Blob Detector and Source Finder)

Default: 5.0

--finder_thresh_pix

Threshold to detect source (peak value) used in BDSF

Default: 10.0

--finder_multichan_option

For multi-channel images, what mode to perform source detection on (single or average)

Default: “single”

--perform_diagnostics

Whether to perform diagnostics of the images (restored and residual)

Default: “False”

--apply_primary

Whether to divide by primary beam after BDSF to correct source flux

Default: “False”

--use_frequency_moment

Whether to use frequency moment images after BDSF to correct spectral index

Default: “False”

--telescope_model

The telescope to generate primary beam correction

Default: “MID”

--check_source

Option to check with original input source catalogue

Default: “False”

--plot_source

Option to plot position and flux errors for source catalogue

Default: “False”

--input_source_filename

If use external source file, the file name of source file

--match_sep

Maximum separation in radians for the source matching

Default: 1e-05

--flux_limit

Minimum flux where comparison plots are generated

Default: 0.001

--trim_image

For spectral index calculation, do we trim the image to avoid the edge effects?

Default: “False”

--trim_box

If trim_image is true, proportion of the box that is trimmed (default is 3%)

Default: 0.03

--quiet_bdsf

If True, suppress bdsf.process_image() text output to screen. Output is still sent to the log file.

Default: “False”

--source_file

Name of output source file

--rascil_source_file

Name of output RASCIL components hdf file

--logfile

Name of output log file

--savefits_rmsim

This parameter is a Boolean (default is False). If True, save background rms image as a FITS file.

Default: “False”

--restart

If true, surpass BDSF when the output already exists. The checker will start from reading the BDSF csv file

Default: “False”

--use_dask

Default: “True”

--dask_scheduler

Externally defined Dask scheduler e.g. 127.0.0.1:8786 or ssh for SSHCluster or existing for current scheduler

--dask_memory

Memory per Dask worker (GB), e.g. 5GB (None means Dask will choose)

--dask_nworkers

Number of workers (None means Dask will choose)

--dask_nthreads

Number of threads in each Dask worker (None means Dask will choose)