.. _qa:

MCCS Calibration Quality Assessment Metrics
===========================================

This package is intended to generate data for the calibration database linked
to specific observing conditions, and as such should produce metrics that
indicate both that the observing conditions are as expected and that any
deviations from previous or similar calibration data are within expectations
given the specifics of the sky model. The observing conditions can in general
be summarised by simple statistics related to common issues: bad antennas, RFI
and the ionosphere. A further complication here is the sky model, with galactic
components (and noise) that vary with baseline and sidereal time and a solar
component that can vary more rapidly with time, frequency and polarisation.
However when metrics indicate that these issues are all benign, calibration
errors should be limited to signal to noise levels and repeatable model errors.

Observation Quality
-------------------

In the course of generating calibration solutions, various measurements are
made that are useful indicators of the general state of the data. These can be
just as useful as goodness of fit metrics in determining whether or not a set
of solutions should be discarded. Measurements that may indicate that the data
are degraded include the following.

  * A change in the number of flagged antennas may indicate a problem somewhere
    in the system. At present the only check is for antennas with low gain
    amplitudes, but this could be replaced with more sophisticated outlier
    detection.
  * Antenna flags can also indicate the presence of RFI. Baseline-based
    flaggers can offer more information, but have not been added at this point.
  * Variations in antenna phases may indicate an active ionosphere. While the
    solvers only have access to a single frequency and cannot check for the
    wavelength dependence, linear trends across the station should be
    informative. Of course, the full array will have a lot of extra information
    on the state of the ionosphere. Phase shifts in time may also indicate
    hardware malfunctions missed during amplitude flagging.
  * Specifics about the sky model may be useful when considering output noise
    levels: the overall brightness and baseline distribution, the elevation of
    the sun and the galactic centre, for example. Also, if the brightness of
    the sun needs to be adjusted by a large amount it may indicate solar
    activity.
  * Variations in time may also be useful if multiple time steps are available.
    Changes in antenna phase, for instance, may indicate and active ionosphere
    or malfunctioning component.

Calibration Quality
-------------------

Useful calibration information that could be returned include:

* General solver information: residual levels (and perhaps some higher order
  statistics), correlation coefficients, number of iterations, etc.
* Compare with previous solutions? Need to be careful if the sky model is
  significantly different.
* Graphical output? Residual and solution plots, histograms, etc.

Output Metrics
--------------

This package has not yet been applied to many datasets and it is not clear
which are the most useful metrics. So at this point the calibration function
simply returns some indicators of the observation quality along with the model
and calibrated data for further inspection. These :class:`xarray.Dataset`
objects contain a lot of information with which to generate statistics and
plots. See
:py:func:`~ska_low_mccs_calibration.calibration.calibrate_mccs_visibility` for
an up-to-date list of what the function returns.