Calibration dataflow

The full implementation plan for real-time calibration is shown in the diagram below. For a given instance of RCAL—for example, per sub-array or station beam—processing is split between one or more RCAL processors (one per receiver pod, and therefore distributed across sub-bands) and an RCAL coordinator to combine all of the solutions, prepare them for application in the CBF tied-array beamformer, and send them to the data queues.

Note

In the current implementation, the processors and coordinator are still combined within a single receive processor. Distribution into one or more RCAL processors and an RCAL coordinator, as well as further integration with Leap Accelerate for direction-dependent corrections, will progress over the coming PIs.

_images/RCal-PI27-flow.drawio.svg

Pre-processing

A number of pre-processing options have been included using SDP Preprocessing functions.

  • RFI masks: Applying pre-defined channel-range masks has been included using Preprocessing function rfi_masks.apply_rfi_masks. At present no built-in channel ranges are used, however they can be added with RCAL option rfi_masks.

  • RFI flagging: Adaptive outlier flagging has been included using Preprocessing function flagger.rfi_flagger. This can be controlled with RCAL option autoflag, which by default is to true. More testing is needed.

  • Short baselines can also be flagged out using RCAL option uv_min. The default value for this parameter is 150 m.

  • Time and frequency averaging: Time and frequency averaging of received visibility datasets has been included using Preprocessing functions averaging.averaging_frequency and averaging.averaging_time. The receiver can be configured to average over a set number of samples, over the full time range, and/or to the Low coarse channel width. Averaging is controlled using RCAL options preproc_ave_time and preproc_ave_frequency. The default values for these parameters when processing Low data are close to 10 seconds in time and quarter of a 781.25 kHz coarse channel (195.3125 kHz) in frequency.

Sky models

The global sky model database service is under development. The framework has been added to the processor, however further updates and testing are required. Alternative sky models can be used by supplying the GLEAM EGC text file or a CSV file containing a sky component list (in the format used by OSKAR). See generate_skycomponents() for details.

For the local sky model, the list of sky components used to form the calibration field is reinitialised using the first dataset received after a start_scan call is made. Sky component selection is handled by cone search parameters fov (width of the search area in degrees, defaulting to 5) and flux_limit (minimum flux density cutoff in Jy, defaulting to 0.5).

Beam models

For Low, the Everybeam package is used to initialise and sample antenna beam models. These models include the station rigid rotations and the parallactic angle rotations, so a given Jones matrix will convert an (ra,dec)-aligned polarised sky signal to the expected instrumental (X,Y) signal, in the rotated station frame.

For Mid, all Jones matrices are at present left as 2x2 identity matrices. It is expected that the Everybeam package will also be used for these matrices.

Beam models are reinitialised after a start_scan call is made. This includes setting up telescope information, loading element patterns, etc. The beam response matrices are re-evaluated for every RCAL solution interval, at the centre of the interval.

Visibility models

Visibility models are generated by combining the predicted visibilities of each local sky model component, after multiplication with the beam model. Visibility prediction is compute intensive and this is expected to limit how many sky components can be included in the sky model in real-time operation. One way to reduce the overhead is to average the data in time and frequency, to reduce the prediction resolution to something like the output Jones matrix resolution. Due to the potential wide field nature of a calibration field, model visibilities are attenuated by bandwidth and time-average smearing factors to account for the decorrelation that will arise from such averages.

Bandpass calibration

Direction-independent bandpass calibration matrices are updated after processing each dataset that is received. This is field-based calibration against the visibility models described above. Bandpass matrices are only reset if the array (i.e. the number of stations and their names) or the frequency axis changes. Otherwise the matrices are retained between scans. For the calibration, the SKA SDP Python Processing Functions solve_gaintable function is used with solver=”gain_substitution” for initialisation and solver=”normal_equations” or “jones_substitution” for refinement. This will be improved as experience is gained with early array assembly datasets.

Ionospheric delays

The SKA SDP Python Processing Functions ionospheric delay solver will be used with the bandpass calibration sky model to estimate direction-independent delays for the field of view. The Leap Accelerate package is also being incorporated to generate direction-dependent delays in the direction of PST and VLBI beams. This is all under development.

Post-processing

A number of post-processing tasks are required before publishing the solutions.

  • Solutions that are missing or corrupted due to RFI or hardware issues are set to zero to indicate to CBF that they are flagged.

  • For Low, bandpass solutions are averaged to the required output frequencies, (integer multiples of 781.25 kHz). It is also possible to interpolate over flagged channels, instead of leaving them set to zero, however this is disabled until it is clear that it is the right option. It may be better to use the last bandpass solution for the given channel.

  • The calibration Jones matrix solutions are multiplied with beam model matrices. For PST beams, direction-dependent beam responses are computed for each beam’s specific phase direction, retrieved from visibility metadata. This enables accurate calibration for multiple PST beams pointing in different directions. For PSS and VLBI tied-array beams, further development is needed.

  • The calibration solution phases need to be modified based on any known ionospheric delays (to do). Until this is implemented, direction-independent solutions will be handled by the bandpass calibration.

  • The combined Jones matrix solutions are then inverted.

  • The inverted matrices can be scaled to have amplitudes of less than one, to simplify conversion to integer precision in CBF while maintaining a known conversion to Jansky units. Ideally a single, constant scale factor will be suitable for all frequencies, beam elevations and stations, but this remains to be seen during commissioning. The current RCalProcessor() options for controlling scaling are:

    • scale_gains: Whether or not to apply scaling.

    • scale_factor: A user-defined scale factor, overriding the options below.

    • low_gain_threshold: The fraction of the median bandpass gain amplitude below which to flag matrices. This is also used as a starting point for the scale factor, since its inverse will be the approximate maximum value in the inverse matrices.

    • threshold_headroom: Decrease the low_gain_threshold value to the nearest power of 2 before setting the threshold. This is an attempt to keep the scale factor somewhat constant during an observation, but neither of these options should be needed once typical values are known.

    • gain_headroom: Additional headroom to build into the scale factor, to allow for station beam variation as a function of pointing direction. Expressed as a fraction of maximum gain.