Stages and configurations

The descriptions of each stage are copied from the docstrings of stages. Each stage has parameters, which are defined in the YAML config file passed to the pipeline.

Instrumental Calibration Stages

load_data

This stage loads the visibility data from either (in order of preference):

1. An existing dataset stored as a zarr file inside the 'cache_directory'.

2. From input MSv2 measurement set. Here it will create an intemediate zarr file with chunks along frequency and use it as input to the pipeline. This zarr dataset will be stored in 'cache_directory' for later use.

Name	Type	Default	Description	Nullable	Allowed Values
nchannels_per_chunk	int	32	Number of frequency channels per chunk in the written zarr file. This is also the size of frequency chunk used across the pipeline.	False
ntimes_per_ms_chunk	int	5	Number of time dimension to include in each chunk while reading from measurement set. This also sets the number of times per chunk for zarr file.	False
cache_directory	str | NoneType	None	Cache directory containing previously stored visibility datasets as zarr files. The directory should contain a subdirectory with same name as the input ms file name, which internally contains the zarr and pickle files. If None, the input ms will be converted to zarr file, and this zarr file will be stored in a new 'cache' subdirectory under the provided output directory.	True
ack	bool	False	Ask casacore to acknowledge each table operation.	False
datacolumn		'DATA'	Measurement set data column name to read data from.	False	'DATA', 'CORRECTED_DATA', 'MODEL_DATA'
field_id	int	0	Field ID of the data in measurement set	False
data_desc_id	int	0	Data Description ID of the data in measurement set	False

predict_vis

Predict model visibilities using a local sky model.

Name	Type	Default	Description	Nullable	Allowed Values
use_everybeam	bool	True	Whether to use everybeam model.	False
normalise_at_beam_centre	bool	True	If true, before running calibration, multiply vis and model vis by the inverse of the beam response in the beam pointing direction.	False
element_response_model	str	'oskar_dipole_cos'	Type of element response model. Required if use_everybeam is True. Refer documentation for more details: https://everybeam.readthedocs.io/en/latest/tree/python/utils.html	False
eb_ms	str | NoneType	None	If everybeam is being used but input ms does not have all of the metadata required by everybeam, this parameter is used to specify a separate dataset to use when setting up the beam models.	True
gleamfile	str | NoneType	None	Specifies the location of gleam catalogue file gleamegc.dat	True
lsm_csv_path	str | NoneType	None	Specifies the location of CSV file containing the sky model. The CSV file should be in OSKAR CSV format.	True
export_sky_model	bool	True	Specifies whether to export the sky model to a CSV file.	False
fov	float	5.0	Specifies the width of the cone used when searching for components, in units of degrees.	False
flux_limit	float	1.0	Specifies the flux density limit used when searching for components, in units of Jy.	False
alpha0	float	-0.78	Nominal alpha value to use when fitted data are unspecified.	False

bandpass_initialisation

Initialises the gains for bandpass calibration

Name	Type	Default	Description	Nullable	Allowed Values
refant	int | str	0	Reference antenna	False
niter	int	200	Number of solver iterations.	False
tol	float	1e-06	Iteration stops when the fractional change in the gain solution is below this tolerance.	False
export_gaintable	bool	True	Export intermediate gain solutions.	False

bandpass_calibration

Performs Bandpass Calibration

Name	Type	Default	Description	Nullable	Allowed Values
run_solver_config	RunSolverConfig	RunSolverConfig(solver='jones_substitution', refant=0, niter=50, phase_only=False, tol=0.001, crosspol=False)	Run Solver parameters	False
visibility_filters	VisibilityFilterConfig	VisibilityFilterConfig(uvdist=None, exclude_baselines=None)	Visibility Filters which are used to flag the visibility data before calibration. These flags are not carry forwarded to the next stages.	False
plot_config	PlotConfig	PlotConfig(plot_table=True, fixed_axis=False)	Plot parameters	False
visibility_key		'vis'	Visibility data to be used for calibration.	False	'vis', 'corrected_vis'
export_gaintable	bool	True	Export intermediate gain solutions.	False

delay_calibration

Extract delays from bandpass solutions for plotting

Name	Type	Default	Description	Nullable	Allowed Values
plot_config	PlotConfig	PlotConfig(plot_table=True, fixed_axis=False)	Plot parameters	False
oversample	int	1	Oversample rate	False
export_gaintable	bool	True	Export intermediate gain solutions.	False

flag_gain

Performs flagging on gains and updates the weight.

Name	Type	Default	Description	Nullable	Allowed Values
plot_config	PlotFlagGainConfig	PlotFlagGainConfig(curve_fit_plot=True, gain_flag_plot=True)	Plot options	False
soltype		'amplitude'	Solution type. There is a potential edge case where cyclic phases my get flagged as outliers. eg -180 and 180	False	'phase', 'amplitude', 'amp-phase', 'real-imag'
order	int	3	Order of the function fitted during detrending.	False
apply_flag	bool	True	Weights are applied to the gains	False
skip_cross_pol	bool	True	Cross polarizations is skipped when flagging	False
max_ncycles	int	5	Max number of independent flagging cycles	False
n_sigma	float	12.0	Flag values greated than n_simga * sigma_hat. Where sigma_hat is 1.4826 * MeanAbsoluteDeviation.	False
n_sigma_rolling	float	0.0	Do a running rms and then flag those regions that have a rms higher than n_sigma_rolling*MAD(rmses).	False
window_size	int	11	Window size for running rms	False
normalize_gains	bool	False	Normailize the amplitude and phase before flagging.	False
export_gaintable	bool	True	Export intermediate gain solutions.	False

ionospheric_delay

Calculates and applies ionospheric delay corrections to visibility data.

This function uses an IonosphericSolver to model phase screens based on the difference between observed visibilities and model visibilities. It derives a gain table representing these phase corrections and applies it to the visibility data. The resulting gain table can be optionally exported to an H5parm file.

Name	Type	Default	Description	Nullable	Allowed Values
cluster_indexes	list[int] | NoneType	None	Array of integers assigning each antenna to a cluster. If None, all antennas are treated as a single cluster	True
block_diagonal	bool	True	If True, solve for all clusters simultaneously assuming a block-diagonal system. If False, solve for each cluster sequentially	False
niter	int	500	Number of solver iterations.	False
tol	float	1e-06	Iteration stops when the fractional change in the gain solution is below this tolerance.	False
zernike_limit	list[int] | NoneType	None	list of Zernike index limits: Generate all Zernikes with n + |m| <= zernike_limit[cluster_id]. If None, a default is used by the solver.	True
timeslice	float |	'full'	Defines time scale over which each gain solution is valid. This is used to define time axis of the GainTable. This parameter is interpreted as follows, float: this is a custom time interval in seconds. Input timestamps are grouped by intervals of this duration, and said groups are separately averaged to produce the output time axis.	False
plot_table	bool	True	Plot all station Phase vs Frequency	False
export_gaintable	bool	True	Export intermediate gain solutions.	False

generate_channel_rm

Estimate a Rotation Measure value for each station, re-predict model visibilities based on RM values, and run gaintable solver using input calibrator visibility and re-predicted model visibilities.

Name	Type	Default	Description	Nullable	Allowed Values
run_solver_config	RunSolverConfig	RunSolverConfig(solver='jones_substitution', refant=0, niter=50, phase_only=False, tol=0.001, crosspol=False)	Run solver parameters	False
plot_rm_config	PlotRMConfig	PlotRMConfig(plot_rm=True, station=0)	Plot parameters for rotational measures	False
oversample	int	5	Oversampling value used in the RM calculation. This determines the resolution of the phasor. Setting this value too high may result in high memory usage.	False
peak_threshold	float	0.5	Height of peak in the RM spectrum required for a rotation detection.	False
refine_fit	bool	True	Whether to refine RM spectrum peak locations with a nonlinear optimisation of station RM values.	False
visibility_key		'vis'	Visibility data to be used for calibration.	False	'vis', 'corrected_vis'
plot_table	bool	True	Plot the generated gain table	False
export_gaintable	bool	True	Export intermediate gain solutions.	False

smooth_gain_solution

Smooth the gain solution.

Name	Type	Default	Description	Nullable	Allowed Values
plot_config	PlotSmoothGainsConfig	PlotSmoothGainsConfig(plot_table=True, plot_path_prefix='smoothed-gain', plot_title='Smoothed Gain')	Plot parameters	False
window_size	int	1	Sliding window size.	False
mode		'median'	Mode of smoothing	False	'mean', 'median'
export_gaintable	bool	True	Export intermediate gain solutions.	False

export_visibilities

Export visibilities to MSv2 file. The visibilities will be exported to a new subdirectory "visibilities" under the pipeline's output directory. Optionally one can apply the gaintable to raw visibilities before exporting.

Name	Type	Default	Description	Nullable	Allowed Values
data_to_export		'all'	Select which visibilities to export. Options are: 1. vis: Input calibrator visibilities 2. modelvis: Model visibilities computed in INST pipeline 3. all: both vis and modelvis	False	'all', 'vis', 'modelvis'
apply_gaintable_to_vis	bool	True	Whether to apply gaintable (computed till this stage) to 'vis', before exporting	False

export_gain_table

Export gain table solutions to a file.

Name	Type	Default	Description	Nullable	Allowed Values
file_name	str	'gaintable'	Gain table file name without extension	False
export_format		'h5parm'	Export file format	False	'h5parm', 'hdf5'
export_metadata	bool	False	Export metadata into YAML file	False

Target Calibration Stages

target_load_data

This stage loads the target visibility data from either (in order of preference):

1. An existing dataset stored as a zarr file inside the 'cache_directory'.

2. From input MSv2 measurement set. Here it will create an intemediate zarr file with chunks along frequency and time, then use it as input to the pipeline. This zarr dataset will be stored in 'cache_directory' for later use.

Name	Type	Default	Description	Nullable	Allowed Values
nchannels_per_chunk	int	32	Number of frequency channels per chunk in the written zarr file.	False
ntimes_per_ms_chunk	int	5	Number of time slots to include in each chunk while reading from measurement set and writing in zarr file. This is also the size of time chunk used across the pipeline.	False
cache_directory	str | NoneType	None	Cache directory containing previously stored visibility datasets as zarr files. The directory should contain a subdirectory with same name as the input target ms file name, which internally contains the zarr and pickle files. If None, the input ms will be converted to zarr file, and this zarr file will be stored in a new 'cache' subdirectory under the provided output directory.	True
timeslice	float	120.0	Defines time scale over which each gain solution is valid in seconds. This is used to define time axis of the GainTable. Input timestamps are grouped by intervals of this duration, and said groups are separately averaged to produce the output time axis.	False
ack	bool	False	Ask casacore to acknowledge each table operation	False
datacolumn		'DATA'	MS data column to read visibility data from.	False	'DATA', 'CORRECTED_DATA', 'MODEL_DATA'
field_id	int	0	Field ID of the data in measurement set	False
data_desc_id	int	0	Data Description ID of the data in measurement set	False

predict_vis

Predict model visibilities using a local sky model.

Name	Type	Default	Description	Nullable	Allowed Values
use_everybeam	bool	True	Whether to use everybeam model.	False
normalise_at_beam_centre	bool	True	If true, before running calibration, multiply vis and model vis by the inverse of the beam response in the beam pointing direction.	False
element_response_model	str	'oskar_dipole_cos'	Type of element response model. Required if use_everybeam is True. Refer documentation for more details: https://everybeam.readthedocs.io/en/latest/tree/python/utils.html	False
eb_ms	str | NoneType	None	If everybeam is being used but input ms does not have all of the metadata required by everybeam, this parameter is used to specify a separate dataset to use when setting up the beam models.	True
gleamfile	str | NoneType	None	Specifies the location of gleam catalogue file gleamegc.dat	True
lsm_csv_path	str | NoneType	None	Specifies the location of CSV file containing the sky model. The CSV file should be in OSKAR CSV format.	True
export_sky_model	bool	True	Specifies whether to export the sky model to a CSV file.	False
fov	float	5.0	Specifies the width of the cone used when searching for components, in units of degrees.	False
flux_limit	float	1.0	Specifies the flux density limit used when searching for components, in units of Jy.	False
alpha0	float	-0.78	Nominal alpha value to use when fitted data are unspecified.	False

complex_gain_calibration

Performs Complex Gain Calibration

Name	Type	Default	Description	Nullable	Allowed Values
run_solver_config	TargetRunSolverConfig	TargetRunSolverConfig(refant=0, niter=200, tol=1e-06, crosspol=False)	Run solver parameters	False
plot_config	PlotConfig	PlotConfig(plot_table=True, fixed_axis=False)	Plot parameters	False
visibility_key		'vis'	Visibility data to be used for calibration.	False	'vis', 'corrected_vis'
export_gaintable	bool	True	Export intermediate gain solutions.	False

export_gain_table

Export gain table solutions to a file.

Name	Type	Default	Description	Nullable	Allowed Values
file_name	str	'gaintable'	Gain table file name without extension	False
export_format		'h5parm'	Export file format	False	'h5parm', 'hdf5'
export_metadata	bool	False	Export metadata into YAML file	False

Target Ionospheric Calibration Stages

target_load_data

This stage loads the target visibility data from either (in order of preference):

1. An existing dataset stored as a zarr file inside the 'cache_directory'.

2. From input MSv2 measurement set. Here it will create an intemediate zarr file with chunks along frequency and time, then use it as input to the pipeline. This zarr dataset will be stored in 'cache_directory' for later use.

Name	Type	Default	Description	Nullable	Allowed Values
nchannels_per_chunk	int	32	Number of frequency channels per chunk in the written zarr file.	False
ntimes_per_ms_chunk	int	5	Number of time slots to include in each chunk while reading from measurement set and writing in zarr file. This is also the size of time chunk used across the pipeline.	False
cache_directory	str | NoneType	None	Cache directory containing previously stored visibility datasets as zarr files. The directory should contain a subdirectory with same name as the input target ms file name, which internally contains the zarr and pickle files. If None, the input ms will be converted to zarr file, and this zarr file will be stored in a new 'cache' subdirectory under the provided output directory.	True
timeslice	float	120.0	Defines time scale over which each gain solution is valid in seconds. This is used to define time axis of the GainTable. Input timestamps are grouped by intervals of this duration, and said groups are separately averaged to produce the output time axis.	False
ack	bool	False	Ask casacore to acknowledge each table operation	False
datacolumn		'DATA'	MS data column to read visibility data from.	False	'DATA', 'CORRECTED_DATA', 'MODEL_DATA'
field_id	int	0	Field ID of the data in measurement set	False
data_desc_id	int	0	Data Description ID of the data in measurement set	False

predict_vis

Predict model visibilities using a local sky model.

Name	Type	Default	Description	Nullable	Allowed Values
use_everybeam	bool	True	Whether to use everybeam model.	False
normalise_at_beam_centre	bool	True	If true, before running calibration, multiply vis and model vis by the inverse of the beam response in the beam pointing direction.	False
element_response_model	str	'oskar_dipole_cos'	Type of element response model. Required if use_everybeam is True. Refer documentation for more details: https://everybeam.readthedocs.io/en/latest/tree/python/utils.html	False
eb_ms	str | NoneType	None	If everybeam is being used but input ms does not have all of the metadata required by everybeam, this parameter is used to specify a separate dataset to use when setting up the beam models.	True
gleamfile	str | NoneType	None	Specifies the location of gleam catalogue file gleamegc.dat	True
lsm_csv_path	str | NoneType	None	Specifies the location of CSV file containing the sky model. The CSV file should be in OSKAR CSV format.	True
export_sky_model	bool	True	Specifies whether to export the sky model to a CSV file.	False
fov	float	5.0	Specifies the width of the cone used when searching for components, in units of degrees.	False
flux_limit	float	1.0	Specifies the flux density limit used when searching for components, in units of Jy.	False
alpha0	float	-0.78	Nominal alpha value to use when fitted data are unspecified.	False

ionospheric_delay

Calculates and applies ionospheric delay corrections to visibility data.

This function uses an IonosphericSolver to model phase screens based on the difference between observed visibilities and model visibilities. It derives a gain table representing these phase corrections and applies it to the visibility data. The resulting gain table can be optionally exported to an H5parm file.

Name	Type	Default	Description	Nullable	Allowed Values
cluster_indexes	list[int] | NoneType	None	Array of integers assigning each antenna to a cluster. If None, all antennas are treated as a single cluster	True
block_diagonal	bool	True	If True, solve for all clusters simultaneously assuming a block-diagonal system. If False, solve for each cluster sequentially	False
niter	int	500	Number of solver iterations.	False
tol	float	1e-06	Iteration stops when the fractional change in the gain solution is below this tolerance.	False
zernike_limit	list[int] | NoneType	None	list of Zernike index limits: Generate all Zernikes with n + |m| <= zernike_limit[cluster_id]. If None, a default is used by the solver.	True
plot_table	bool	True	Plot all station Phase vs Frequency	False

export_gain_table

Export gain table solutions to a file.

Name	Type	Default	Description	Nullable	Allowed Values
file_name	str	'gaintable'	Gain table file name without extension	False
export_format		'h5parm'	Export file format	False	'h5parm', 'hdf5'
export_metadata	bool	False	Export metadata into YAML file	False