.. _vis_chunking:

Processing Chunks of Visibilities
=================================

The pipeline provides users the option to use chunks of the visibilities by
selecting a specified frequency range, splitting the full range into smaller
frequency chunks, and applying the static RFI mask. Any combination of these are
allowed. The order of operation in the pipeline is as follows:

- Select visibilities for some frequency range with the ``--start_freq`` and
  ``--end_freq`` command line arguments in MHz. Selecting visibilities by
  channel numbers from the command line is not supported, and we do not intend
  to provide it unless many users of this pipeline have the need for it.
- Split the visibilities into frequency chunks with the ``--num_chunks`` command
  line argument. Note that the first and last chunk in the band are always
  discarded. Therefore, the pipeline does not allow the use of ``--num_chunks``
  less or equal 2. Currently we do not provide the option to override
  discarding the first and last chunk in the band.
- The static RFI mask is then applied within each frequency chunk. If all
  channels in a frequency chunk are flagged, the chunk is discarded and not used
  further in the calibration of the pointing offsets.

Frequency Range Selection
-------------------------

The user is allowed to select visibilities by specifying the frequency range in
MHz, but the Measurement Set reader used in the pipeline only has access to
channel numbers. It infers the frequency of each visibility based on the channel
numbers. Therefore, we first determine the channel numbers corresponding to the
user supplied frequencies before passing this information to the Measurement Set
reader which extracts the visibilities in the selected channels. The pipeline
does not support selecting visibilities based on time ranges, and we do not
intend to add such functionality, unless many users have a need for it.

Splitting into Frequency Chunks
-------------------------------

The visibilities from a frequency selection or full band can be split into
multiple frequency chunks. As described above, the first and last chunks in the
band are discarded so ``--num_chunks``, if given, should be larger than 2.
Otherwise, any number of chunks is allowed if and only if the available
frequencies can be divided (either equally or unequally) by the requested number
of chunks. For example, a request to divide a 32-channel dataset into 64 chunks
will results in an error since it is impossible to do so. Hence the maximum
number of chunks in this case would be 32 and the minimum is 1.


Removing Bad Frequency Channels
--------------------------------

A detailed step-by-step procedure showing how to apply a static RFI mask to the
visibilities is described in the :ref:`static_rfi_mask` section. The mask can be
applied to the full-band, or to chunks of visibilities. When applied to the
latter, if all channels in a frequency chunk are flagged, that chunk is
discarded. The remaining chunks are then used to determine the pointing offsets
of interest.