Buffer holding the data of a timeslot/band. More...

#include <DPBuffer.h>

Public Types
using	DataType = aocommon::xt::UTensor< std::complex< float >, 3 >

using	FlagsType = xt::xtensor< bool, 3 >

using	SolutionType = std::vector< std::vector< std::complex< double > >>
	For every channel, contains n_antennas x n_polarizations solutions. More...

using	UvwType = xt::xtensor< double, 2 >

using	WeightsType = xt::xtensor< float, 3 >

Public Member Functions
	DPBuffer (const DPBuffer &)=default

	DPBuffer (const DPBuffer &that, const common::Fields &fields)

	DPBuffer (double time=0.0, double exposure=0.0)
	Construct object with empty arrays. More...

	DPBuffer (DPBuffer &&)
	The move constructor moves all data without using reference semantics. More...

void	AddData (const std::string &name)

void	Copy (const DPBuffer &that, const common::Fields &fields, const bool extra_data=false)

void	CopyData (const DPBuffer &source, const std::string &source_name, const std::string &target_name)

DataType &	GetData (const std::string &name="")

const DataType &	GetData (const std::string &name="") const

std::vector< std::string >	GetDataNames () const

double	GetExposure () const

FlagsType &	GetFlags ()

const FlagsType &	GetFlags () const

const casacore::Vector< common::rownr_t > &	GetRowNumbers () const

const SolutionType &	GetSolution () const

double	GetTime () const

UvwType &	GetUvw ()

const UvwType &	GetUvw () const

WeightsType &	GetWeights ()

const WeightsType &	GetWeights () const

bool	HasData (const std::string &name="") const

void	MoveData (DPBuffer &source, const std::string &source_name, const std::string &target_name)

DPBuffer &	operator= (const DPBuffer &)

DPBuffer &	operator= (DPBuffer &&)
	Move assignment moves all data without using reference semantics. More...

void	RemoveData (const std::string &name="")

void	SetExposure (double exposure)
	Get or set the exposure. More...

void	SetRowNumbers (const casacore::Vector< common::rownr_t > &rownrs)

void	SetSolution (const SolutionType &solution)

void	SetTime (double time)
	Get or set the time. More...

DataType	TakeData ()

WeightsType	TakeWeights ()

Detailed Description

Buffer holding the data of a timeslot/band.

This class holds the data for one time slot in XTensor objects.

The following data can be kept in a DPBuffer object.

TIME	The time slot center of the current data (in MJD seconds).
ROWNRS	The row numbers of the current time slot. It can be empty when e.g. a time slot is inserted or if data are averaged.
DATA	The visibility data as [n_baselines,n_channels,n_correlations].
FLAG	The data flags as [n_baselines,n_channels,n_correlations] (True is bad). Note that the n_correlations axis is redundant because DP3 will always have the same flag for all correlations. The reason all correlations are there is because the MS expects them. TODO(AST-1373): Investigate using a single flag for all correlations.
WEIGHT	The data weights as [n_baselines,n_channels,n_correlations].
UVW	The UVW coordinates in meters as [n_baselines,3].

Each data member (DATA, FLAG, UVW, WEIGHTS) is filled in if any Step needs it (the information about the required fields per each Step can be read with the getRequiredFields() function). The first Step (MsReader) will read the requested fields from the MS into the DPBuffer. In that way, as little memory as needed is used. Note that e.g. the AOFlagger can use a lot of memory if a large time window is used.

Until early 2015, DP3 used the strategy of shallow data copies. I.e., a Step increased the data reference counter and did not make an actual copy. Only when data were changed, a new data array was made. Thus, MsReader allocated a new array when it read the data. However, it appeared this strategy lead to memory fragmentation and to sudden jumps in memory usage on Linux systems.
Therefore, the strategy was changed to having each Step preallocate its buffers and, at that time, making deep copies when moving data from one Step to the next one. It appeared that it not only improved memory usage, but also improved performance, possible due to far less mallocs. Since 2023, DP3 uses a new strategy where it stores DPBuffers in unique pointers, which allows moving the buffers from one Step to the next without making deep copies.

The Buffer/Step guidelines are as follows:

If a Step accumulates DPBuffers for later processing (e.g. AOFlagger), it can move the DPBuffers it receives to its internal list. After processing the buffers on that list, it can move them from the list and forward them to the next Step without making deep copies.
If a Step processes the data immediately (e.g. Averager), it can update the DPBuffer it receives and forward the updated buffer to the next Step.

Member Typedef Documentation

◆ DataType

using dp3::base::DPBuffer::DataType = aocommon::xt::UTensor<std::complex<float>, 3>

◆ FlagsType

using dp3::base::DPBuffer::FlagsType = xt::xtensor<bool, 3>

◆ SolutionType

using dp3::base::DPBuffer::SolutionType = std::vector<std::vector<std::complex<double> >>

For every channel, contains n_antennas x n_polarizations solutions.

◆ UvwType

using dp3::base::DPBuffer::UvwType = xt::xtensor<double, 2>

◆ WeightsType

using dp3::base::DPBuffer::WeightsType = xt::xtensor<float, 3>

Constructor & Destructor Documentation

◆ DPBuffer() [1/4]

dp3::base::DPBuffer::DPBuffer	(	double	time = `0.0`,
		double	exposure = `0.0`
	)

explicit

Construct object with empty arrays.

◆ DPBuffer() [2/4]

dp3::base::DPBuffer::DPBuffer ( const DPBuffer & )

default

The copy constructor copies all data from the source buffer. It copies row numbers using reference semantics.

◆ DPBuffer() [3/4]

dp3::base::DPBuffer::DPBuffer ( DPBuffer && )

The move constructor moves all data without using reference semantics.

◆ DPBuffer() [4/4]

dp3::base::DPBuffer::DPBuffer	(	const DPBuffer &	that,
		const common::Fields &	fields
	)

This constructor copies the given fields only, without using reference semantics. It copies row numbers using reference semantics. It does not copy extra data fields yet (TODO in AST-1241).

Member Function Documentation

◆ AddData()

void dp3::base::DPBuffer::AddData ( const std::string & name )

Adds an extra visibility buffer. The new visibility buffer gets the same shape as the default data buffer.

Parameters

name	Name for the new buffer. The name may not be empty. It may also not equal the name of an existing extra buffer.

◆ Copy()

void dp3::base::DPBuffer::Copy	(	const DPBuffer &	that,
		const common::Fields &	fields,
		const bool	extra_data = `false`
	)

Copy that to this. Copy row numbers using reference semantics. Copy all other members using value semantics. Do not copy extra data fields yet (TODO in AST-1241).

Parameters

fields Copy the given fields from that.

◆ CopyData()

void dp3::base::DPBuffer::CopyData	(	const DPBuffer &	source,
		const std::string &	source_name,
		const std::string &	target_name
	)

Copy a data buffer of another DPBuffer into an extra data buffer in the current DPBuffer. Overwrites the extra data buffer if it exists.

Parameters

source_name	Name of a data buffer in 'source'. If empty, uses the main data buffer in 'source'.
target_name	Name of the target data buffer. May not be empty: Copying to the main data buffer is not supported.

◆ GetData() [1/2]

DataType& dp3::base::DPBuffer::GetData ( const std::string & name = "" )

inline

◆ GetData() [2/2]

const DataType& dp3::base::DPBuffer::GetData ( const std::string & name = "" ) const

inline

Accesses the data (visibilities) in the DPBuffer.

Parameters

name	Data buffer name. An empty string indicates the main data buffer. A non-empty string indicates an extra data buffer.

Returns: An XTensor object with the data of the given name. The data has shape (n_baselines, n_channels, n_correlations).

◆ GetDataNames()

std::vector<std::string> dp3::base::DPBuffer::GetDataNames ( ) const

inline

Returns: The sorted names of all visibility buffers.

◆ GetExposure()

double dp3::base::DPBuffer::GetExposure ( ) const

inline

◆ GetFlags() [1/2]

FlagsType& dp3::base::DPBuffer::GetFlags ( )

inline

◆ GetFlags() [2/2]

const FlagsType& dp3::base::DPBuffer::GetFlags ( ) const

inline

Accesses the flags for the data (visibilities) in the DPBuffer.

Returns: An XTensor object with the flags. The object has shape (n_baselines, n_channels, n_correlations).

◆ GetRowNumbers()

const casacore::Vector<common::rownr_t>& dp3::base::DPBuffer::GetRowNumbers ( ) const

inline

◆ GetSolution()

const SolutionType& dp3::base::DPBuffer::GetSolution ( ) const

inline

◆ GetTime()

double dp3::base::DPBuffer::GetTime ( ) const

inline

◆ GetUvw() [1/2]

UvwType& dp3::base::DPBuffer::GetUvw ( )

inline

◆ GetUvw() [2/2]

const UvwType& dp3::base::DPBuffer::GetUvw ( ) const

inline

Accesses the UVW coordinates in the DPBuffer.

Returns: An XTensor object with the UVW coordinates. The object has shape (n_baselines, 3).

◆ GetWeights() [1/2]

WeightsType& dp3::base::DPBuffer::GetWeights ( )

inline

◆ GetWeights() [2/2]

const WeightsType& dp3::base::DPBuffer::GetWeights ( ) const

inline

Accesses weights for the data (visibilities) in the DPBuffer.

Returns: An XTensor object with the weights. The object has shape (n_baselines, n_channels, n_correlations).

◆ HasData()

bool dp3::base::DPBuffer::HasData ( const std::string & name = "" ) const

inline

Check if the DPBuffer has a data buffer for the given name.

Parameters

name	Name. An empty string indicates the main data buffer (always exists). A non-empty string indicates an extra data buffer.

Returns: If the requested data buffer exists. It can be empty, though.

◆ MoveData()

void dp3::base::DPBuffer::MoveData	(	DPBuffer &	source,
		const std::string &	source_name,
		const std::string &	target_name
	)

Move a data buffer from 'source' into an extra data buffer of the current DPBuffer. If the target buffer already exists, it is overwritten.

Parameters

source_name	Name of a data buffer in 'source'. If empty, uses the main data buffer in 'source'.
target_name	Name of the target data buffer. May not be empty: Moving to the main data buffer is not supported.

◆ operator=() [1/2]

DPBuffer& dp3::base::DPBuffer::operator= ( const DPBuffer & )

Copy assignment copies all data from the source buffer. It copies row numbers using reference semantics.

◆ operator=() [2/2]

DPBuffer& dp3::base::DPBuffer::operator= ( DPBuffer && )

Move assignment moves all data without using reference semantics.

◆ RemoveData()

void dp3::base::DPBuffer::RemoveData ( const std::string & name = "" )

Removes extra visibility buffers

Parameters

name	Name of the buffer to remove. If empty, removes all extra data buffers.

◆ SetExposure()

void dp3::base::DPBuffer::SetExposure ( double exposure )

inline

Get or set the exposure.

◆ SetRowNumbers()

void dp3::base::DPBuffer::SetRowNumbers ( const casacore::Vector< common::rownr_t > & rownrs )

inline

Get or set the row numbers used by the InputStep class. It can be empty (e.g. when MsReader inserted a dummy time slot).

◆ SetSolution()

void dp3::base::DPBuffer::SetSolution ( const SolutionType & solution )

inline

◆ SetTime()

void dp3::base::DPBuffer::SetTime ( double time )

inline

Get or set the time.

◆ TakeData()

DataType dp3::base::DPBuffer::TakeData ( )

inline

Returns the data and clears the storage in this object.

Some Steps need to add elements and need to resize the shape of the storage. Resizing is "destructive". This function allows callers to "steal" the storage before resizing.

◆ TakeWeights()

WeightsType dp3::base::DPBuffer::TakeWeights ( )

inline

Returns the weights and clears the storage in this object.

Some Steps need to add elements and need to resize the shape of the storage. Resizing is "destructive". This function allows callers to "steal" the storage before resizing.

The documentation for this class was generated from the following file:

DPBuffer.h

Public Types

Public Member Functions

Detailed Description

Member Typedef Documentation

◆ DataType

◆ FlagsType

◆ SolutionType

◆ UvwType

◆ WeightsType

Constructor & Destructor Documentation

◆ DPBuffer() [1/4]

◆ DPBuffer() [2/4]

◆ DPBuffer() [3/4]

◆ DPBuffer() [4/4]

Member Function Documentation

◆ AddData()

◆ Copy()

◆ CopyData()

◆ GetData() [1/2]

◆ GetData() [2/2]

◆ GetDataNames()

◆ GetExposure()

◆ GetFlags() [1/2]

◆ GetFlags() [2/2]

◆ GetRowNumbers()

◆ GetSolution()

◆ GetTime()

◆ GetUvw() [1/2]

◆ GetUvw() [2/2]

◆ GetWeights() [1/2]

◆ GetWeights() [2/2]

◆ HasData()

◆ MoveData()

◆ operator=() [1/2]

◆ operator=() [2/2]

◆ RemoveData()

◆ SetExposure()

◆ SetRowNumbers()

◆ SetSolution()

◆ SetTime()

◆ TakeData()

◆ TakeWeights()