DP3 step class to flag data based on Median Average Deviation (MAD) filtering. More...

#include <MadFlagger.h>

Inheritance diagram for dp3::steps::MadFlagger:

Public Member Functions
	MadFlagger (const common::ParameterSet &, const std::string &prefix)

	~MadFlagger () override

void	computeFactors (const std::vector< unsigned int > &timeEntries, unsigned int bl, int chan, int corr, int nchan, int ncorr, float &Z1, float &Z2, std::vector< float > &tempBuffer, common::NSTimer &moveTimer, common::NSTimer &medianTimer)

void	finish () override
	Finish the processing of this step and subsequent steps. More...

void	flag (unsigned int index, const std::vector< unsigned int > &timeEntries)

void	getExprValues (int maxNChan, int maxNTime)
	Get the values of the expressions for each baseline. More...

common::Fields	getProvidedFields () const override

common::Fields	getRequiredFields () const override
	Get the fields required by the current step. More...

bool	process (std::unique_ptr< base::DPBuffer >) override

void	show (std::ostream &) const override
	Show the step parameters. More...

void	showCounts (std::ostream &) const override
	Show the flagger counts. More...

void	showTimings (std::ostream &, double duration) const override
	Show the timings. More...

void	updateInfo (const base::DPInfo &) override

Public Member Functions inherited from dp3::steps::Step
	Step ()

virtual	~Step ()

virtual bool	accepts (MsType dt) const
	Boolean if this step can process this type of data. More...

const base::DPInfo &	getInfoIn () const
	Get access to the info of the input. More...

const base::DPInfo &	getInfoOut () const
	Get access to the info of the output. More...

const Step::ShPtr &	getNextStep () const
	Get the next step. More...

Step *	getPrevStep () const
	Get the previous step. More...

virtual MsType	outputs () const
	Return which datatype this step outputs. More...

virtual bool	process (std::unique_ptr< base::BdaBuffer >)

void	setInfo (const base::DPInfo &)

virtual void	setNextStep (Step::ShPtr nextStep)
	Set the next step. More...

void	setPrevStep (Step *prevStep)
	Set the previous step. More...

Protected Attributes
std::vector< xt::xtensor< float, 3 > >	itsAmplitudes
	amplitudes of the data More...

bool	itsApplyAutoCorr

std::vector< int >	itsAutoCorrIndex
	baseline index of autocorrelations More...

std::vector< double >	itsBLengths
	length of each baseline More...

std::vector< std::unique_ptr< base::DPBuffer > >	itsBuffers

common::NSTimer	itsComputeTimer
	move/median timer More...

std::vector< unsigned int >	itsFlagCorr

base::FlagCounter	itsFlagCounter

unsigned int	itsFreqWindow

std::vector< unsigned int >	itsFreqWindowArr
	freq window size per baseline More...

std::string	itsFreqWindowStr

double	itsMaxBLength
	maximum baseline length More...

double	itsMedianTime
	median timer (sum of all threads) More...

double	itsMinBLength
	minimum baseline length More...

double	itsMoveTime
	data move timer (sum all threads) More...

std::string	itsName

unsigned int	itsNrAutoCorr

unsigned int	itsNTimes

unsigned int	itsNTimesDone

float	itsThreshold

std::vector< float >	itsThresholdArr
	threshold per baseline More...

std::string	itsThresholdStr

common::NSTimer	itsTimer

unsigned int	itsTimeWindow

std::vector< unsigned int >	itsTimeWindowArr
	time window size per baseline More...

std::string	itsTimeWindowStr

Additional Inherited Members
Public Types inherited from dp3::steps::Step
enum class	MsType { kRegular , kBda }
	To check compatibility between steps before running. More...

typedef std::shared_ptr< Step >	ShPtr

Static Public Member Functions inherited from dp3::steps::Step
static void	SetThreadingIsInitialized ()

Static Public Attributes inherited from dp3::steps::Step
static constexpr dp3::common::Fields	kDataField

static constexpr dp3::common::Fields	kFlagsField

static constexpr dp3::common::Fields	kUvwField

static constexpr dp3::common::Fields	kWeightsField

Protected Member Functions inherited from dp3::steps::Step
virtual void	addToMS (const std::string &msName)

base::DPInfo &	GetWritableInfoOut ()

Detailed Description

DP3 step class to flag data based on Median Average Deviation (MAD) filtering.

This class is a Step class flagging data points based on the median of the absolute difference of the data and the median of the data. Both medians are taken in a time/frequency window around the data point. Only unflagged data points in the window are taken into account. The size of the window is given in the parset file.

The window around data points at the edges is formed by mirroring the data at the edge. For example, for channel 0 and a window size of 7 the data are mirrored, thus channels 3,2,1,0,1,2,3 will be used. For channel 1 the channels 2,1,0,1,2,3,4 will be used. The test program tMirror.cc can be used to check the correctness of the algorithm to determine the channels to use.

Taking the median is an O(N) operation, thus doing it for all data points is an O(N^2) operation. The test program tMedian.cc can be used to test the performance of the algorithms to determine the median. It shows that casacore's kthLargest outperforms STL's nth_element.
Shuffling the data around to be able to determine the medians is also an expensive operation, but takes less time than the medians themselves.

When a correlation is flagged, all correlations for that data point are flagged. It is possible to specify which correlations have to be taken into account when flagging. Using, say, only XX may boost performance with a factor 4, but miss points to be flagged. It is also possible to specify the order in which the correlations have to be tested.

It is possible to flag specific baselines only using a selection on baseline length.
Furthermore it is possible to only flag the autocorrelations and apply the resulting flags to the crosscorrelations, possibly selected on baseline length.

Constructor & Destructor Documentation

◆ MadFlagger()

dp3::steps::MadFlagger::MadFlagger	(	const common::ParameterSet &	,
		const std::string &	prefix
	)

Construct the object. Parameters are obtained from the parset using the given prefix.

◆ ~MadFlagger()

dp3::steps::MadFlagger::~MadFlagger ( )

override

Member Function Documentation

◆ computeFactors()

void dp3::steps::MadFlagger::computeFactors	(	const std::vector< unsigned int > &	timeEntries,
		unsigned int	bl,
		int	chan,
		int	corr,
		int	nchan,
		int	ncorr,
		float &	Z1,
		float &	Z2,
		std::vector< float > &	tempBuffer,
		common::NSTimer &	moveTimer,
		common::NSTimer &	medianTimer
	)

Compute the median factors for given baseline, channel, and correlation.

◆ finish()

void dp3::steps::MadFlagger::finish ( )

overridevirtual

Finish the processing of this step and subsequent steps.

Implements dp3::steps::Step.

◆ flag()

void dp3::steps::MadFlagger::flag	(	unsigned int	index,
		const std::vector< unsigned int > &	timeEntries
	)

Flag for the entry at the given index. Use the given time entries for the medians. Process the result in the next step.

◆ getExprValues()

void dp3::steps::MadFlagger::getExprValues	(	int	maxNChan,
		int	maxNTime
	)

Get the values of the expressions for each baseline.

◆ getProvidedFields()

common::Fields dp3::steps::MadFlagger::getProvidedFields ( ) const

inlineoverridevirtual

Get the fields provided (modified and/or created) by the current step. The returned fields thus should not include (required) fields that are forwarded without modifications.

Implements dp3::steps::Step.

◆ getRequiredFields()

common::Fields dp3::steps::MadFlagger::getRequiredFields ( ) const

inlineoverridevirtual

Get the fields required by the current step.

Implements dp3::steps::Step.

◆ process()

bool dp3::steps::MadFlagger::process ( std::unique_ptr< base::DPBuffer > )

overridevirtual

Process the data. When processed, it invokes the process function of the next step.

Reimplemented from dp3::steps::Step.

◆ show()

void dp3::steps::MadFlagger::show ( std::ostream & ) const

overridevirtual

Show the step parameters.

Implements dp3::steps::Step.

◆ showCounts()

void dp3::steps::MadFlagger::showCounts ( std::ostream & ) const

overridevirtual

Show the flagger counts.

Reimplemented from dp3::steps::Step.

◆ showTimings()

void dp3::steps::MadFlagger::showTimings	(	std::ostream &	,
		double	duration
	)		const

overridevirtual

Show the timings.

Reimplemented from dp3::steps::Step.

◆ updateInfo()

void dp3::steps::MadFlagger::updateInfo ( const base::DPInfo & )

overridevirtual

Update the general info. It is used to adjust the parms if needed.

Reimplemented from dp3::steps::Step.

Member Data Documentation

◆ itsAmplitudes

std::vector<xt::xtensor<float, 3> > dp3::steps::MadFlagger::itsAmplitudes

protected

amplitudes of the data

◆ itsApplyAutoCorr

bool dp3::steps::MadFlagger::itsApplyAutoCorr

protected

◆ itsAutoCorrIndex

std::vector<int> dp3::steps::MadFlagger::itsAutoCorrIndex

protected

baseline index of autocorrelations

◆ itsBLengths

std::vector<double> dp3::steps::MadFlagger::itsBLengths

protected

length of each baseline

◆ itsBuffers

std::vector<std::unique_ptr<base::DPBuffer> > dp3::steps::MadFlagger::itsBuffers

protected

◆ itsComputeTimer

common::NSTimer dp3::steps::MadFlagger::itsComputeTimer

protected

move/median timer

◆ itsFlagCorr

std::vector<unsigned int> dp3::steps::MadFlagger::itsFlagCorr

protected

◆ itsFlagCounter

base::FlagCounter dp3::steps::MadFlagger::itsFlagCounter

protected

◆ itsFreqWindow

unsigned int dp3::steps::MadFlagger::itsFreqWindow

protected

◆ itsFreqWindowArr

std::vector<unsigned int> dp3::steps::MadFlagger::itsFreqWindowArr

protected

freq window size per baseline

◆ itsFreqWindowStr

std::string dp3::steps::MadFlagger::itsFreqWindowStr

protected

◆ itsMaxBLength

double dp3::steps::MadFlagger::itsMaxBLength

protected

maximum baseline length

◆ itsMedianTime

double dp3::steps::MadFlagger::itsMedianTime

protected

median timer (sum of all threads)

◆ itsMinBLength

double dp3::steps::MadFlagger::itsMinBLength

protected

minimum baseline length

◆ itsMoveTime

double dp3::steps::MadFlagger::itsMoveTime

protected

data move timer (sum all threads)

◆ itsName

std::string dp3::steps::MadFlagger::itsName

protected

◆ itsNrAutoCorr

unsigned int dp3::steps::MadFlagger::itsNrAutoCorr

protected

◆ itsNTimes

unsigned int dp3::steps::MadFlagger::itsNTimes

protected

◆ itsNTimesDone

unsigned int dp3::steps::MadFlagger::itsNTimesDone

protected

◆ itsThreshold

float dp3::steps::MadFlagger::itsThreshold

protected

◆ itsThresholdArr

std::vector<float> dp3::steps::MadFlagger::itsThresholdArr

protected

threshold per baseline

◆ itsThresholdStr

std::string dp3::steps::MadFlagger::itsThresholdStr

protected

◆ itsTimer

common::NSTimer dp3::steps::MadFlagger::itsTimer

protected

◆ itsTimeWindow

unsigned int dp3::steps::MadFlagger::itsTimeWindow

protected

◆ itsTimeWindowArr

std::vector<unsigned int> dp3::steps::MadFlagger::itsTimeWindowArr

protected

time window size per baseline

◆ itsTimeWindowStr

std::string dp3::steps::MadFlagger::itsTimeWindowStr

protected

The documentation for this class was generated from the following file:

MadFlagger.h

Public Member Functions

Protected Attributes

Additional Inherited Members

Detailed Description

Constructor & Destructor Documentation

◆ MadFlagger()

◆ ~MadFlagger()

Member Function Documentation

◆ computeFactors()

◆ finish()

◆ flag()

◆ getExprValues()

◆ getProvidedFields()

◆ getRequiredFields()

◆ process()

◆ show()

◆ showCounts()

◆ showTimings()

◆ updateInfo()

Member Data Documentation

◆ itsAmplitudes

◆ itsApplyAutoCorr

◆ itsAutoCorrIndex

◆ itsBLengths

◆ itsBuffers

◆ itsComputeTimer

◆ itsFlagCorr

◆ itsFlagCounter

◆ itsFreqWindow

◆ itsFreqWindowArr

◆ itsFreqWindowStr

◆ itsMaxBLength

◆ itsMedianTime

◆ itsMinBLength

◆ itsMoveTime

◆ itsName

◆ itsNrAutoCorr

◆ itsNTimes

◆ itsNTimesDone

◆ itsThreshold

◆ itsThresholdArr

◆ itsThresholdStr

◆ itsTimer

◆ itsTimeWindow

◆ itsTimeWindowArr

◆ itsTimeWindowStr