Unpacking PCAP files containing SPS station data

The sps-data-unpacker tool can be used to unpack data from PCAP files containing SPS station data. It supports two modes of operation: the first mode dumps the SPEAD data stream, contained in the PCAP file, in a human-readable format to STDOUT; the second mode creates PNG images from the sample data contained in the SPEAD stream.

Tip

The command-line tools are installed automatically when using Poetry, see Using Poetry.

Usage

$ sps-data-unpacker --help
usage: sps-data-unpacker [-h] [--output-base OUTPUT_BASE] [--write-png] file

Unpacks a PCAP file containing a SPS station data SPEAD stream.

positional arguments:
  file                  Path to the PCAP file

options:
  -h, --help            show this help message and exit
  --output-base OUTPUT_BASE, -o OUTPUT_BASE
                        Base filename for output PNG files
  --write-png, -w       Write H-pol & V-pol real and imaginary sample
                        components as PNG files

Sample output

$ sps-data-unpacker sps-data.pcap
###[ SpeadPacket ]###
  \header    \
   |###[ SpeadHeader ]###
   |  magic     = 0x53
   |  version   = 0x4
   |  itemPointerWidth= 0x2
   |  heapAddrWidth= 0x6
   |  reserved  = 0x0
   |  numberOfItems= 0x8
  \items     \
   |###[ ItemPointer ]###
   |  itemAddressMode= Immediate
   |  itemIdentifier= 0x0001
   |  \heapCounterItem\
   |   |###[ HeapCounter ]###
   |   |  stationChannelId= 1
   |   |  packetCounter= 32
   |###[ ItemPointer ]###
   |  itemAddressMode= Immediate
   |  itemIdentifier= 0x0004
   |  \packetLengthItem\
   |   |###[ PacketLength ]###
   |   |  packetPayloadLength= 0
   |###[ ItemPointer ]###
   |  itemAddressMode= Immediate
   |  itemIdentifier= 0x1027
   |  \syncTimeItem\
   |   |###[ SyncTime ]###
   |   |  syncTimeReserved= 0x0
   |   |  unixEpochTime_s= 0
   |###[ ItemPointer ]###
   |  itemAddressMode= Immediate
   |  itemIdentifier= 0x1600
   |  \timestampItem\
   |   |###[ Timestamp ]###
   |   |  timestamp_ns= 7077888
   |###[ ItemPointer ]###
   |  itemAddressMode= Immediate
   |  itemIdentifier= 0x1011
   |  \centerFreqItem\
   |   |###[ CenterFreq ]###
   |   |  frequency_hz= 310156250
   |###[ ItemPointer ]###
   |  itemAddressMode= Immediate
   |  itemIdentifier= 0x3000
   |  \channelInfoItem\
   |   |###[ ChannelInfo ]###
   |   |  channelInfoReserved= 0x0
   |   |  beamId    = 1
   |   |  frequencyId= 397
   |###[ ItemPointer ]###
   |  itemAddressMode= Immediate
   |  itemIdentifier= 0x3001
   |  \antennaInfoItem\
   |   |###[ AntennaInfo ]###
   |   |  substationId= 1
   |   |  subarrayId= 1
   |   |  stationId = 1
   |   |  antennaInfoReserved= 1
   |###[ ItemPointer ]###
   |  itemAddressMode= Absolute
   |  itemIdentifier= 0x3300
   |  \sampleOffsetItem\
   |   |###[ SampleOffset ]###
   |   |  payloadOffset= 0
  \values    \
   |###[ Value ]###
   |  vPolReal  = 0
   |  vPolImag  = 0
   |  hPolReal  = 0
   |  hPolImag  = 0
   |###[ Value ]###
   |  vPolReal  = 0
   |  vPolImag  = 0
   |  hPolReal  = 0
   |  hPolImag  = 0
   |###[ Value ]###
   |  vPolReal  = 0
...

The output above shows the data values for two-and-a-bit samples of the first packet from a Version 1 packet stream. Each packet contains 2048 samples of data and each packet contains the data from a single station. The station, among other things, is identified by the AntennaInfo section.

Version 2 of the packet stream replaces the Centre Frequency item with a Station ID item; it also renames the logicalChannelId field of the HeapCounter item to stationChannelId

$ sps-data-unpacker -w sps-data.pcap

If the output base filename is not specified, the base file name of the input data is used as default.

Depending on the number of channels in the input data, and the number if integration periods the stream spans, determines the height dimension of the output PNG image – one image row for each station, for each integration period.

The number of image columns (the image width dimension) is fixed at 2048: one image pixel for each data sample.

Two output images are created, named <output-base>_re.png and <output-base>_im.png, respectively. The reason one cannot cannot specify the full output image name as argument is because there are always two output images created. The output images are colour images, and therefore each pixel is composed of three colour components, namely red, green, and blue. Only two of these colour bands in each image contains data: the _re image represents the real components of the complex sample values, while the _im image contains the imaginary components.

To make the images visually distinct, the “real” image uses the samples’ H-polarisation real component as the red channel value, and the V-polarisation real component for the blue channel. The “imaginary” image uses the H-polarisation value for the red colour band, and the V-polarisation value as the green colour band. This gives the “real” image a magenta hue, and the “imaginary” image a cyan hue. Note that the sample values have a value that ranges from -127 as the minimum to 127 as the maximum. In order map these values to the colour band values of 0 to 255, a bias value of 127 is added to each sample. The unused channel for each pixel, however, does not have this bias applied and is set to have a zero value.