alveo2gemini

This directory contains software that bridges the OpenCL PCI interface of Xilinx Alveo cards to the Gemini Network Protocol. It allows Alveo cards to be controlled in the same way as Gemini FPGA cards are controlled. The software is responsible for loading FPGA firmware into the card and then executing an openCL kernel, implemented in FPGA RTL, to read and write registers in the RTL design.

hbm2file

The directory contains source code for the hbm2file application which works with alveo2gemini to dump Alveo card HBM contents.

Dependencies

Install openCL sudo apt-get install opencl-headers

Build

To build the software, Xilinx Vitis software suite must first be installed and the following environment variables need to be defined to provide the path to Xilinx software

  • XILINX_VITIS (eg export XILINX_VITIS=/tools/Xilinx/Vitis/2020.1)

  • XILINX_VIVADO (eg export XILINX_VIVADO=/tools/Xilinx/Vivado/2020.1) need to setup XRT reference as well source /opt/xilinx/xrt/setup.sh

The software can be built by issuing the ‘make’ command

Usage

The alveo2gemini executable has several optional parameters:

  • -f fpgaXclbinFilename (xclbin file containing FPGA firmware to be run)

  • -k kernelName (name of the OpenCL kernel within the FPGA firmware)

  • -p serverPort (port number that Gemini Server should listen on)

  • -b argsBaseAddr (address of Vitis shared memory in ARGS address space)

  • -w (wait after launch for FPGA debugger to be attached)

  • -v (show build date and git version, then exit)

If no parameters are specified, the defaults are:

  • -f aug28.xclbin -k vitisAccelCore -p 30333 -b 0x8000

The hbm2file executable has three optional command line parameters:

  • -f outputFilename (name of file to which HBM contents are dumped)

  • -i hbm_index (zero-based index indicating which HBM to dump)

  • -v (show build date and git version, then exit)

If no parameters are specified, the defaults are:

  • -f hbm.dat -i 0

Internal Details

Main routine (file: main.cpp)

The main routine is the top-level of the alveo2gemini executable. The main loop that handles the gemini packets received from clients runs until a user kills it or an unexpected error occurs. Its pseudo-code outline is:

  1. Create an “Alveo_cl” object which initialises OpenCL, loading FPGA with its bitfile

  2. Create 1st UDP socket that will broadcast Gemini “event” packets to all-network-hosts, port 30001

  3. Construct a Gemini “publish” event payload that can be sent every 2 seconds by the socket

  4. Create 2nd UDP socket, bound to port 30333 by default, that Gemini clients will connect to

  5. While (user hasn’t sent ctrl-c to kill the program) do:

    1. wait up to 50msec for a packet to be received from a gemini client on the 2nd socket

    2. if a packet was received:

      • pass packet payload to the gemini-protocol-handler

      • send the gemini-protocol-handler response back to the user via the 2nd socket

    3. if more than 2 seconds have elapsed since last publish packet was sent

      • send a publish packet via the 1st socket

Gemini-protocol-handler (files: gemini.h, gemini.cpp, c++ class: Gemini)

The gemini-protocol-handler (gemini.cpp) processes gemini client packets in a way that mimics the processing done by gemini FPGA cards. The one key difference is that the FPGA implementation of the processing includes packet-retry mechanisms to deal with lost network packets. This handler does not retry and drops retries because there can be no lost packets over the PCI bus connection to the FPGA card.

The gemini-protocol-handler has a single “packet_in” function. The function calls the OpenCl object’s “reader” and “writer” methods to read/write to gemini registers in the FPGA. In pseudo-code, the operations that the packet_in method performs are:

  1. check that packet payload is actually gemini protocol, and drop if not

  2. if the packet is a Gemini “Connect” operation

    • create an “ACK” payload that’s returned to the caller

  3. If the packet is a Gemini “Read Inc” operation

    • invoke OpenCL to perform a register read operation on the Alveo card

    • put the read results into a Gemini packet that’s returned to the caller

  4. If the packet is a Gemini “Write Inc” operation

    • invoke OpenCL to perform a register write operation on the Alveo card

    • create an “ACK” payload that’s returned to the caller

  5. If the packet is any other kind of Gemini packet (unknown type)

    • drop the packet, no response to the caller

Alveo OpenCL interface (file: alveo_cl.h, alveo_cl.cpp, c++ class: Alveo_cl)

This class contains methods that call OpenCL to communicate with an Alveo card. It has three key methods:

  1. Initialisation:

    1. loads FPGA bitfile,

    2. creates gigabyte buffers in HBM in the FPGA and on the host

    3. creates Posix Shared memory used for dumping HBM contents for debug

  2. write:

    1. performs OpenCL PCI-bus transactions that write registers in the FPGA

  3. read:

    1. performs OpenCL PCI-bus transactions that read registers from the FPGA

    2. also handles dump requests for any of the HBM buffers

      1. performs OpenCL PCI bus transactions that copy HBM contents to host

      2. copies buffer data to shared memory, accessible by other applications

hbm_dump_sharedmem (files:hbm_dump_sharedmem.h, hbu_dump_sharedmem.cpp, c++ class: HBMDumpSharedMem)

This class is intended to be used by alveo2gemini to create a Posix shared memory block via which other programs can request dumping of HBM memory blocks. The alveo2gemini program uses the class as an owner of the shared memory, responsible for creating and cleaning up the memory when it exits. Other programs will use this class as a client that simply connects to the shared memory.

Key things to note are:

  • Instances of the class are created by the create() method which will return a null pointer if creation fails.

  • The “Shared_info” structure in the class header defines the layout of the data in the shared memory. It includes semaphores which are used to protect data from simultaneous access, and to indicate when requests are made or new data has been dumped.

  • It is only designed for a single client at a time.

The alveo2gemini owner of the shared memory will use two of the public methods to respond to HBM dump requests:

  • check_for_request()

  • wait_write_to_share()

Clients will use the remaining two public methods to make HBM dump requests and to read the data out

  • dump_request()

  • wait_read_from_share()