Developer Guide

This document complements the guidelines set out in the SKA telescope developer portal.

Clone the repository and its submodules:

git clone --recursive git@gitlab.com:ska-telescope/sdp/ska-sdp-global-sky-model.git

The repository is integrated with the ska-cicd-makefile submodule, which provides standardised make commands, for example, for linting and testing.

The GSM code is based on FastAPI and SQLAlchemy.

It connects to two database backends to provide its full functionality:

• PostgreSQL Database

  Stores the global sky model catalogue data in the following tables:

  • SkyComponent table: Contains all sky component properties

  • GlobalSkyModelMetadata table: Stores catalogue version information and reference frequency

  The schema is dynamically generated from the ska-sdp-datamodels package to ensure consistency with the canonical data model. See PostgreSQL Database.

• etcd

  A distributed key-value store used by the SKA SDP Configuration database. The application uses etcd to watch for flow requests (data processing workflows) (see SDP Processing).

  The request_responder.py module starts a background thread that watches etcd for flow entries requesting local sky models. When a flow is detected, it processes the request and writes the results to the specified location.

The FastAPI service provides REST endpoints for:

• Querying the Global Sky Model database (see Viewing LSM data in a browser)

• Uploding new catalogue data (see API Endpoints)

• Monitoring the upload process (see API Endpoints)

Running the application tests

The API is tested using the pytest framework alongside FastAPI’s TestClient. The tests can be run with:

$ make python-tests

Test database setup

The tests use an in-memory SQLite database instead of PostgreSQL to avoid the need for a running database instance during testing.

Database Mocking Strategy

The test suite implements several key mocking strategies:

1. In-memory SQLite database: Tests use sqlite:///:memory: instead of PostgreSQL, with StaticPool to maintain a single connection across tests.

2. Q3C function mocking: PostgreSQL’s Q3C extension for spherical coordinate searches is mocked for SQLite using a simple box check approximation:

   @event.listens_for(engine, "connect")
   def register_q3c_mock(dbapi_conn, connection_record):
       def q3c_radial_query_mock(ra1, dec1, ra2, dec2, radius):
           # Simple box check - not accurate but sufficient for testing
           ra_diff = abs(ra1 - ra2)
           dec_diff = abs(dec1 - dec2)
           return 1 if (ra_diff <= radius and dec_diff <= radius) else 0
       dbapi_conn.create_function("q3c_radial_query", 5, q3c_radial_query_mock)
   

3. JSONB compatibility: PostgreSQL’s JSONB type is replaced with JSON for SQLite compatibility:

   @event.listens_for(Base.metadata, "before_create")
   def replace_jsonb_sqlite(target, connection, **kw):
       if connection.dialect.name == "sqlite":
           for table in target.tables.values():
               table.schema = None
               for column in table.columns:
                   if isinstance(column.type, JSONB):
                       column.type = JSON()
   

4. Startup function mocking: Database connection checks and background threads are mocked to prevent actual connections during tests:

   with patch("ska_sdp_global_sky_model.api.app.main.wait_for_db"), \\
        patch("ska_sdp_global_sky_model.api.app.main.start_thread"), \\
        patch("ska_sdp_global_sky_model.api.app.main.q3c_index"), \\
        patch("ska_sdp_global_sky_model.api.app.main.engine", engine):
       # Test code here