CHANGELOG

0.4.0

  • [Update] Use new metadata structure for LSM metadata yaml file (MR122)

  • [Update] Dependencies, including (MR120, MR122):

    • ska-sdp-config 1.6.1

    • ska-sdp-datamodels 1.4.0

    • ska-sdp-dataproduct-metadata 1.3.0

    • alembic 1.18.4

  • [Changed] Local sky model endpoint updated. (MR118)

    • CSV downloads by adding format=csv as query parameter.

    • LSM table has pagination.

  • [Update] Catch issue when multiple versions are created for the same catalogue at the same time (MR117)

  • [Change] Update LocalSkyModelQuery to do (MR116):

    • Allow HTTP endpoint to return multiple catalogues

    • Do not allow Flow based search to use multiple catalogues

    • Make Flow based LSM be forced to query by catalogue_name

    • Update HTML page to be clear when no components match the query

    • Upgrade fastapi+uvicorn

  • [Update] Delete metadata if ingest fails, or if upload is rejected (MR115)

  • [New] new commands to cleanup catalogues (MR106)

  • [Update] Redirect from ‘/’ to ‘/docs’ Improve swagger example code. (MR112, MR114)

  • [BREAKING, Remove] Healpix related columns from schema and code (MR105)

  • [Remove] Unused dependencies from pyproject.toml (MR105)

  • [Deprecate] HTML upload page is deprecated (MR105)

  • [Update] Improve error reporting during CSV file validation (MR108)

  • [Fix] Allow only a single metadata file to be uploaded (MR107)

  • [Update] Allow for LSM queries based on any metadata field (MR103)

  • [Update] Process LSM flows with multiple sources requesting different LSM data (MR100, MR109, MR111)

  • [Change] Location of the metadata file no longer relies on searching for ska_sdm directory (MR98)

  • [Update] Add all metadata to Local Sky Model and fix bug with resource management (MR99)

  • [Breaking change, Update] Update datamodels to latest (MR96)

    • Move Catalogue level metadata from components to the catalogue

    • Remove redundant fields

    • Cleanup data files

    • Allow docs pages to work in a kubernetes cluster

  • [Update] Make the GSM aware of the Resource Management (MR95)

  • [Update] Added generic migration commands for developers (MR94)

  • [Update] Move LocalSkyModel class to ska-sdp-datamodels repository to provide CSV read functionality for pipelines (MR92)

  • [Update] Removed deprecation warning, added more tests (MR90)

  • [Changed] Catalogue version is now auto-assigned at commit time by incrementing the minor version per catalogue (e.g. 0.1.0 → 0.2.0), independently for each catalogue name. The version field has been removed from the upload metadata JSON — it is ignored if present. (MR89)

  • [Fix] Fix use of FOV, converting it to radius (MR88)

  • [Updated] Allow requests of sky components to include operators on any component field (MR87)

  • [Updated] Write all Spectral Index Vectors with a fixed length (MR85)

  • [Updated] Updated models to follow latest standard (MR84)

  • [Added] Endpoint for querying metadata table that supports generic queries and multiple operators (MR83)

0.3.0

  • [Updated] Documentation structure and content (MR81)

  • [Removed] Unused endpoints( (MR79)

  • [Removed] Code generation scripts (generate_models.py, db_config.py) in favor of runtime dynamic generation (MR69)

  • [Removed] Unused post_process function and to_json method from Source model (MR69)

  • [Removed] Unused /optimise-json API endpoint (MR69)

  • [Removed]Remove the dependencie on a running PostgreSQL instance for tests (MR59)

  • [Removed] Helm chart (MR58)

  • [BREAKING] Implemented catalogue-level versioning with metadata files. Upload endpoint now requires a metadata.json file containing catalogue version, name, description, reference frequency, and epoch. All components in an upload share a single semantic version from the metadata file.

  • [Added] Batch upload feature with staging table and review workflow for sky survey data: (MR74)

    • New sky_component_staging table for temporary upload storage with upload_id tracking

    • Browser-based upload interface at /upload endpoint with drag-and-drop support

    • Dataset-level semantic versioning (0.1.0 → 0.2.0)

    • Supports tracking with upload IDs, status queries, and automatic cleanup of temporary files

  • [Added] Script to add sample data to database (MR76)

  • [Added] Batch upload endpoint /upload-sky-survey-batch for atomic multi-file ingestion. Supports tracking with upload IDs, status queries, and automatic cleanup of temporary files. (MR61)

  • [Added] Schema-level data validation for uploaded components. Invalid components are logged with ingestion failing if any validation errors occur. (MR61)

  • [Added] Scripts to upgrade/downgrade from migrations into scripts in container (MR72)

  • [Added] Utilities to write a local sky model CSV file and associated metadata. (MR71, MR75, MR77)

  • [Added] etcd service to docker-compose setup and updated documentation (MR60)

  • [Added] Alembic migrations to ease schema management. (MR57)

  • [Added] Implemented _query_gsm_for_lsm function to query the Global Sky Model database and return components within a specified field of view as GlobalSkyModel objects. (MR55)

  • [Added] Watcher for creating local sky models from Flow entries (MR51)

  • [Added] Re-instated postgres backend storage starting from version 0.1.4. (MR49)

  • [Updated] /local_sky_model endpoint to take optional version (only semantic version) (MR82)

  • [Updated] Asynchronous batch upload processing. Uploads now run in background tasks, returning immediately with “uploading” status while ingestion proceeds asynchronously. This keeps the API responsive during large file uploads. (MR61)

  • [Updated] Dynamic database models for automatic synchronization with ska-sdp-datamodels. (MR69)

    Breaking Changes:

    • Removed separate Telescope, Band, WideBandData, and NarrowBandData tables

    • Database schema now consists of Source and GlobalSkyModelMetadata tables

    • Both models dynamically generate columns from their respective dataclasses at module import time

    • Field names match ska-sdp-datamodels exactly: ra, dec, i_pol, healpix_index

  • [Updated] Migrated from deprecated @app.on_event("startup") to modern FastAPI lifespan context manager for application startup and shutdown handling. (MR61)

  • [Updated] Logger ENV variable has changed behaviour, and remove links to Redis (MR58)

Note

The 0.2.0 release has been shelved and should not be used as a reference to main. The ‘new’ main is based on version 0.1.4. The development changes post 0.2.0 (under ‘Unreleased’) were not released and can also be disregarded.

Unreleased

  • Reorganized and updated the documentation (MR44)

  • Fix RTD by upgrading to poetry 2 (MR42)

  • Update documentation and Make commands for datasets (MR41)

0.2.0 (former)

  • Prepare deployment and release new version (MR40)

  • Update uvicorn command line (MR39)

  • Fix Deployment, and small fixes (MR38)

  • Update the Chart and documentation (MR37)

  • Initial Updates for restructure (MR36)

  • Update readme (MR35)

  • Only return default columns by default (MR34)

  • Added free form search to local sky model (MR32)

  • Added file base (polars) backend, sped up MWA (vizier) ingest and RACS (file based) ingest. (MR31)

  • Allow deployments from ci pipelines to techops cluster (MR30)

  • Reimplement precise cone search (MR29)

0.1.4

  • Bumped version to 0.1.4 (MR28)

  • Update docs to reflect changes to GSM (MR27)

  • Increase code coverage (MR26)

  • Use the ska-sdp-python image (MR25)

  • Modify schema to include spectral index, variability, curvature, polarisation (MR24)

  • Added flux minimum to query (MR23)

0.1.3

  • Changes to allow ingest of an arbitrary catalogue for testing. (MR22)

  • Improve Local Sky Model creation speed (MR21)

  • Make dev db available (MR20)

  • Quick fix for key error (MR19)

  • Speed up cone search (MR18)

  • Fixed up dockerfile, ensure DB connection is up and running before finishing app startup (MR17)

  • Make the ingest generic to be able to process both vizier and file based catalogue (MR16)

  • Implement Cone Search (MR15)

  • PG-Sphere extension removal (MR14)

  • Addition of some logs (MR13)

  • Added AOI model for DB based selects (MR12)

  • Add probes, update volumes, update docker compose (MR11)

  • Ingest GLEAM catalogue, updated models, added test endpoints (MR10)

  • Use sqlalchemy for Data models (MR7)

  • Add basic documentation (MR5)

  • Documentation structure updates (MR4)

  • Update dependencies to fix failing docs build (MR3)

0.1.1

  • Bump release (MR2)

0.1.0

  • Helm chart for Postgresql backend with pgsphere extension and a fastAPI (MR1)