Access via HTTP Method

Status endpoint

Verify the API’s status by sending a GET request to the /status endpoint. The response will indicate the API’s operational state. This is typically the first check to make sure the API is running and available. For this section and below, the responses are example outputs, and what you get may vary based on your local time and configurations.

Request

GET /status

Example Response

{
    "api_running": true,
    "api_version": "0.17.0",
    "startup_time": "2026-05-21T08:48:00.626127+00:00",
    "available_data_stores": [
        "DLM",
        "SDP",
        "shared"
    ],
    "indexing": {
        "in_progress": false,
        "progress": {
            "files_processed": 0,
            "total_files": 4703,
            "files_added": 0,
            "files_updated": 0,
            "files_skipped": 4703,
            "indexing_step": "completed"
        },
        "coordination_state": {
            "volume_prefix": "dpd_d4c137fbd432",
            "indexed_by": "ska-dataproduct-api-59468d6f8b-zf4l7",
            "indexing_running": false,
            "started_at": "2026-05-21T08:48:00.505781+00:00",
            "last_heartbeat": "2026-05-21T09:32:58.912691+00:00",
            "completed_at": "2026-05-21T09:32:58.912691+00:00",
            "files_indexed": 4703
        }
    },
    "pv_interface_status": {
        "metadata_store_name": "shared",
        "pv_available": true,
        "number_of_data_products_on_pv": 601,
        "last_index_duration": "0:44:58.406910",
        "index_time_modified": "2026-05-21T09:32:58.900573+00:00",
        "indexing_timeout_seconds": 21600,
        "indexing_batch_size": 100,
        "indexing_parallel_workers": 8,
        "indexing_work_unit_depth": 3,
        "last_index_run": {
            "timed_out": false,
            "work_units": 4571,
            "files_ingested": 0
        }
    },
    "metadata_store_status": {
        "store_type": "Stateless PostgreSQL metadata store",
        "db_status": {
            "configured": true,
            "running": true
        },
        "last_metadata_update_time": "2026-05-21T08:49:57.522388+00:00",
        "number_of_dataproducts": 601,
        "dlm_interface_enabled": true,
        "dlm_interface_status": {
            "number_of_items": 12583,
            "dlm_data_item_accessible": true,
            "dlm_tables_accessible": true,
            "dlm_tables_error": null
        }
    },
    "search_store_status": {
        "metadata_search_store_in_use": "PostgreSQL Search Store"
    },
    "sdp_config_db": {
        "connected": true,
        "watcher_connected": true,
        "description": "SDP Configuration Database (etcd) for data product flow information"
    }
}

Key Status Fields:

  • indexing.in_progress: Whether indexing is currently running

  • indexing.progress.indexing_step: Current stage — fixed values: idle, starting, scanning_filesystem, completed, timeout_partial_completion, error; or dynamic values during active indexing: scanning_and_ingesting (N found, N ingested) and ingesting_remaining (N files)

  • indexing.progress.total_files / files_added / files_updated / files_skipped: Indexing run summary

  • indexing.coordination_state: Multi-instance coordination record — which pod holds the indexing lease, when it started, and how many files were indexed

  • pv_interface_status.number_of_data_products_on_pv: Data products in the active metadata store (PostgreSQL row count or in-memory store count)

  • pv_interface_status.last_index_duration: How long the last indexing run took (H:MM:SS.ffffff format)

  • pv_interface_status.last_index_run.timed_out: Whether the last run hit the timeout

  • metadata_store_status.number_of_dataproducts: Row count in the active metadata store (PostgreSQL table count or in-memory store count)

  • metadata_store_status.dlm_interface_status.number_of_items: Items registered in the DLM table (a separate count, unrelated to the PV product count)

  • search_store_status.metadata_search_store_in_use: Which search backend is active (PostgreSQL Search Store or In memory search store)

Note

If indexing.progress.indexing_step shows timeout_partial_completion or error, manually trigger reindexing using the /reindexdataproducts endpoint to ensure all data products are indexed.

Search endpoint

Use the search endpoint to query the data products. You can specify a time range and key-value pairs to filter the results. The response prioritizes products within the timeframe that best match your criteria.

Request

POST /dataproductsearch

Example Body

{
    "start_date": "2000-12-12",
    "end_date": "2032-12-12",
    "key_value_pairs": ["execution_block:eb-m005-20231031-12345"]
}

Example Response

[
    {
        "execution_block": "eb-m005-20231031-12345",
        "date_created": "2023-10-31",
        "dataproduct_file": "eb-m005-20231031-12345",
        "metadata_file": "eb-m005-20231031-12345/ska-data-product.yaml",
        "config.cmdline": "-dump /product/eb-m004-20191031-12345/ska-sdp/pb-m004-20191031-12345/vis.ms",
        ...
        "obscore.instrument_name": "SKA-LOW",
        "id": 6
    }
]

Re-index data products endpoint

Trigger a fresh index of the persistent volume via:

Request

GET /reindexdataproducts

Response

"Metadata re-index request has been added to the background tasks"

The endpoint returns immediately with HTTP 202. Indexing runs in the background; the API remains responsive throughout. Use the /status endpoint to monitor progress. New and modified files are processed; already-indexed files are skipped efficiently.

Returns HTTP 409 if a reindex is already in progress.

Note

Indexing does not automatically retry on failure or timeout. If indexing.progress.indexing_step shows timeout_partial_completion or error, trigger a manual reindex to ensure all data products are available.

See Data Product Indexing for more details on the indexing process and timeout behavior.

Download data product endpoint

Sending a post request to the download endpoint will return a stream response of the specified data product as a tar archive. The body of the post request must contain the execution block id or the UUID of the data product you want to download.

Note

Downloads are only available for data products stored on the persistent volume (PVC). Data products that originate solely from the DLM or SDP Configuration database, and have no corresponding file on the PVC, cannot be downloaded via this endpoint.

Request

POST /download

Example Body

{
    "execution_block": "eb-test-20200325-00001"
}

or

{
    "uid": "a0a2a10f-e382-31ba-0949-9a79204dfcad"
}

Response

A stream response of the specified data product as a tar archive

Note

A data product with an execution block id can contain ‘sub’ data products, that is defined by another metadata file. If the user requests to download the product with the execution_block, all the products of that execution block id will be downloaded.

Retrieve metadata of a data product endpoint

Sending a post request to the /dataproductmetadata endpoint will return a Response with the metadata of the data product in a JSON format. The body of the post request must contain the UUID of the data product.

For example, the POST request body:

Request

POST /dataproductmetadata

Example Body

{
    "uid": "6a11ddaa-6b45-6759-47e7-a5abd5105b0e"
}

Example Response

{
    "interface": "http://schema.skao.int/ska-data-product-meta/0.1",
    "execution_block": "eb-m005-20231031-12345",
    "context": {
        "observer": "AIV_person_1",
        "intent": "Experimental run as part of XYZ-123",
        "notes": "Running that signal from XX/YY/ZZ through again, things seem a bit flaky"
    },
    "config": {
        "processing_block": "pb-m004-20191031-12345",
        ...
    },
    "files": [
        {
            "crc": "2a890fbe",
            ...
        }
    ],
    "obscore": {
        "access_estsize": 1,
        "dataproduct_type": "MS",
        "calib_level": 0,
        ...
    },
    "date_created": "2023-10-31",
    "dataproduct_file": "tests/test_files/product/eb-m005-20231031-12345",
    "metadata_file": "tests/test_files/product/eb-m005-20231031-12345/ska-data-product.yaml",
    "uid": "6a11ddaa-6b45-6759-47e7-a5abd5105b0e"
}

Ingest new data product

Sending a POST request to the /ingestnewdataproduct endpoint will load and parse a file at the supplied filename, and add the data product to the metadata store.

Request

POST /ingestnewdataproduct

Example Body

{
    "execution_block": "eb-test-20200325-00001",
    "relativePathName": "product/eb-test-20200325-00001"
}

Example Response

{
    "status": "success",
    "message": "New data product received and search store index updated",
    "uid": "f0b91aa5-d54b-e11a-410e-3e4edca5346f"
}

Ingest new metadata endpoint

Sending a POST request to the /ingestnewmetadata endpoint will parse the supplied JSON data as data product metadata, and add the data product to the metadata store.

For example, the POST request body:

Request

POST /ingestnewmetadata

Example Body

{
    "interface": "http://schema.skao.int/ska-data-product-meta/0.1",
    "execution_block": "eb-test-20240806-99999",
    "context": {
        "observer": "REST ingest",
        "intent": "",
        "notes": ""
    },
    "config": {
        "processing_block": "",
        "processing_script": "",
        "image": "",
        "version": "",
        "commit": "",
        "cmdline": ""
    },
    "files": [],
    "obscore": {
        "access_estsize": 0,
        "access_format": "application/unknown",
        "access_url": "0",
        "calib_level": 0,
        "dataproduct_type": "MS",
        "facility_name": "SKA",
        "instrument_name": "SKA-LOW",
        "o_ucd": "stat.fourier",
        "obs_collection": "Unknown",
        "obs_id": "eb-test-20240806-99999",
        "obs_publisher_did": "",
        "pol_states": "XX/XY/YX/YY",
        "pol_xel": 0,
        "s_dec": 0,
        "s_ra": 0.0,
        "t_exptime": 5.0,
        "t_max": 57196.962848574476,
        "t_min": 57196.96279070411,
        "t_resolution": 0.9,
        "target_name": ""
    }
}

Example Response

{
    "status": "success",
    "message": "New data product metadata received and saved in the DPD datastore.",
    "uid": "1f8250d0-0e2f-2269-1d9a-ad465ae15d5c"
}

Annotation POST endpoint

Note

Annotation functionality is only available if the API is running with a PostgreSQL persistent metadata store.

Annotations are used to add notes to specific data products and are stored in the metadata store in a separate table.

Sending a POST request to the /annotation endpoint will parse the supplied JSON data as data product annotation, and add the annotation to the Postgres database. This method can be used to create a data annotation or update an existing data annotation. The method used depends on the existence of the annotation_id.

For example, the POST request body for a create request:

Request

POST /annotation

Example Body

{
    "data_product_uid": "1f8250d0-0e2f-2269-1d9a-ad465ae15d5c",
    "annotation_text": "Example annotation text message."
}

Response

{
    "status": "success",
    "message": "New Data Annotation received and successfully saved."
}

An example of a POST request body for an update request:

Request

POST /annotation

Example Body

{
    "annotation_text": "Example annotation text message.",
    "annotation_id": 23
}

Response

{
    "status": "success",
    "message": "Data Annotation received and updated successfully."
}

An example of a response when PostgreSQL is not available (HTTP 503):

Response

{
    "status": "Received but not processed",
    "message": "PostgresSQL is not available, cannot access data annotations."
}

Annotations GET endpoint

Note

Annotation functionality is only available if the API is running with a PostgreSQL persistent metadata store.

Sending a GET request to the /annotations endpoint will retrieve a list of the annotations linked to the specified data product UID. If PostgreSQL is not available, a status code of 202 will be received.

Request

GET /annotations/1f8250d0-0e2f-2269-1d9a-ad465ae15d5c

Example Response

[
    {
        "annotation_id": 21,
        "data_product_uid": "1f8250d0-0e2f-2269-1d9a-ad465ae15d5c",
        "annotation_text": "Example annotation text message.",
        "user_principal_name": "test.user@skao.int",
        "timestamp_created": "2024-11-13T14:32:00",
        "timestamp_modified": "2024-11-13T14:32:00"
    },
    {
        "annotation_id": 36,
        "data_product_uid": "1f8250d0-0e2f-2269-1d9a-ad465ae15d5c",
        "annotation_text": "Example annotation text message.",
        "user_principal_name": "test.user@skao.int",
        "timestamp_created": "2024-11-13T14:45:00",
        "timestamp_modified": "2024-11-13T14:45:00"
    }
]

An example of a response when PostgreSQL is not available (HTTP 202):

Response

{
    "status": "Received but not processed",
    "message": "PostgresSQL is not available, cannot access data annotations."
}