Data Product Dashboard
This page describes how to customize the integration between PST and the Data Product Dashboard (DPD).
PST (Internal) DPD
Deployment
The PST test-parent chart contains the default configuration necessary for deploying the Data Product Dashboard (DPD) helm chart
and is deployed as follows.
make k8s-install-chart
The DPD contains the applications dataproduct dashboard and dataproduct api.
When developing locally (without a local DNS server), port-forwarding to both the dashboard and the api is established by running the following commands in two separate terminals. Note that these commands do not exit. Also note that, when using port-forwarding, the dashboard will not be able to interact with the API because it expects the fully-qualified domain name of the bare-metal host paired with the URI of the API application.
In the first terminal, run
# NOTE: this command does not exit
# Terminal 1.
make dpd-forward-dashboard
And in the second terminal, run
# NOTE: this command does not exit
# Terminal 2.
make dpd-forward-api
The default ports are 8000 for the api pod and 8100 for the dashboard pod.
To override the them, additional steps are required.
The following example overrides the values to 30000 and 31000.
Update the following configuration in the helm values file:
ska-sdp-dataproduct-dashboard:
enabled: true
urls:
override: true
dashboardurl: "http://localhost:31000"
apiUrl: "http://localhost:30000"
Append overrides in make command.
# Install test parent helm chart
make k8s-install-chart
# Execute the following in different terminals
# NOTE: this command does not exit
# Terminal 1.
make dpd-forward-dashboard DPD_LOCAL_DASHBOARD_PORT=31000
# NOTE: this command does not exit
# Terminal 2.
make dpd-forward-api DPD_LOCAL_API_PORT=30000
Data Transfer
Sample data can be obtained from https://confluence.skatelescope.org/download/attachments/218404511/eb-m001-20230921-245.tar.gz?version=1&modificationDate=1692321019065&api=v2. NOTE: This should be replaced by a sample data generator
Download and extract data
tar -xvzf eb-m001-20230921-245.tar.gz
Create product directory in api pod and copy the file
# Create product sub directory
kubectl -n pst exec deployment/ska-sdp-dataproduct-api -- bash -c 'mkdir -p /mnt/data/product'
# Obtain api pod name
kubectl -n $KUBE_NAMESPACE get pods
# Copy file
kubectl -n $KUBE_NAMESPACE cp eb-m001-20230921-245 $API_POD_NAME:/mnt/data/product
Reindex metadata via api and confirm metadata list
# Reindex metadata
curl -X 'GET' \
'http://localhost:8000/reindexdataproducts' \
-H 'accept: application/json'
# Get metadata list
curl -X 'GET' \
'http://localhost:8000/dataproductlist' \
-H 'accept: */*'
SDP (External) DPD
In production, the data product dashboard is expected to be deployed at its own kubernetes namespace (integration-ska-sdp-dataproduct-dashboard).
This requires configuration change with respect to the PST test-parent helm chart
Prerequisites
shared persistent volume must be created.
# Investigate integration-ska-sdp-dataproduct-dashboard k8s namespace
kubectl -n integration-ska-sdp-dataproduct-dashboard get pods,pvc
# On the scenario where the persistent volume claim `shared-mnl` does not exist in the integration-ska-sdp-dataproduct-dashboard k8s namespace
# Execute the below command. This simulates the creation of the shared storage by an external entity
kubectl apply -f - <<EOF
# Source: ska-pst-test-parent/templates/pvc-shared-storage.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: shared-mnl
namespace: integration-ska-sdp-dataproduct-dashboard
labels:
helm.sh/chart: ska-pst-test-parent-0.8.0
app.pst/component: send
app: ska-pst-send
app.kubernetes.io/name: ska-pst-test-parent
app.kubernetes.io/instance: test
app.kubernetes.io/version: "0.8.0"
app.kubernetes.io/managed-by: Helm
spec:
storageClassName: nfss1
accessModes:
- ReadWriteMany
volumeMode: Filesystem
resources:
requests:
storage: 5Gi
EOF
SDP dataproduct dashboard chart must be deployed. Dashboard deployment details can be obtained by investigating the following jobs in the dataproduct dashboard pipeline https://gitlab.com/ska-telescope/sdp/ska-sdp-dataproduct-dashboard/-/tree/master?ref_type=heads
deploy-integration-psi-low
info-integration-psi-low
redeploy-integration-psi-low
stop-integration-psi-low
clone the persistent volume claim used by the data product dashboard in the
integration-ska-sdp-dataproduct-dashboardk8s namespace into thepstk8s namespace.
# Save the sdp persistentvolumeclaim runtime configuration and modify as needed
kubectl -n integration-ska-sdp-dataproduct-dashboard get pvc shared-mnl -o yaml > pst-pvc-shared-mnl.yaml
# Update the following fields
# metadata:
# annotations:
# kubectl.kubernetes.io/last-applied-configuration - DELETE
# meta.helm.sh/release-name: test - ADD these lines. They're used by helm as reserved fields.
# meta.helm.sh/release-namespace: pst - ADD these lines. They're used by helm as reserved fields.
# namespace - CHANGE this to pst
# creationTimestamp - DELETE this field
# labels - this should match the common metadata labels rendered in ska-pst-core below as an example
# app: ska-pst-send
# app.kubernetes.io/instance: test
# app.kubernetes.io/managed-by: Helm
# app.kubernetes.io/name: ska-pst-test-parent
# app.kubernetes.io/version: 0.8.0
# app.pst/component: send
# helm.sh/chart: ska-pst-test-parent-0.8.0
# resourceVersion - DELETE this field
# uid: - DELETE this field
# status: DELETE the entire status block
vi pst-pvc-shared-mnl.yaml
# Apply the pvc configuration
kubectl -n pst apply -f pst-pvc-shared-mnl.yaml
Deploying the Test Parent Chart
Update chart configuration.
# Create a copy of the test parents value.yaml
cp charts/test-parent/values.yaml pst-sdp-integration-external.yaml
# Override the following fields.
# global:
# data-product-pvc-name: shared-mnl - Update
#
# ska-sdp-dataproduct-dashboard:
# enabled: false - Disable deployment of DPD by test-parent
#
# ska-pst-core:
# send:
# enabled: true
# dpdApiIntegration: false - Disable api integration. The DPD FQDN in send is not dynamic yet.
# It does not affect the process concerning the files being transferred.
vi pst-sdp-integration-external.yaml
Install the pst test-parent chart
make K8S_CHART_VALUES=pst-sdp-integration-external.yaml k8s-install-chart