Data Copy Chart

The copydata-s3 chart is used to copy data to and from K8s PVC to AWS S3 buckets, used by batch processing scripts. In a k8s Job, it uses the ska-python base image and execute the data/boto3_threaded_copy.py python script, which uses boto3 to copy data to and from AWS S3 buckets.

Core Values

Name	Description	Default
image	OCI image to deploy	"artefact.skao.int/ska-sdp-python:0.1.0"
imagePullPolicy	Image pull policy	IfNotPresent
env	Environment variables to pass to processing script	null
resources	Kubernetes resource requests and limits for the Job	null
args	Arguments for the Python script boto3_threaded_copy.py	null

Connecting to the data storage Persistent Volume Claim

At the root level in values.yaml, add the dataProductStorage key with following value entries:

Name	Description	Default
name	Name of the PVC to connect to	shared

Connecting to the AWS S3 buckets

The VaultStaticSecrets key values.yaml configures AWS credentials stored as a secret in the vault. Modify with following value entries:

Name	Description	Default
mount	Where to mount it inside the vault	dev/
path	Path to the secret in the vault	phoenix/sdp-processing-scripts
secretName	Name of the secret	sdp-processing-scripts
refreshAfter	Validity of the credentials	24h