See Marvin validate, sign, and promote artefacts#
This guided walkthrough takes you through the full artefact lifecycle by feeding Marvin real pushes and watching what it does:
a compliant artefact that is promoted (Harbor) or accepted (Nexus);
a non-compliant one that is held back, labelled or quarantined, and raised as a merge request;
the OCI signature flow end to end — an image that is gated until it is signed, an SBOM that Marvin signs on its own, and the verification chain that proves both.
It is learning-oriented: follow the steps in order and you will see each gate act in turn. For the reasoning behind the gates see How the Artefact Validations service works; for the exhaustive settings, toggles, and retry budgets see Endpoints, toggles, configuration, and metrics; for day-to-day operational recipes see Run and operate the Artefact Validations.
It assumes Marvin is already running (see Deploy with the Helm chart)
and only covers exposing it, wiring the webhooks, and feeding it artefacts.
Nexus validates every supported format (Helm, Raw, Conan and PyPI); this
walkthrough uses a PyPI package in the pypi-test repository, and the
Harbor side uses an OCI image.
What you need#
Marvin deployed and reachable, with its Celery worker logs to hand (you will follow each decision as it happens).
Push access to a Harbor staging test project (
test-validations) and visibility of the production test project (test-promotion), plus a Nexuspypi-testrepository.For the signature flow, on your own machine:
Note
<...> marks a value specific to your environment — a host, a project, a
tag, or a digest you read off Harbor.
Step 1 — expose Marvin to the registries#
Nexus (artefact.skao.int) and Harbor (harbor.skao.int) must be able to
reach Marvin’s webhook receiver over the internet.
Running locally: put a tunnel in front of the receiver, for example with ngrok:
ngrok http <marvin-port>Note the public
https://<subdomain>.ngrok.ioURL it prints; that is your<base-url>for the next step.If Marvin is deployed to a local cluster (for example one provisioned with ska-cicd-deploy-minikube) the receiver is not directly reachable from your host, so forward its service to a local port first and tunnel that (the receiver listens on port
80):kubectl port-forward -n <marvin namespace> --address 0.0.0.0 svc/<marvin-service> 8080:80 ngrok http 8080
Running in a cluster with a public Ingress: skip the tunnel and use the receiver’s Ingress host as
<base-url>(https://<marvin-ingress-host>); the registries reach it directly.
Step 2 — point the test-project webhooks at Marvin#
Configure the Harbor and Nexus webhooks as described in
Register the webhooks, using the <base-url> from Step 1:
Harbor uses two webhooks, on two projects. Both set the
Auth Headerto the value of Marvin’sHARBOR_TOKEN(Harbor sends it as theauthorizationheader and Marvin rejects the request unless it matches):On the staging project (
test-validations), an Artifact pushed webhook to<base-url>/harbor— this drives validation and promotion.On the production project (
test-promotion), a Scanning finished webhook to<base-url>/harbor/sbom_sign— this drives SBOM signing.
Nexus — on the
pypi-testPyPI repository, add a Webhook: Repository capability for thecomponentevent pointing at<base-url>/nexus. Set the Secret Key to the value of Marvin’sNEXUS_HMAC_SIGNATURE_SECRET(Nexus HMAC-signs the payload with it and Marvin verifies the signature using the same secret).
Check whether these webhooks already exist before creating new ones; often you only need to repoint the URL.
Note
Promotion copies to whatever config.harbor.promotion_repository is set
to. The chart’s default is test-promotion, which matches the production
test project above, so test-validations → test-promotion works out
of the box. A real deployment sets it to production (see
Configure the service behaviour).
Before feeding artefacts, make sure the checks you want to exercise are enabled (see Enable and disable feature toggles) and tail the Celery worker logs so you can follow each decision as it happens.
Step 3 — feed a compliant artefact#
Publish a compliant artefact from your test project’s CI pipeline: one
whose name follows the SKAO convention (ska-...) and whose version is valid.
Push an OCI image with a SemVer tag to the test-validations project, or a
PyPI package with a PEP 440 version to pypi-test.
Marvin runs the checks and, because they all pass:
Harbor — copies the image into the
test-promotionproject (the worker logs... promoted to test-promotion repository). The image now appears there.Nexus — leaves the PyPI package in place; nothing is quarantined.
Step 4 — feed a non-compliant artefact#
Now publish a deliberately non-compliant artefact — for example, a name
without the ska- prefix, or an invalid tag such as latest or v1.
Marvin records the failed checks and holds the artefact back:
Harbor — applies an
Invalid*label (InvalidName,InvalidTag, …) to the artefact intest-validationsand does not promote it.Nexus — moves the PyPI package to the
pypi-quarantinerepository (the quarantine repo is the source project name up to the first-plus-quarantine, sopypi-test→pypi-quarantine).
In both cases Marvin then opens a merge request in the artefact’s GitLab project, with a table of the failed checks and how to fix them, assigned to the author.
Important
The merge request and the metadata check rely on CI-injected metadata
(CI_PROJECT_ID, CI_JOB_ID, GITLAB_USER_ID and the other
CI_*/GITLAB_* fields). Publish through the pipeline so these are
present. A hand-pushed artefact still shows the labels and quarantine, but
Marvin cannot open the merge request without them.
Step 5 — test the OCI signature flow end to end#
So far promotion depended only on the checks. This step adds the signature gate: with it on, an image is promoted only once both the image and its SBOM are signed. You will watch Marvin refuse to promote an unsigned image, sign the image yourself (standing in for the pipeline’s image signer), watch Marvin sign the SBOM, and then verify the whole chain.
Two roles produce the two signatures:
The image is signed by an external signer — in production, the CI pipeline. Here you sign it with
notation.The SBOM is signed by Marvin: when Harbor reports that a scan has finished, the
/harbor/sbom_signwebhook has Marvin mint a short-lived Vault certificate and countersign the SBOM withnotation(see How the Artefact Validations service works).
Turn on the gate#
Enable two toggles for the test project (see Enable and disable feature toggles):
require-signatures— makes the promotion gate require a signed image and a present SBOM (and a signed SBOM too, ifverify_sbom_signatureis on).sign-sbom— lets Marvin sign the SBOM when the scan finishes.
Whether the gate also requires the SBOM to be signed is governed by the
verify_sbom_signature config field, which is false by default: the gate
then requires the SBOM only to be present, not signed. This matches the
flow you are testing, where Marvin signs the SBOM after promotion (on the
production scan), so requiring a signed SBOM before promotion would deadlock.
Leave it false unless your project signs SBOMs in staging — see
Configure the service behaviour.
Push an unsigned image and watch the gate hold#
Push a compliant, unsigned single-platform image to staging:
docker push harbor.skao.int/test-validations/pandabox:1.0.0
Harbor fires the Artifact pushed webhook and Marvin runs the checks, then the
signature gate. Because the image is not signed yet, the worker logs that the
artefact is “not fully signed yet”, polls its accessories
signature_check_retries × signature_check_retry_delay_s (5 × 30 s by
default), then raises ArtefactNotSignedError and retries the whole task
every 10 minutes for up to an hour. The image stays in test-validations,
unpromoted.
Sign the image#
Read the image’s manifest digest off its page in Harbor (or with
regctl manifest head), then sign that digest — signing the tag would sign a
moving target:
notation sign --timestamp-url http://timestamp.digicert.com \
--timestamp-root-cert <path to DigiCertTrustedRootG4.crt> \
harbor.skao.int/test-validations/pandabox@sha256:<image-digest>
The --timestamp-* flags countersign the signature with the DigiCert RFC 3161
timestamping authority so it stays verifiable after the signing certificate
expires — the same authority Marvin uses for the SBOM.
Watch Marvin promote it#
Either wait for the next task retry (up to 10 minutes) or replay the push
immediately without re-pushing the image — see
Re-trigger a Harbor validation. This time the image signature is present,
so once the SBOM requirement is also met the gate passes and Marvin copies the
image into test-promotion (... promoted to test-promotion repository).
When Harbor then reports the scan finished, the /harbor/sbom_sign webhook
has Marvin sign the SBOM.
Note
If the image never promotes, the signature gate is the usual cause. Walk through Investigate an image that never gets promoted to find which accessory is missing.
Verify the whole chain#
Confirm the image signature verifies on the promoted copy, by tag and by digest:
notation verify -v harbor.skao.int/test-promotion/pandabox:1.0.0
notation verify -v harbor.skao.int/test-promotion/pandabox@sha256:<promoted-image-digest>
Then list the artefact’s accessories and confirm both an sbom.harbor SBOM
and the signature.notation signatures are present (the image signature, and
Marvin’s signature nested on the SBOM):
regctl image copy --referrers harbor.skao.int/test-promotion/pandabox:1.0.0 ocidir:///tmp:pandabox
regctl artifact list harbor.skao.int/test-promotion/pandabox:1.0.0
Prove SBOM signing is idempotent#
Marvin signs each SBOM once. Re-deliver the Scanning finished event for the
same image — post a SCANNING_COMPLETED body to /harbor/sbom_sign the
same way you replay a push (see Re-trigger a Harbor validation). The body
must name the SBOM scan, otherwise it is ignored:
{
"type": "SCANNING_COMPLETED",
"occur_at": 1744716446,
"operator": "admin",
"event_data": {
"resources": [
{
"digest": "sha256:<image-digest>",
"resource_url": "harbor.skao.int/test-promotion/<repo>@sha256:<image-digest>"
}
],
"repository": {"name": "<repo>", "namespace": "test-promotion"},
"scan": {"scan_type": "sbom"}
}
}
Because the SBOM already carries a signature.notation accessory, the worker
logs that it is “already signed” and returns without creating a second
signature. Duplicate webhooks, Celery retries, and re-scans are therefore cheap
no-ops.
Swap "scan_type": "sbom" for "vulnerability" and the request is
acknowledged with {"message": "Ignored"} instead: Harbor reports both scans
under the same event type, and only the SBOM one has an SBOM to sign.
Multi-platform images#
For a multi-platform image (an OCI index) the gate requires the index and every child manifest to be signed, and every child to have its own SBOM (except BuildKit attestation manifests, which only need a signature). Copy a multi-platform image into staging and sign each platform digest in turn:
regctl image copy artefact.skao.int/ska-base:1.0.0-rc.1 \
harbor.skao.int/test-validations/ska-base:1.0.0-rc.1
notation sign --timestamp-url http://timestamp.digicert.com \
--timestamp-root-cert <path to DigiCertTrustedRootG4.crt> \
harbor.skao.int/test-validations/ska-base@sha256:<platform-digest>
# repeat for every remaining platform digest, and for the index digest
After promotion, confirm the index and each child carry their signatures:
regctl manifest get harbor.skao.int/test-promotion/ska-base:1.0.0-rc.1
regctl artifact list harbor.skao.int/test-promotion/ska-base@sha256:<index-or-child-digest>
If a single child is left unsigned the gate holds the whole image back, exactly as in the single-platform case.
Where to next#
The gate logic and the reasoning behind it: How the Artefact Validations service works.
Exact toggles, config fields, and retry budgets: Endpoints, toggles, configuration, and metrics.
Operating Marvin day to day — replaying webhooks, diagnosing a stuck image, running the cleanup script: Run and operate the Artefact Validations.