ADR 0054 - VSHN Managed Forgejo Runners

The official Forgejo runner Helm chart is used to deploy the runner. It is the officially maintained chart and is structurally identical to the wrenix chart used in the PoC, which was its upstream before the Forgejo project took it over.

It exposes everything we need:

The chart can be forked if we need to tweak it for our use case in the future, similar to how we handle other third-party charts.

The runner is modelled as an AddOn, not a nested service. A nested service is a part the outer service requires and cannot run without, and that is not billed separately. PostgreSQL in Keycloak and Nextcloud follows this pattern. The runner is the inverse: Forgejo runs perfectly well without it, enabling it is optional, and it carries its own billing. That makes it an AddOn, the same model used for Collabora. The runner cannot function without a Forgejo instance, so it is tightly coupled, but coupling alone does not make something a nested service. Optionality and separate billing are what make it an AddOn.

This classification is independent of where the runner Pod is placed: the same AddOn can live in the Forgejo instance namespace or in its own namespace. The options below cover the shared-vs-per-instance choice, the per-instance placement choice, and, for completeness, the out-of-cluster alternative.

The CSP compute option also unblocks privileged / Docker-in-Docker execution. On a shared cluster, a privileged job or a container escape reaches the node, and potentially other tenants and the control plane, so it must be avoided (see Job execution and isolation). On a dedicated VM, ideally ephemeral (--ephemeral, one job per runner), the blast radius of an escape is the disposable VM itself. That makes container.privileged = true and DinD an acceptable trade-off for builds that genuinely need them. The cost is provisioning latency, recurring CSP compute spend, and a dependency on a per-CSP provisioning provider. The one remaining security item is network segmentation: an isolated subnet plus firewall, so an escaped job cannot reach internal services.

This option has two sub-decisions of its own, both evaluated below: how the runner is installed on the VM, and how the VM itself is provisioned per CSP.

A natural concern is whether the runner spawns additional job Pods, and if so whether their number per instance must be capped, since this drives capacity management and resource billing.

With the forgejo-runner chart it does not spawn extra Pods per job. A CI run is confined to the pre-provisioned runner Pod(s): jobs execute inside the running runner, and when all runner capacity is busy, further jobs queue until a runner frees up rather than scaling out new Pods. (Whether multiple jobs can share a single Pod concurrently depends on the runner’s capacity setting; in the PoC the chart ran jobs within the provisioned Pod rather than fanning out.)

This bounds resource and storage usage to the provisioned runner Pods, which keeps capacity and billing straightforward. Concurrency is determined by the configured runner capacity and the number of runner Pods per instance, both fixed at provisioning time, instead of an unbounded pool of dynamically spawned job Pods. Customers therefore pick from a small set of pre-defined runner sizes (CPU / memory / storage), and the instance’s runner footprint is known up front.

The runner executes arbitrary customer CI workloads, so the executor backend is a security-relevant choice. The runner runs as a Kubernetes Pod, so host and LXC executors are not viable: host execution requires direct host access, and LXC needs kernel-level container nesting, both incompatible with restricted SCCs on a shared cluster. The real choice is between Docker-in-Docker and rootless container execution.

Forgejo’s official security guidance and docker access documentation are the reference for the evaluation below. Docker socket automount (container.docker_host = automount) is ruled out in both options, since Forgejo’s docs classify it as offering "no security isolation."

We built a proof of concept (appcat#684) to validate the registration flow and the Helm chart integration. It deploys into the Forgejo instance namespace, matching the placement decided here, and uses the registration mechanism we are deciding on. provider-http registers the runner against the Forgejo admin API, the composition reads the returned token and writes it into a .runner config secret, and the forgejo-runner Helm chart consumes that secret.

What remains is to expose runner enablement and sizing on the claim instead of always provisioning one, and, on Servala, to wire up the gVisor RuntimeClass for the job containers.

Namespace isolation plus rootless, confined-Pod execution (see Job execution and isolation) is the decided baseline and meets the requirement on our target platforms (see the Decision). The following are optional defence-in-depth we can layer on if we choose to harden further, not prerequisites:

CI jobs pull large volumes from external sources: npm, PyPI, Maven, Go modules, ad-hoc curl … | sh. Spegel covers container image layers on Servala, but not these other artifacts. Two gaps to address in implementation:

Storage sizing for this cache ties into the pre-defined runner sizes from Job concurrency and capacity.

Customers need access to their job logs (surfaced in the Forgejo UI) and runner metrics (runner metrics endpoint). No SLOs exist on Servala yet, but when they are defined the responsibility line should be drawn explicitly: runner availability is on us, job success is on the customer. Keeping that distinction clear avoids support-ticket ambiguity.

Review raised a possible future use case: offering the runner as a standalone service where the customer brings their own Forgejo instance, managed by them or a third party, rather than a VSHN-managed one. This is out of scope for this ADR, which scopes the runner as an AddOn to a VSHN-managed Forgejo instance, and the decision above is unchanged. It is captured here so the constraint it imposes is not lost.

The registration mechanism decided here does not extend to that use case. provider-http registers the runner by calling the Forgejo admin API (POST /api/v1/admin/actions/runners) with admin credentials AppCat controls, against the in-cluster instance reached over internal service DNS (<instance>-http.<namespace>.svc:3000). For a customer-owned instance we have neither admin credentials nor any business mutating an instance we do not operate, so server-side registration is off the table.

The natural fit is to invert the flow: instead of AppCat registering the runner, the customer generates a runner registration token on their own Forgejo and hands it to us. The claim would carry a secretRef (a corev1.SecretKeySelector, mirroring the existing UnmanagedBucket and Keycloak custom-mount patterns in AppCat) pointing at a Secret with that token, plus the instance’s external URL. The composition reads the token and renders the .runner config Secret directly, then points the forgejo-runner chart at it via runner.config.existingSecret, the same chart hook the AddOn already uses, just skipping the provider-http registration step entirely.

This has knock-on consequences that confirm it belongs in a separate decision rather than this one:

If this use case is pursued, it should be a new ADR building on the secret-ref idea above, not a modification of this AddOn.