ADR0013 - Automatic Service Maintenance

Maintenance could get a controller of its own, in the way custos owns credentials and ampulla owns backups. It could also be split, with adept growing recurring rituals and chrysopoeia growing window-gated updates.

We give it to adept. Adept already handles operations: a Definition is a job template, an Action is one run of it, and it already builds Jobs in the instance namespace under the right ServiceAccount. Maintenance is that same idea with a schedule, not a new idea. Splitting it across two controllers was the worst of the options, because nothing would own the ordering between the two halves and the windows would have to live in one of them arbitrarily.

AppCat lets each instance name its own day and time. That is the most flexible option and the one our users already know, but it means every instance can pick an hour nobody is awake for, and the platform can never batch, stagger or staff anything. The opposite extreme, a single cluster-wide window, takes the choice away entirely.

We put a catalog in between. The service operator defines a small number of named windows as cluster-scoped objects, and a claim picks one by name.

kubectl get maintenancewindows lists all available windows. Instances sharing a window are spread across it by an offset derived from the instance itself, the same trick ferment already uses to stop every backup in the cluster starting in the same minute.

The window is expressed as days, a time and a duration rather than as a raw cron expression. It is the vocabulary users already have from AppCat, and it is the only form the framework can reliably shift an instance’s offset within.

An instance that names no window falls into a default one, the way an ampulla BackupPolicy falls back to the operator’s default bucket classes. Maintenance being on unless you opt out is the right default for a framework whose job includes shipping security patches.

Kubernetes CronJobs. The alternative we considered was a scheduler inside adept, keeping the next run time in an object’s status and requeueing until it came due. But we would have to rebuild a large chunk of Kubernete’s cronjob logic.

The CronJob is directly derived from the Definition’s `JobTemplateSpec. So the CronJob will directly run the logic defined.

Alternatively, the CronJob simply creates an Action CR and let’s adept handle the rest. But the overhead to spawn a job, to create a CR which spawns another job, is too high.

The per-instance contract is a small object the reagent chart renders into the instance namespace, exactly the way ferment already renders a BackupPolicy, an Arcanum and a Seal.

An empty spec is a complete schedule that takes the operator’s defaults, in the same spirit as ampulla’s policy. Going through the chart leaves the Service Maintainer with the most flexibility.

There are two different things people mean by "update the service", and each gets its own switch.

The first is the reagent chart version. Chrysopoeia already discovers the available versions, and the generated API group is scoped to a single major, so the versions on offer are minor and patch. Major version upgrades are out of scope and should always be manual actions.

spec.version controls the chart updates. Left empty it means "managed, keep me current"; set to a value it means "this exact version and nothing else". Stopping chart updates therefore means naming a version, and the claim’s status already reports which one is running. A pinned instance still runs its ritual, because switching off updates should not switch off housekeeping.

The second is the service’s own version inside the instance. This is individual for every service, and only the maintainer knows what moving it involves, so it has to be specified in the ritual.

The framework still owns the switch. A user should not have to learn a different way to freeze every service they run. Ferment ships a maintenance block in its values, so the same field sits at the same path on every reagent’s claim:

appVersion should behave the same way as spec.version. If left empty, the ritual should bump the app images to the latest minor/patch versions.

This is where the framework’s guarantee ends. It guarantees the switch exists, is named the same everywhere, and reaches the ritual. It cannot guarantee the ritual honours it, the way it enforces spec.version itself. So it’s in the maintainers responsibility to configure the ritual accordingly. It’s also feasible that a maintainer simply pins the app version to a chart version, skipping a separate appVersion.

Running the update first would let a ritual be written against the version it is about to run under, which is occasionally what you want. It also means waiting for a Helm release to settle before firing a job, and deciding what to do when it never settles.

We run the ritual first and only move the version if it succeeded. An instance that cannot complete its own maintenance is an instance we should not be upgrading, and a failure then has exactly one meaning: nothing changed, and the next window will try again.

There is no framework-level retry. The Job’s own backoffLimit lives in the maintainer’s Definition, which is where the knowledge about whether the operation is safe to repeat actually is.

Something has to record "this instance may move to the newer version now".

Writing it into the claim’s spec.version is the obvious move and the wrong one. Claims are usually managed by the customer’s GitOps, so the framework and Argo would both be owners of the same field, and adept would need write access to every claim in the cluster to do it. Kubernetes permissions cannot be narrowed to a single annotation, so that grant would also let adept rewrite any claim’s spec.

We write it on the instance namespace instead. The namespace is created by the framework, not by the customer, so there is no drift for a GitOps tool to fight over, and adept needs only to patch namespaces rather than to hold write access on every service instance in the cluster. The claim’s status will still reflect, what’s available and what is actually running.