ADR0013 - Automatic Service Maintenance

Maintenance could get a controller of its own, in the way custos owns credentials and ampulla owns backups. It could also be split, with adept growing recurring rituals and chrysopoeia growing window-gated updates.

We give it to adept. Adept already handles operations: a Definition is a job template, an Action is one run of it, and it already builds Jobs in the instance namespace under the right ServiceAccount. Maintenance is that same idea with a schedule, not a new idea. Splitting it across two controllers was the worst of the options, because nothing would own the ordering between the two halves and the windows would have to live in one of them arbitrarily.

AppCat lets each instance name its own day and time. That is the most flexible option and the one our users already know, but it means every instance can pick an hour nobody is awake for, and the platform can never batch, stagger or staff anything. The opposite extreme, a single cluster-wide window, takes the choice away entirely.

We put a catalog in between. The service operator defines a small number of named windows as cluster-scoped objects, and a claim picks one by name.

kubectl get maintenancewindows lists all available windows. Instances sharing a window are spread across it by an offset derived from the instance itself, the same trick ferment already uses to stop every backup in the cluster starting in the same minute.

The window is expressed as days, a time and a duration rather than as a raw cron expression. It is the vocabulary users already have from AppCat, and it is the only form the framework can reliably shift an instance’s offset within.

An instance that names no window falls into a default one, the way an ampulla BackupPolicy falls back to the operator’s default bucket classes. Maintenance being on unless you opt out is the right default for a framework whose job includes shipping security patches.

Kubernetes CronJobs. The alternative we considered was a scheduler inside adept, keeping the next run time in an object’s status and requeueing until it came due. But we would have to rebuild a large chunk of Kubernetes' cronjob logic.

The CronJob is directly derived from the Definition’s `JobTemplateSpec. So the CronJob will directly run the logic defined.

Alternatively, the CronJob simply creates an Action CR and lets adept handle the rest. But the overhead to spawn a job, to create a CR which spawns another job, is too high.

The per-instance contract is a small object the reagent chart renders into the instance namespace, exactly the way ferment already renders a BackupPolicy, an Arcanum and a Seal.

An empty spec is a complete schedule that takes the operator’s defaults, in the same spirit as ampulla’s policy. Going through the chart leaves the Service Maintainer with the most flexibility.

There are two different things people mean by "update the service", and each one is handled the same way.

The first is the reagent chart version. Chrysopoeia already discovers the available versions, and the generated API group is scoped to a single major, so the versions on offer are minor and patch. Major version upgrades are out of scope and should always be manual actions.

spec.version controls the chart updates. Left empty it means "managed, keep me current"; set to a value it means "this exact version and nothing else". Stopping chart updates therefore means naming a version, and the claim’s status already reports which one is running. A pinned instance still runs its ritual, because switching off updates should not switch off housekeeping.

The second is the service’s own version inside the instance. This is individual for every service, and only the maintainer knows what moving it involves. As such it has to be handled in the reagent chart. Best practice: the service maintainer defines a map of major versions as keys and exact image versions as value. Image version will then be bumped by the service maintainer, with the help of renovate and put into new reagent revisions.

The framework still owns the switch. A user should not have to learn a different way to freeze every service they run. Ferment ships a maintenance block in its values, so the same field sits at the same path on every reagent’s claim:

We don’t specify when the ritual runs, it will be an implementation detail. The only thing guaranteed is that the ritual is not run during the helm upgrade.

There is no framework-level retry. The Job’s own backoffLimit lives in the maintainer’s Definition, which is where the knowledge about whether the operation is safe to repeat actually is.

At a first glance they both do the same thing. One big difference: the ritual will run every maintenance window, even if the version is pinned. Its job is primarily doing housekeeping, like vacuum a database.

Any auxiliary jobs that should run during a helm upgrade should be handled via helm hooks. This is an established mechanism and a familiar concept for everyone who uses helm.

Something has to record "this instance may move to the newer version now".

Writing it into the claim’s spec.version is the obvious move and the wrong one. Claims are usually managed by the customer’s GitOps, so the framework and Argo would both be owners of the same field, and adept would need write access to every claim in the cluster to do it.

We write it into status.version instead. The resolution logic is simple. If there’s a value in spec.version, it takes precedence, otherwise status.version is written to the revision. If both are not set, the status.version is updated with the latest available.