Triggering Automated Maintenance

We can extend our customized Renovate fork to support updating the desired OpenShift 4 version in the configuration hierarchy. Since we’ve already written multiple custom Renovate managers, identifying and extracting the current desired OpenShift 4 version from the configuration should be straightforward.

Renovate’s data sources are used to determine available updates for a dependency identified by a manager. In contrast to custom managers, we’ve not yet implemented a custom data source for Renovate. However, looking at the documentation, it should be fairly straightforward to implement a data source which queries api.openshift.com for available OpenShift 4 upgrades. There’s already code which queries this API in the OpenShift managed upgrade operator and our own custom OpenShift 4 maintenance tool.

To ensure upgrades are started at a specified time we can leverage Renovate’s support for scheduled updates and automerging.

Monitoring trigger failures for the Renovate approach would require monitoring both Renovate’s execution, and the Commodore CI pipelines for all clusters. This would require a significant amount of engineering, since we don’t have any infrastructure to do so in place yet.

Skipping maintenance when a cluster is unhealthy would require either a feedback loop between Renovate or the Commodore CI pipeline and the cluster, or an additional component on the cluster which can block a desired upgrade pushed into the cluster catalog. Having an additional component on the cluster will introduce additional complexity and raise the question of why that component doesn’t a larger amount of work to orchestrate the upgrade itself. On the other hand, implementing a feedback loop between the cluster and Renovate or the CI pipeline creates additional dependencies which may not be desirable.

We can use RHACM to automate cluster upgrades. RHACM itself appears to be targeted at manual operations work more than automated GitOps workflows. For example, upgrading clusters is documented as a manual process through clicking in the web interface.

Given this documentation, it looks like we’d need to engineer custom tooling which interacts with RHACM to update managed clusters to the desired version at the right time.

Overall, RHACM does a lot more than just cluster upgrades, and doesn’t provide enough upgrade functionality out of the box to fulfil our requirements. Given our experience with managing Openshift 4 through GitOps, it’s quite likely that trying to interact with RHACM through GitOps would be finicky, if it’s even possible at all.

The Openshift managed upgrade operator supports scheduled upgrades. The operator is installed on each cluster, and watches UpgradeConfig resources. It sets the cluster’s desired version based on the UpgradeConfig resources present on the cluster. Notably, the UpgradeConfig contains a desired OpenShift patch version, which would need to be injected into the cluster somehow. See the README on GitHub for details on how the upgrade process works in detail.

To be able to fully utilize the managed upgrade operator, we’d have to adjust the implementation, since the operator currently only looks at the worker machine config pool to determine whether the node upgrades have completed.

The managed upgrade operator already performs some simple health checks and won’t trigger a requested upgrade if the cluster is unhealthy.

The managed upgrade operator does provide some metrics which can be used to monitor an upgrade. There’s a metric which indicates whether there was an error while triggering the upgrade, as well as metrics indicating the progress of the upgrade.

We could use a policy tool, such as Kyverno, to orchestrate OpenShift 4 upgrades.

A possible policy to orchestrate upgrades would check the ClusterVersion object for available updates and would update the cluster’s desired version to the latest available upgrade. However, it’s unclear how we could extend such a policy to ensure that upgrades are only applied at a specified time.

Finally, our experience with Kyverno for APPUiO Cloud policies has shown that any reasonably complex policies are hard to engineer and hard to test.

Monitoring the upgrade would be feasible with most policy tools, as they usually provide metrics regarding policy execution errors.

Introducing a pre-upgrade health check should be possible with a policy, since most policy tools have some concept of preconditions which must hold for the policy execution. We might have to implement an additional component which checks the cluster health periodically and updates a configmap with the health check results in order to make them available to the policy.

Instead of adjusting openshift-managed-upgrade-operator to support our requirements, or trying to write a rather complex policy for orchestrating automated updates, we could implement our own controller which orchestrates OpenShift 4 version updates.

By running an active component in the cluster we can easily monitor the cluster’s state and upgrade progress from the same tool which triggers the upgrade. If we implement the active component from scratch, we’ve got full control over the tool’s development. This allows quick turnaround for adding features and fixing bugs, and avoids the potentially high maintenance burden of pulling in upstream updates if we were to for openshift-managed-upgrade operator.

Additionally, it’s unlikely that we could upstream our changes to openshift-managed-upgrade-operator, as we probably have different requirements than the OpenShift Dedicated product. Since we already have sufficient experience in implementing Kubernetes controllers/operators, the additional effort for the initial implementation should be acceptable.

In order for this approach to actually reduce complexity compared to the Renovate-based approach, we need the controller to autonomously update the desired cluster version. Notably, we don’t gain anything except for additional moving parts if we make the desired OpenShift patch version an input for the controller. Therefore the biggest drawback of this approach is—​as with any option that’s based on an active component in the cluster—​that we lose some amount of control over the desired version of a cluster in the GitOps repositories. Additionally, we don’t automatically have a persistent history of version updates for each cluster based on the commits in the cluster’s catalog repository.

However, by implementing our own controller, we should be able to add support for a persistent update history outside the cluster with reasonable effort.