Custom Machine API Provider

For the Machine API to be able to interact with cloudscale.ch or Exoscale resources we will need to implement a custom Machine API provider.

At its core a Machine API provider watches Machine resources and creates, deletes, or updates virtual machines of the cloud provider. To do this we can leverage a framework provided by the Machine API Operator. We essentially "only" need to implement the Actuator interface. When implementing such a provider we can look at existing provider such as the machine-api-provider-gcp.

We see two valid approaches to implement such a provider:

Directly using the SDK would result in fewer moving parts and no direct dependency on crossplane, while using crossplane might reduce the amount of custom code and/or results in a unified way to interact with the underlying cloud provider.

For officially supported Machine API providers the Machine API Operator handles the deployment of all controllers. This includes the provider-machine-controller, machineset-controller, node-link-controller, machine-healthcheck controller, and multiple rbac proxies.

We can’t leverage this operator to deploy our own controllers, as the list of supported providers is hard coded. There is no clear reason why this couldn’t be handled in a more generic way, but in the foreseeable future we won’t be able to deploy our custom provider through the operator.

We will have to write a component that deploys all the controllers that are usually managed by the operator. This currently seems to be a single deployment, but we need to invest some effort to "reverse-engineer" the operator setup.

After deploying the custom Machine API provider, autoscaling workers should be as easy as creating a MachineSet and configuring the cluster autoscaler.

A machine infrastructure provider is responsible for managing the lifecycle of provider-specific machine instances.

This is essentially equivalent to the Machine API provider of the first option. The Machine Infrastructure Provider watches (different) Machine resources and creates, deletes, or updates virtual machines of the cloud provider. This could again be implement through crossplane or by directly using the SDK, but in any case we will need a specialized controller as the Cluster API resources are incompatible with crossplane resources.

There is a contract for a Machine Infrastructure Provider.

The Machine API provides the bootstrap configuration for new nodes in a well-known secret. For the Cluster API this is handled by the Bootstrap Provider. The provider writes the necessary bootstrap information to a secret on the management cluster and provides this secret to the Machine Infrastructure Provider.

It also needs to handle the initial bootstrapping of the cluster, but for our purposes it will only need to fetch the well-known secret from the target cluster and make it available on the manager cluster.

There is a contract for a Bootstrap Provider, we would probably only need to develop a subset of this to be usable for auto scaling.

To deploy the Cluster API we should define a central management cluster. We will have to write a component to deploy it, together with all implemented providers. We then need to give it access to the target cluster, probably through a service account.

Alternatively it should be possible to deploy the cluster API on every cluster, effectively being both management and target cluster.

After deploying the Cluster API with custom provider, autoscaling workers should be as easy as installing and configuring the cluster autoscaler

We need to implement the interface for the upstream autoscaler to interact with cloudscale.ch. We should most likely implement this as a gRPC service.

The cluster autoscaler assumes that each nodes is part of an instance pool that can be scaled (we can disable this for some nodes, for example for master nodes). This isn’t really the case for cloudscale.ch. They have the notion of servers and server groups, however server groups are only really used for anti-affinity and can’t be used to deploy and scale servers, so we would need to implement this ourselves.

We see two possible approaches to solve this:

On OpenShift automatic CSR approval is handled by the cluster-machine-approver. This however only supports nodes deployed through the machine API. So if we use this approach we would need to implement a similar controller ourselves. The controller by postfinance might solve this for us.

For this option we would need to deploy the upstream autoscaler, our cloudscale.ch gRPC provider, and our custom CSR approver. The advantage here would be that we need to change very little in the cluster setup and for existing clusters. Nodes deployed by terraform need to be annotated to not be removed and the rest should just work.

The Karpenter code base is generally designed to be extendable, however as we would be (one of the) first other cloud provider implementation we need to expect unexpected difficulties. After a quick assessment of the code base we would:

With that (and deployment and unexpected issues) we should have a standalone Karpenter instance that can create nodes on cloudscale.ch/Exoscale.

We most likely will still need a custom solution for CSR approvals.

All deployment guides are very AWS specific, however the deployment doesn’t seem very complicated. There is a helm chart that we probably need to adapt and we would need to think about the current terraform provisioning and how it would change.

Booting different nodes with CPU and Memory resources and ratio could be interesting to optimize utilization and for APPUiO Cloud we could potentially change our current fair use policies.

It’s unclear if and how we could use this to deploy all nodes as part of the installation.

Compared to the cluster-autoscaler this is a very young project. There isn’t much precedence for other cloud provider implementations so we expect subtle issues, incompatible designs, and upstream breaking our implementation with upgrades. Also the advantages over the standard cluster-autoscaler are in my opinion minor for our applications.