Crossplane Split Architecture

This page contains descriptions about how we can operate services in a split architecture.

Figure 1. Overview AppCat Architecture

RBAC and Service Accounts

The connection between the control-plane and the service-clusters is 1:N. Each control-plane will need kubeconfigs to N service clusters. Each service-cluster will need kubeconfigs to one control-plane.

To make provisioning the kubeconfigs easier, the component will pre-provision service accounts and the correct RBAC on each of the clusters.

On the control-plane: there’s a service account named appcat-service-cluster, which service clusters can use to connect to the control-plane
On service-clusters: there’s a service account named appcat-control-plane which the control-plane can use to connect to each service-cluster.

Due to the nature of SYN, we can’t create the kubeconfigs during compilation time. So each time a new service-cluster gets added to a control-plane, the kubeconfig has to be generated and put into Vault manually.

The steps how to generate a kubeconfig from a service-account are documented in the APPUiO Docs.

Gotchas and Pitfalls

This chapter shows the pitfalls and gotchas that we need to keep in mind if we develop for a split architecture.

Provider Configs

The first issue we need to tackle is managing multiple provider configs for provider helm and Kubernetes. They need to be able to connect to the service clusters from the control cluster.

Connection Details

Connection details are only available on the control cluster. They are not propagated to the service clusters.

There are various services were the connection details are saved directly into the instance namespaces. By splitting the control cluster from the service cluster, this won’t be possible anymore. Instead of writing the connection details directly into the instance namespaces, we’ll have to wrap them into separate secrets and deploy them via provider-kubernetes.

Webhooks

The backend deletion protection webhooks need to be able to check against the composites on the control cluster.

So each of the service clusters also needs a connection back to the control cluster, in order for the webhooks to do the lookups.

Managed Resource vs Plain Kubernetes resources

By splitting the control clusters from the service clusters it gets even more important to cleanly differentiate between Crossplane Managed Resources and plain Kubernetes resources.

Managed resources need to always be deployed on the control cluster. They will not be available on the service clusters.

Usually those objects implement the resource.Managed interface. But unfortunately not all the objects do that, for example ProviderConfigs and Usages. Ideally the runtime should take care of wrapping objects automatically.