Distributing cluster CA and kubeconfig to users
Problem
We’ve decided that VSHN Managed Talos uses the default per-cluster self-signed CA certificate managed by cluster API. Because of that decision, we need to define how we distribute VSHN Managed Talos CA certificates to users.
Additionally, we’ve decided to use OIDC authentication via id.vshn.net for user authentication.
Because of that decision, we also need to define how we distribute VSHN Managed Talos kubeconfigs with a preconfigured user section to users.
Notably, since we need to distribute kubeconfigs anyway, we may not distribute the CA certificate on its own at all.
Proposals
Central service that serves all VSHN Managed Talos kubeconfigs
We could run a central service where VSHN customer users can download their VSHN Managed Talos kubeconfigs. However, such a service would probably require authentication so that it can provide each customer a tailored list of their VSHN Managed Talos clusters.
Additionally, such a service requires that either each cluster sends its current CA certificate and SSO configuration to the service, or that the service pulls all cluster CA certificates and SSO configurations from the VSHN Managed Talos clusters. Since we already have the Project Syn Lieutenant API where we can push arbitrary "dynamic facts" from each cluster, we’d definitely use a push model to have each cluster provide its current CA certificate as a dynamic fact. Because we use Project Syn as the source of truth for configuring the OIDC client for each cluster, the SSO configuration can be derived from existing data in the Lieutenant API already.
There are multiple options for implementing and hosting such a service.
One obvious choice would be to integrate this feature in the VSHN control panel (control.vshn.net). With this approach, the authentication and authorization part would already be solved through the control panel’s usual authentication. Additionally, customers already use control.vshn.net to interact with VSHN, so adding a section where they can see their VSHN Managed Talos clusters probably is a good idea anyway. The VSHN control panel also already has Lieutenant integration, so it could fetch each cluster’s CA certificate (or a pre-rendered kubeconfig) from the Lieutenant API easily.
Alternatively, we could host the central service on the Lieutenant vcluster. This would neatly fit the envisioned approach of each cluster pushing its CA certificate to the Lieutenant API. However, since this approach would require implementing yet another control panel / portal, it makes this option significantly less attractive.
On-cluster service that serves the kubeconfig
When looking at approaches that serve a pre-rendered kubeconfig containing the cluster’s self-signed CA certificate and a user configuration that enables OIDC login via id.vshn.net from the cluster, we need to decide how we render this kubeconfig.
The obvious choice here is Espejote which can dynamically re-render the kubeconfig as soon as the cluster’s self-signed CA certificate changes and which can store the rendered kubeconfig in a ConfigMap on the cluster.
Anonymous access to the ConfigMap via cluster API
While we could enable anonymous authentication (possibly only starting with Talos 1.14), to allow users to download the rendered kubeconfig from the cluster’s Kubernetes API, a large part of the problem we’re solving here is to distribute the cluster CA to users. Notably, we need to distribute the cluster CA to users so they can securely connect to the cluster’s Kubernetes API. Therefore implementing cluster CA distribution over the Kubernetes API isn’t a great choice: users would need to fetch the kubeconfig (and CA certificate) while skipping certificate verification. This presents an opportunity for malicious actors to intercept these requests and serve a kubeconfig containing a CA certificate that they control. Essentially, this approach enables anyone who can modify a user’s DNS to intercept kubeconfig download requests and inject a man-in-the-middle CA.
Service exposed on the cluster’s ingress controller
The alternative is to implement a small on-cluster service which serves the cluster’s CA certificate and a pre-rendered kubeconfig over an endpoint secured by a public certificate authority, such as Let’s Encrypt. This addresses the shortcomings of the first option without requiring a central service to be available.
While the actual implementation of this service isn’t directly relevant to the decision, we will briefly outline a few options in the rest of this section.
For serving the resulting kubeconfig, the first option would be to leverage the Kubernetes ingress backend.resource feature.
However, the Cilium ingress controller panics when trying to render an ingress that sets backend.resource, so we currently can’t leverage that feature.
The alternative is to serve the resulting kubeconfig from a small web server.
Since we’ve made good previous experiences with Caddy, that’s an obvious choice to serve the mostly static pre-rendered kubeconfig from a ConfigMap.
Additionally we can easily serve a small landing page that provides "First time setup" instructions and shows the kubeconfig in copy-pastable form for users that prefer to copy the kubeconfig to a local file from their browser.
Finally, we can enable command line users to directly download the kubeconfig from the top-level URL if they provide a specific Accept http header.
Decision
We implement CA certificate and kubeconfig distribution as an on-cluster service exposed on the cluster’s ingress controller.
We use Espejote to render a kubeconfig that uses the kubectl kubelogin plugin for OIDC authentication via id.vshn.net.
The kubeconfig (and a static explanatory landing page) are served by a Caddy server exposed on the cluster’s ingress controller and secured by a Let’s Encrypt certificate.
Rationale
Distributing a kubeconfig that works out of the box with OIDC authentication via id.vshn.net makes the setup (and kubeconfig update) process very straightforward for customers.
We’ve decided to distribute the kubeconfig through a service running in the cluster partially because this is the easiest method to implement short term. However, this approach also offers the most self-contained approach that doesn’t require any central services (except for id.vshn.net) to be available in order to enable users to access their VSHN Managed Talos clusters.
Notably, we may still implement the option for users to view their VSHN Managed Talos clusters and download kubeconfigs via control.vshn.net in the future.