API Certificates

Problem

We need to decide how we handle the Kubernetes API server certificate for VSHN Managed Talos clusters.

Talos generates a self-signed Kubernetes root CA per cluster and dynamically issues the API server’s serving certificate from that CA on each control plane node. The serving certificate contains internal and external IPs, the control plane endpoint, and the standard in-cluster names (kubernetes.default.svc etc.) in its subject alternative names. When provisioning clusters through Cluster API, the CA is generated once by the Talos bootstrap provider (CABPT) and stored in the management cluster (with our Cluster API architecture, that’s the cluster itself).

This raises the question of whether users' kubeconfigs must contain the cluster CA certificate, whether we can instead present a publicly trusted certificate on the API endpoint, and what happens when the CA is rotated.

Goals

  • Decide how the Kubernetes API server certificate is issued for VSHN Managed Talos clusters

  • Decide what kind of API server certificate a VSHN Managed Talos cluster presents to external clients

    • Understand and document the implications for users (kubeconfig contents, CA rotation)

  • The certificate must be valid for the endpoints users actually connect to, including an external FQDN

  • The decided approach must work with our Cluster API-based provisioning (CABPT/CACPPT)

Non-Goals

  • Completely replace the in-cluster CA

  • Certificates for Ingress or workloads

  • User authentication

  • Talos API (apid) certificates. These are only relevant for VSHN-internal access via talosctl

Proposals

Let’s Encrypt / public CA for the API server certificate

Have the API server present a certificate issued by a publicly trusted CA (for example Let’s Encrypt via cert-manager), so that kubeconfigs wouldn’t need to pin a CA certificate.

This doesn’t work on Talos:

  • Talos pins --tls-cert-file/--tls-private-key-file to its own dynamically-issued certificate. Both flags are on the deny list for cluster.apiServer.extraArgs, so the serving certificate can’t be replaced (control_plane_final.go).

  • The only loophole is --tls-sni-cert-key (not denied), which would serve a public certificate for one SNI hostname while keeping the Talos certificate as default. The API server itself would even hot-reload the file, but Talos has no supported path from a renewed certificate (for example a cert-manager Secret) to the host file the static pod mounts, since machine.files only writes static machine-config content under /var. Closing that gap needs a custom DaemonSet writing the certificate into a /var hostPath, which is unsupported and a bootstrap hazard (the API server would depend on a file produced by a workload running on that same API server). OpenShift serves custom API server certificates through this same --tls-sni-cert-key flag, but its kube-apiserver operator manages the certificate delivery and reload that Talos doesn’t (openshift-kube-apiserver flags.go).

  • Even with a public serving certificate, the cluster CA is still required. Kubelets and all in-cluster clients reaching the API server at kubernetes.default.svc validate it against the Talos-managed cluster CA. Client certificate authentication also still requires the cluster CA.

  • A public CA can only issue for public FQDNs, never for the internal IPs and service names that must be in the certificate’s SANs, which is why a single publicly-issued certificate can’t cover the API server’s needs in the first place.

Default per-cluster CA (Talos/CABPT default)

Keep the self-signed per-cluster Kubernetes CA that Talos generates. With Cluster API, CABPT generates the secrets bundle once per cluster and stores it in the management cluster (full bundle in the <cluster>-talos secret, Kubernetes CA in <cluster>-ca). CACPPT mints admin kubeconfigs from it following the standard CAPI contract (<cluster>-kubeconfig secret, clusterctl get kubeconfig).

  • The CA is valid for 10 years. The API server’s serving certificate is issued from it per node with 1 year validity and is automatically rotated by Talos at half-life.

  • Extra SANs, for example an external FQDN for the API endpoint, can be added through a config patch on the TalosControlPlane resource setting cluster.apiServer.certSANs. The control plane endpoint host from the Cluster resource is included as a SAN automatically. Note that changing config patches on an existing TalosControlPlane triggers a rolling replacement of the control plane machines.

  • User kubeconfigs must embed the cluster CA certificate (certificate-authority-data). This matches what all major managed Kubernetes providers (GKE, EKS, AKS) and Cluster API itself do.

  • Rotating the cluster CA (talosctl rotate-ca --kubernetes) is possible but disruptive. It’s a multi-phase rolling operation, and previously distributed kubeconfigs become invalid during it (the serving certificate switches to the new CA mid-rotation, and client certificates issued by the old CA are rejected once the old CA is dropped from cluster.acceptedCAs).

VSHN-managed CA (Vault) with per-cluster intermediate CA

Operate a VSHN root CA (for example in Vault) and issue a per-cluster intermediate CA which is used as the cluster’s Kubernetes CA. This is mechanically possible. The Kubernetes CA in the Talos secrets bundle can be replaced with our own CA, and CABPT reuses a pre-seeded <cluster>-talos secret rather than generating one (the secret must contain a complete secrets bundle in talosctl gen secrets format).

  • The advertised benefit is rotating a cluster’s intermediate CA without users re-fetching the CA certificate, if kubeconfigs pinned the VSHN root instead of the intermediate. This benefit is marginal. Rotating the intermediate still invalidates the client certificates it issued, which is exactly what we want from a rotation (see the Rationale), so users still fetch a new credential. The only thing saved is redistributing the CA certificate, and with OIDC as the primary external auth few users hold long-lived client certs anyway.

  • The intermediate’s private key must be present on every control plane node (Talos issues the API server’s serving certificate from it locally on each node, and the controller manager signs certificate requests with it) and in the management cluster, which weakens the security argument for a hierarchical PKI, since compromising one cluster’s control plane exposes a CA chained to the VSHN root.

  • Only Talos v1.14+ (#13805, merged 2026-07-20, first shipped in v1.14.0-beta.0, no backport to v1.13) ships the issuing CA in the serving certificate chain. On all current stable releases, clients trusting only the root can’t validate the API server.

  • Requires us to run proper CA infrastructure (Vault PKI, key ceremonies, revocation, monitoring) and custom provisioning glue around CABPT, for a trust anchor that’s still private and still has to be distributed through kubeconfig.

Authenticating proxy with a public certificate

Front the API server with a TLS-terminating reverse proxy that holds a publicly trusted certificate and authenticates users (typically through OIDC), then re-originates to the real API server using the cluster’s own credentials. The user’s kubeconfig points at the proxy, contains no certificate-authority-data, and carries no client certificate. The per-cluster Talos CA stays intact underneath.

This is SideroLabs' own answer. Sidero Omni fronts every managed cluster’s API this way (kubectl to the Omni proxy to kube-apiserver), and it’s a well-established pattern elsewhere (kube-oidc-proxy, Rancher’s proxied cluster access, Teleport, Pinniped).

  • It’s the only approach that delivers a "clean," publicly trusted kubeconfig, but it does so with a proxy layer, not by changing the API certificate.

  • Client certificate authentication can’t survive TLS termination, so this only works with bearer-token / OIDC authentication.

  • Omni itself isn’t an option for us. It’s SideroLabs' own management plane (SideroLink-based provisioning and lifecycle) and would replace, not complement, our decision to use Cluster API. Only the proxy pattern is reusable, and we’d have to run and secure our own proxy.

  • This belongs to user authentication and access (OIDC login, separate decision), not to the API certificate decision. It sits on top of whichever CA we choose here.

Decision

We use the default self-signed per-cluster Kubernetes CA generated by the Talos bootstrap provider.

The external FQDN of the API endpoint is added to the API server certificate via a cluster.apiServer.certSANs config patch on the TalosControlPlane resource.

User kubeconfigs embed the cluster CA certificate. Because Kubernetes has no client-certificate revocation, we rotate the cluster CA when we need to invalidate distributed kubeconfigs, and affected users fetch a new kubeconfig afterwards.

Rationale

A publicly trusted certificate can’t be served by the Talos API server itself in any supportable way, and wouldn’t remove the private CA even if it could, since in-cluster trust and client certificate authentication depend on it. A publicly trusted endpoint is achievable, but only by fronting the API with an authenticating proxy (as Sidero Omni does). That’s an access-layer concern for the OIDC login decision. It sits on top of the per-cluster CA decided here and isn’t a substitute for it. Adopting Omni itself is ruled out separately, since it would replace our Cluster API provisioning.

The per-cluster CA is the industry standard. GKE, EKS and AKS all use a self-signed per-cluster CA distributed inside the kubeconfig. (GKE’s newer DNS-based endpoint is an exception. Its front end presents a certificate signed by a public Google CA, so clients validate it through the public trust store instead of the cluster CA, which is the proxy pattern described above rather than a different CA model.) Users and tooling universally handle certificate-authority-data in kubeconfigs, so this requires no explanation beyond "use the kubeconfig we provide."

The VSHN-managed CA hierarchy adds substantial operational load (running Vault PKI and custom CABPT glue) and a real security liability (root-chained key material on every customer control plane), and buys little in return. Its one advantage, rotating a CA without redistributing the CA certificate, is marginal once client certs are break-glass only, and it doesn’t change the fundamental picture that the trust anchor is still private and still shipped in the kubeconfig.

We expect to rotate cluster CAs regularly, not as a rare event. Kubernetes has no client-certificate revocation, so rotating the cluster CA is the only way to invalidate previously distributed admin kubeconfigs. We must do that when a VSHNeer leaves, and likely when a customer’s set of cluster-admins changes. The per-cluster CA supports this directly, since rotation is a self-contained talosctl rotate-ca operation on the cluster itself and affected users re-fetch their kubeconfig afterwards. A VSHN-managed PKI hierarchy would have to be rotated for the same reason, so it saves nothing here while costing far more to run.

Implications for users

  • Kubeconfigs handed out to users contain the cluster CA certificate, since the API server certificate can’t be validated through the system trust store.

  • The API endpoint FQDN and IPs are covered by the certificate’s SANs, so any standard Kubernetes client works out of the box with the provided kubeconfig.

  • When we rotate a cluster’s CA to invalidate distributed credentials, the affected users must fetch a new kubeconfig. Serving certificate rotation (automatic, roughly every 6 months) uses the same CA and is transparent to users.

  • External user authentication will be OIDC, and external non-interactive clients will use service-account tokens (separate decision). Client-certificate kubeconfigs are a break-glass and pre-OIDC path, not the everyday user credential. The CAPI admin kubeconfig uses a 1 year client certificate that CACPPT regenerates at half-life.

References