Emergency credentials

Problem

We need to decide on an emergency credentials mechanism for VSHN Managed Talos.

Goals

Define emergency credentials mechanism
Only VSHNeers need to be able to use emergency credentials

Non-Goals

Define regular user authentication

Proposals

Cluster API admin kubeconfig

The first obvious option is to just store the admin kubeconfig generated by Cluster API in Passbolt. However, the cluster API admin kubeconfig has a hard-coded lifetime of one year. Additionally, because Kubernetes doesn’t support client certificate revocation, the only way to invalidate an admin kubeconfig client certificate is to rotate the cluster’s CA certificate.

However, there’s also no way to completely prevent users which have cluster-admin level permissions from accessing the admin kubeconfig which is stored in a secret on the cluster, because every cluster is its own CAPI management cluster.

APPUiO emergency-credentials-controller

The other obvious option is to use the APPUiO emergency-credentials-controller. This is the same solution that we use for VSHN Managed OpenShift, and retrieving credentials managed by the emergency-credentials-controller is tightly integrated into other VSHN tooling, such as Kharon.

Using the emergency-credentials-controller gives us a well-designed emergency credentials mechanism that automatically rotates all emergency credentials frequently, which makes offboarding VSHNeers straightforward.

Manually managed emergency client certificate or token

We could manually manage emergency client certificates or tokens for each VSHN Managed Talos cluster. However, given the other options that are available, this option isn’t attractive at all.

Decision

We will use the APPUiO emergency-credentials-controller to generate emergency access tokens for VSHN Managed Talos clusters.

Since we can’t fully deny access to the admin kubeconfig managed by cluster API, we’ll still need to make sure that VSHNeer offboarding somehow triggers a CA certificate rotation for all VSHN Managed Talos clusters.

Rationale

By using the emergency-credentials-controller, and configuring the same S3 buckets and list of Passbolt PGP public keys as we use for VSHN Managed OpenShift, we enable VSHNeers to use the same emergency credentials client (Kharon) for VSHN Managed OpenShift and VSHN Managed Talos.

This is a big operational win, since on-call engineers only need a single tool (which is already part of VSHN’s established on-call process) to gain emergency access to all VSHN Managed clusters.