Uninstallation on Exoscale

Steps to remove an OpenShift 4 cluster from Exoscale.

The commands are idempotent and can be retried if any of the steps fail.
In the future, this procedure will be mostly automated

Always follow the 4-eye data deletion (Internal link) principle when decommissioning productive clusters.

Prerequisites

Exoscale API key
docker
jq
yq yq YAML processor (version 4 or higher)
exo >= v1.28.0, Exoscale CLI

Cluster Decommission

Create a new API key with role unrestricted for the decommissioning

Export the following vars

export EXOSCALE_ACCOUNT=<exoscale-account>
export EXOSCALE_API_KEY=<exoscale-key>
export EXOSCALE_API_SECRET=<exoscale-secret>
export EXOSCALE_ZONE=<cluster-region>  # e.g. ch-gva-2

export CLUSTER_ID=<cluster-name>

# From https://git.vshn.net/-/user_settings/personal_access_tokens
export GITLAB_TOKEN=<gitlab-api-token>
export GITLAB_USER=<gitlab-user-name>

# For example: https://api.syn.vshn.net
# IMPORTANT: do NOT add a trailing `/`. Commands below will fail.
export COMMODORE_API_URL=<lieutenant-api-endpoint>

Compile cluster catalog to get input variables
```
commodore catalog compile ${CLUSTER_ID}
```

Configure Terraform secrets

cat <<EOF > catalog/manifests/openshift4-terraform/terraform.env
EXOSCALE_API_KEY
EXOSCALE_API_SECRET
EOF

Setup Terraform

Prepare Terraform execution environment

# Set terraform image and tag to be used
tf_image=$(\
  yq eval ".parameters.openshift4_terraform.images.terraform.image" \
  dependencies/openshift4-terraform/class/defaults.yml)
tf_tag=$(\
  yq eval ".parameters.openshift4_terraform.images.terraform.tag" \
  dependencies/openshift4-terraform/class/defaults.yml)

# Generate the terraform alias
base_dir=$(pwd)
alias terraform='touch .terraformrc; docker run -it --rm \
  -e REAL_UID=$(id -u) \
  -e TF_CLI_CONFIG_FILE=/tf/.terraformrc \
  --env-file ${base_dir}/terraform.env \
  -w /tf \
  -v $(pwd):/tf \
  --ulimit memlock=-1 \
  "${tf_image}:${tf_tag}" /tf/terraform.sh'

export GITLAB_REPOSITORY_URL=$(curl -sH "Authorization: Bearer $(commodore fetch-token)" ${COMMODORE_API_URL}/clusters/${CLUSTER_ID} | jq -r '.gitRepo.url' | sed 's|ssh://||; s|/|:|')
export GITLAB_REPOSITORY_NAME=${GITLAB_REPOSITORY_URL##*/}
export GITLAB_CATALOG_PROJECT_ID=$(curl -sH "Authorization: Bearer ${GITLAB_TOKEN}" "https://git.vshn.net/api/v4/projects?simple=true&search=${GITLAB_REPOSITORY_NAME/.git}" | jq -r ".[] | select(.ssh_url_to_repo == \"${GITLAB_REPOSITORY_URL}\") | .id")
export GITLAB_STATE_URL="https://git.vshn.net/api/v4/projects/${GITLAB_CATALOG_PROJECT_ID}/terraform/state/cluster"

pushd catalog/manifests/openshift4-terraform/

Initialize Terraform

terraform init \
  "-backend-config=address=${GITLAB_STATE_URL}" \
  "-backend-config=lock_address=${GITLAB_STATE_URL}/lock" \
  "-backend-config=unlock_address=${GITLAB_STATE_URL}/lock" \
  "-backend-config=username=${GITLAB_USER}" \
  "-backend-config=password=${GITLAB_TOKEN}" \
  "-backend-config=lock_method=POST" \
  "-backend-config=unlock_method=DELETE" \
  "-backend-config=retry_wait_min=5"

Grab location of LB backups and potential Icinga2 satellite host before decommissioning VMs.

declare -a LB_FQDNS
for id in 1 2; do
  LB_FQDNS[$id]=$(terraform state show "module.cluster.module.lb.exoscale_domain_record.lb[$(expr $id - 1)]" | grep hostname | cut -d'=' -f2 | tr -d ' "\r\n')
done
for lb in ${LB_FQDNS[*]}; do
  ssh "${lb}" "sudo grep 'server =' /etc/burp/burp.conf && sudo grep 'ParentZone' /etc/icinga2/constants.conf"
done

Set downtimes for both LBs in Icinga2.
Remove APPUiO hieradata Git repository resource from Terraform state
```
terraform state rm 'module.cluster.module.lb.module.hiera[0].gitfile_checkout.appuio_hieradata'
```
This step is necessary to ensure the subsequent terraform destroy completes without errors.
Delete resources using Terraform
```
terraform destroy
```

Use Exoscale CLI tool to empty and remove buckets

mkdir -p ~/.config/exoscale
cat <<EOF >> ~/.config/exoscale/exoscale.toml

[[accounts]]
  account = "${EXOSCALE_ACCOUNT}"
  defaultZone = "${EXOSCALE_ZONE}"
  endpoint = "https://api.exoscale.ch/v1"
  name = "${CLUSTER_ID}"
EOF

# Bootstrap bucket
exo storage delete -r -f "sos://${CLUSTER_ID}-bootstrap/"
exo storage rb -f "${CLUSTER_ID}-bootstrap"
# OpenShift Image Registry bucket
exo storage delete -r -f "sos://${CLUSTER_ID}-image-registry/"
exo storage rb -f "${CLUSTER_ID}-image-registry"

Delete the cluster-backup bucket

Verify that the cluster backups aren’t needed anymore before cleaning up the backup bucket. Consider extracting the most recent cluster objects and etcd backups before deleting the bucket. See the Recover objects from backup how-to for instructions. At this point in the decommissioning process, you’ll have to extract the Restic configuration from Vault instead of the cluster itself.

exo storage delete -r -f "sos://${CLUSTER_ID}-cluster-backup/"
exo storage rb -f "${CLUSTER_ID}-cluster-backup"

Delete the cluster’s API keys and the API key created for decommissioning
Decommission Puppet-managed LBs according to the VSHN documentation (Internal link).

Don’t forget to remove the LB configuration in the APPUiO hieradata and the nodes hieradata.
Remove cluster DNS records from VSHN DNS
Decommission cluster in Project Syn