Uninstallation on Exoscale

Steps to remove an OpenShift 4 cluster from Exoscale.

  • The commands are idempotent and can be retried if any of the steps fail.

  • In the future, this procedure will be mostly automated

Always follow the 4-eye data deletion (Internal link) principle when decommissioning productive clusters.


Cluster Decommission

  1. Export the following vars

    export EXOSCALE_ACCOUNT=<exoscale-account>
    export EXOSCALE_API_KEY=<exoscale-key>
    export EXOSCALE_API_SECRET=<exoscale-secret>
    export EXOSCALE_REGION=<cluster-region>
    export CLUSTER_ID=<cluster-name>
    # From https://git.vshn.net/-/profile/personal_access_tokens
    export GITLAB_TOKEN=<gitlab-api-token>
    export GITLAB_USER=<gitlab-user-name>
    # For example: https://api.syn.vshn.net
    # IMPORTANT: do NOT add a trailing `/`. Commands below will fail.
    export COMMODORE_API_URL=<lieutenant-api-endpoint>
    export TF_VAR_lb_exoscale_api_key=irrelevant
    export TF_VAR_lb_exoscale_api_secret=irrelevant
  2. Compile cluster catalog to get input variables

    commodore catalog compile ${CLUSTER_ID}
  3. Configure Terraform secrets

    cat <<EOF > catalog/manifests/openshift4-terraform/.env
  4. Setup Terraform

    Prepare Terraform execution environment
    # Set terraform image and tag to be used
      yq eval ".parameters.openshift4_terraform.images.terraform.image" \
      yq eval ".parameters.openshift4_terraform.images.terraform.tag" \
    # Generate the terraform alias
    alias terraform='docker run -it --rm \
      -e REAL_UID=$(id -u) \
      --env-file ${base_dir}/terraform.env \
      -w /tf \
      -v $(pwd):/tf \
      --ulimit memlock=-1 \
      "${tf_image}:${tf_tag}" /tf/terraform.sh'
    export GITLAB_REPOSITORY_URL=$(curl -sH "Authorization: Bearer $(commodore fetch-token)" ${COMMODORE_API_URL}/clusters/${CLUSTER_ID} | jq -r '.gitRepo.url' | sed 's|ssh://||; s|/|:|')
    export GITLAB_CATALOG_PROJECT_ID=$(curl -sH "Authorization: Bearer ${GITLAB_TOKEN}" "https://git.vshn.net/api/v4/projects?simple=true&search=${GITLAB_REPOSITORY_NAME/.git}" | jq -r ".[] | select(.ssh_url_to_repo == \"${GITLAB_REPOSITORY_URL}\") | .id")
    export GITLAB_STATE_URL="https://git.vshn.net/api/v4/projects/${GITLAB_CATALOG_PROJECT_ID}/terraform/state/cluster"
    pushd catalog/manifests/openshift4-terraform/
    Initialize Terraform
    terraform init \
      "-backend-config=address=${GITLAB_STATE_URL}" \
      "-backend-config=lock_address=${GITLAB_STATE_URL}/lock" \
      "-backend-config=unlock_address=${GITLAB_STATE_URL}/lock" \
      "-backend-config=username=${GITLAB_USER}" \
      "-backend-config=password=${GITLAB_TOKEN}" \
      "-backend-config=lock_method=POST" \
      "-backend-config=unlock_method=DELETE" \
  5. Grab location of LB backups and potential Icinga2 satellite host before decommissioning VMs.

    declare -a LB_FQDNS
    for id in 1 2; do
      LB_FQDNS[$id]=$(terraform state show "module.cluster.module.lb.exoscale_domain_record.lb[$(expr $id - 1)]" | grep hostname | cut -d'=' -f2 | tr -d ' "\r\n')
    for lb in ${LB_FQDNS[*]}; do
      ssh "${lb}" "sudo grep 'server =' /etc/burp/burp.conf && sudo grep 'ParentZone' /etc/icinga2/constants.conf"
  6. Set downtimes for both LBs in Icinga2.

  7. Remove APPUiO hieradata Git repository resource from Terraform state

    terraform state rm 'module.cluster.module.lb.module.hiera[0].gitfile_checkout.appuio_hieradata'
    This step is necessary to ensure the subsequent terraform destroy completes without errors.
  8. Delete resources using Terraform

    terraform destroy
  9. Use Exoscale CLI tool to empty and remove buckets

    mkdir -p ~/.config/exoscale
    cat <<EOF >> ~/.config/exoscale/exoscale.toml
      account = "${EXOSCALE_ACCOUNT}"
      defaultZone = "${EXOSCALE_REGION}"
      endpoint = "https://api.exoscale.ch/v1"
      name = "${CLUSTER_ID}"
    # Bootstrap bucket
    exo storage delete -r -f "sos://${CLUSTER_ID}-bootstrap/"
    exo storage rb -f "${CLUSTER_ID}-bootstrap"
    # OpenShift Image Registry bucket
    exo storage delete -r -f "sos://${CLUSTER_ID}-image-registry/"
    exo storage rb -f "${CLUSTER_ID}-image-registry"
  10. Decommission Puppet-managed LBs according to the VSHN documentation (Internal link).

    Don’t forget to remove the LB configuration in the APPUiO hieradata and the nodes hieradata.
  11. Remove cluster DNS records from VSHN DNS

  12. Remove cluster IPs from LDAP allowlist, if applicable

  13. Decommission cluster in Project Syn