Uninstallation on cloudscale.ch

Steps to remove an OpenShift 4 cluster from cloudscale.ch.

  • The commands are idempotent and can be retried if any of the steps fail.

  • In the future, this procedure will be mostly automated


Cluster Decommission

  1. Export the following vars

    export CLOUDSCALE_TOKEN=<cloudscale-api-token> # From https://control.cloudscale.ch/user/api-tokens
    export CLUSTER_ID=<lieutenant-cluster-id>
    export TENANT_ID=<lieutenant-tenant-id>
    export REGION=<region> # rma or lpg (without the zone number)
    export GITLAB_TOKEN=<gitlab-api-token> # From https://git.vshn.net/profile/personal_access_tokens
    export GITLAB_USER=<gitlab-user-name>
  2. Setup Terraform

    Prepare Terraform execution environment
    # Set terraform image and tag to be used
      yq eval ".parameters.openshift4_terraform.images.terraform.image" \
      yq eval ".parameters.openshift4_terraform.images.terraform.tag" \
    # Generate the terraform alias
    alias terraform='docker run -it --rm \
      -e REAL_UID=$(id -u) \
      --env-file ${base_dir}/terraform.env \
      -w /tf \
      -v $(pwd):/tf \
      --ulimit memlock=-1 \
      "${tf_image}:${tf_tag}" /tf/terraform.sh'
    export GITLAB_REPOSITORY_URL=$(curl -sH "Authorization: Bearer ${COMMODORE_API_TOKEN}" ${COMMODORE_API_URL}/clusters/${CLUSTER_ID} | jq -r '.gitRepo.url' | sed 's|ssh://||; s|/|:|')
    export GITLAB_CATALOG_PROJECT_ID=$(curl -sH "Authorization: Bearer ${GITLAB_TOKEN}" "https://git.vshn.net/api/v4/projects?simple=true&search=${GITLAB_REPOSITORY_NAME/.git}" | jq -r ".[] | select(.ssh_url_to_repo == \"${GITLAB_REPOSITORY_URL}\") | .id")
    export GITLAB_STATE_URL="https://git.vshn.net/api/v4/projects/${GITLAB_CATALOG_PROJECT_ID}/terraform/state/cluster"
    pushd catalog/manifests/openshift4-terraform/
    Initialize Terraform
    terraform init \
      "-backend-config=address=${GITLAB_STATE_URL}" \
      "-backend-config=lock_address=${GITLAB_STATE_URL}/lock" \
      "-backend-config=unlock_address=${GITLAB_STATE_URL}/lock" \
      "-backend-config=username=${GITLAB_USER}" \
      "-backend-config=password=${GITLAB_TOKEN}" \
      "-backend-config=lock_method=POST" \
      "-backend-config=unlock_method=DELETE" \
  3. Delete resources from clouscale.ch using Terraform

    # The first time it will fail
    terraform destroy
    # Destroy a second time to delete private networks
    terraform destroy
  4. After all resources are deleted we need to remove the bucket

    # Use already exiting bucket user
    response=$(curl -sH "Authorization: Bearer ${CLOUDSCALE_TOKEN}" \
      https://api.cloudscale.ch/v1/objects-users | \
      jq -e ".[] | select(.display_name == \"${CLUSTER_ID}\")")
    # configure minio client to use the bucket
    mc config host add \
      "${CLUSTER_ID}" "https://objects.${REGION}.cloudscale.ch" \
      $(echo $response | jq -r '.keys[0].access_key') \
      $(echo $response | jq -r '.keys[0].secret_key')
    # delete bootstrap-ignition object
    mc rb "${CLUSTER_ID}/${CLUSTER_ID}-bootstrap-ignition" --force
    # delete image-registry object
    mc rb "${CLUSTER_ID}/${CLUSTER_ID}-image-registry" --force
    # delete cloudscale.ch user object
    curl -i -H "Authorization: Bearer ${CLOUDSCALE_TOKEN}" -X DELETE $(echo $response | jq -r '.href')
  5. Delete vault entries:

    # Vault login
    export VAULT_ADDR=https://vault-prod.syn.vshn.net
    vault login -method=ldap username=<your.name>
    # delete token secret
    vault kv delete clusters/kv/${TENANT_ID}/${CLUSTER_ID}/cloudscale
    # delete registry secret
    vault kv delete clusters/kv/${TENANT_ID}/${CLUSTER_ID}/registry
    # delete ldap secret
    vault kv delete clusters/kv/${TENANT_ID}/${CLUSTER_ID}/vshn-ldap
  6. Delete cluster from Lieutenant API (via portal)

    • Select the Lieutenant API Endpoint

    • Search cluster name

    • Delete cluster entry using the delete button

  7. Delete LDAP service (via portal)

    • Search cluster name

    • Delete cluster entry service using the delete button

  8. Remove IPs from LDAP allowlist

    • Search cluster IPs and remove those lines and any comments related.

    • Create a Merge Request and invite a colleague for a review/approve/merge

  9. Delete all DNS records related with cluster (zonefiles)

  10. Update any related documentation