Recover cluster objects from Azure Blob Storage backup


  • Executables used in this guide:

  • API access to the target cluster

General procedure

  1. Collect configuration for restic

  2. Identify and retrieve restic snapshot

  3. Extract files containing the desired objects and prepare them

  4. Apply objects to the cluster

Collect restic configuration

Restic requires the environment variables RESTIC_REPOSITORY, RESTIC_PASSWORD, AZURE_ACCOUNT_NAME and AZURE_ACCOUNT_KEY to be set. They can be obtained from the target cluster itself. They can also be obtained from the cluster catalog and Vault.

Obtaining restic configuration from cluster

export RESTIC_REPOSITORY=$(kubectl -n syn-cluster-backup get schedule objects -o jsonpath='azure:{.spec.backend.s3.bucket}:/')
export RESTIC_PASSWORD=$(kubectl -n syn-cluster-backup get secret objects-backup-password -o jsonpath='{.data.password}' | base64 --decode)
export AZURE_ACCOUNT_NAME=$(kubectl get secret admin-credentials-minio-k8up -n syn-minio -o jsonpath='{.data.accesskey}' | base64 --decode)
export AZURE_ACCOUNT_KEY=$(kubectl get secret admin-credentials-minio-k8up -n syn-minio -o jsonpath='{.data.secretkey}' | base64 --decode)

Obtaining restic configuration from catalog and vault

  1. Obtain the repository URL of the cluster’s catalog and export it to REPO_URL

    Get the URL from Alternatively, get it from the Lieutenant API or the Kubernetes API Lieutenant is running on.

    Use the Lieutenant API
    REPO_URL=$(curl -sH "${LIEUTENANT_AUTH}" "https://${LIEUTENANT_URL}/clusters/${CLUSTER_ID}" | jq -r .gitRepo.url)
    Use the Kubernetes API
    REPO_URL=$(kubectl -n ${LIEUTENANT_NS} get cluster -o jsonpath='{.spec.gitRepoURL}' ${CLUSTER_ID})
  2. Download and extract the cluster catalog

    mkdir catalog
    git archive --remote ${REPO_URL} master | tar -xC catalog
  3. Login to Vault

    export VAULT_ADDR=
    vault login -method=ldap username=<your username>
  4. Export restic configuration

    export RESTIC_REPOSITORY=azure:$(yq e '. | select(.kind == "Schedule").spec.backend.s3.bucket' catalog/manifests/cluster-backup/05_schedule.yaml):/
    PASSWORD_KEY="$(yq e '. | select(.kind == "Secret" and == "objects-backup-password").stringData.password' catalog/manifests/cluster-backup/05_schedule.yaml | cut -d: -f2)"
    export RESTIC_PASSWORD=$(vault kv get -format json "clusters/kv/${PASSWORD_KEY%/*}" | jq -r "${PASSWORD_KEY##*/}")
    ID_KEY="$(yq e '. | select(.kind == "Secret" and == "object-backup-s3-credentials").stringData.username' catalog/manifests/cluster-backup/05_schedule.yaml | cut -d: -f2)"
    export AZURE_ACCOUNT_NAME=$(vault kv get -format json "clusters/kv/${ID_KEY%/*}" | jq -r "${ID_KEY##*/}")
    SECRET_KEY="$(yq e '. | select(.kind == "Secret" and == "object-backup-s3-credentials").stringData.password' catalog/manifests/cluster-backup/05_schedule.yaml | cut -d: -f2)"
    export AZURE_ACCOUNT_KEY=$(vault kv get -format json "clusters/kv/${SECRET_KEY%/*}" | jq -r "${SECRET_KEY##*/}")

Identify and retrieve snapshot

  1. List the available snapshots. Identify the one you do want to restore. Take note of its ID.

    restic snapshots
  2. Retrieve the backup archive

    restic restore <ID> --target cluster-backup-object-restore-$(date +%F)

Extract and prepare files

  1. Change to the restore directory

    cd cluster-backup-object-restore-$(date +%F)
  2. List files in the backup. Take note of the path containing the required files.

    tar tvf syn-cluster-backup-object-dumper.tar.gz
  3. Extract required files. If all files should be extracted, path/inside/archive can be omitted. Files will be put in the directory restore within the current working directory.

    mkdir restore
    tar -C restore -xf syn-cluster-backup-object-dumper.tar.gz [path/inside/archive]
  4. Prepare files

    Depending on the restore requirements, the extracted files need to be altered before they can be applied to the cluster.

Apply objects

Apply the extracted and prepared objects to the target cluster.

Apply single file
kubectl --as cluster-admin apply -f <path/to/file>
Apply all files within a directory
kubectl --as cluster-admin apply -Rf <path/to/dir>