Recover objects from backup


  • Executables used in this guide:

    • kubectl

    • jq

    • yq yq (version 4 or higher)

    • restic

  • API access to the target cluster

General procedure

  1. Collect configuration for restic

  2. Identify and retrieve restic snapshot

  3. Extract files containing the desired objects and prepare them

  4. Apply objects to the cluster

Collect restic configuration

Restic requires the environment variables RESTIC_REPOSITORY, RESTIC_PASSWORD, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to be set. They can be obtained from the target cluster itself. They can also be obtained from the cluster catalog and Vault.

Obtaining restic configuration from cluster

export RESTIC_REPOSITORY=$(kubectl -n syn-cluster-backup get schedule objects -o jsonpath='s3:{.spec.backend.s3.endpoint}/{.spec.backend.s3.bucket}')
export RESTIC_PASSWORD=$(kubectl --as cluster-admin -n syn-cluster-backup get secret objects-backup-password -o jsonpath='{.data.password}' | base64 --decode)
export AWS_ACCESS_KEY_ID=$(kubectl --as cluster-admin -n syn-cluster-backup get secret objects-backup-s3-credentials -o jsonpath='{.data.username}' | base64 --decode)
export AWS_SECRET_ACCESS_KEY=$(kubectl --as cluster-admin -n syn-cluster-backup get secret objects-backup-s3-credentials -o jsonpath='{.data.password}' | base64 --decode)

Obtaining restic configuration from catalog and vault

  1. Obtain the repository URL of the clusters catalog and export it to REPO_URL

    Get the URL from Alternatively, get it from the Lieutenant API or the Kubernetes API Lieutenant is running on.

    Use the Lieutenant API
    REPO_URL=$(curl -sH "${LIEUTENANT_AUTH}" "https://${LIEUTENANT_URL}/clusters/${CLUSTER_ID}" | jq -r .gitRepo.url)
    Use the Kubernetes API
    REPO_URL=$(kubectl -n ${LIEUTENANT_NS} get cluster -o jsonpath='{.spec.gitRepoURL}' ${CLUSTER_ID})
  2. Download and extract the cluster catalog

    mkdir catalog
    git archive --remote ${REPO_URL} master | tar -xC catalog
  3. Login to vault

    vault login -method=oidc
  4. Export restic configuration

    export RESTIC_REPOSITORY=s3:$(yq -o=json 'select(.kind == "Schedule")| .spec.backend.s3 | .endpoint + "/" + .bucket' catalog/manifests/cluster-backup/10_object.yaml)
    PASSWORD_KEY="$(yq -o=json 'select(.kind == "Secret" and == "objects-backup-password") | .stringData.password' catalog/manifests/cluster-backup/10_object.yaml | cut -d: -f2)"
    export RESTIC_PASSWORD=$(vault kv get -format json "clusters/kv/${PASSWORD_KEY%/*}" | jq -r "${PASSWORD_KEY##*/}")
    ID_KEY="$(yq -o=json 'select(.kind == "Secret" and == "objects-backup-s3-credentials") | .stringData.username' catalog/manifests/cluster-backup/10_object.yaml | cut -d: -f2)"
    export AWS_ACCESS_KEY_ID=$(vault kv get -format json "clusters/kv/${ID_KEY%/*}" | jq -r "${ID_KEY##*/}")
    SECRET_KEY="$(yq -o=json 'select(.kind == "Secret" and == "objects-backup-s3-credentials") | .stringData.password' catalog/manifests/cluster-backup/10_object.yaml | cut -d: -f2)"
    export AWS_SECRET_ACCESS_KEY=$(vault kv get -format json "clusters/kv/${SECRET_KEY%/*}" | jq -r "${SECRET_KEY##*/}")

Identify and retrieve snapshot

  1. List the available snapshots. Identify the one you do want to restore. Take note of its ID.

    restic snapshots
  2. Retrieve the backup archive

    restic restore <ID> --target .

Extract and prepare files

  1. List files in the backup. Take note of the path containing the required files.

    tar tvf syn-cluster-backup-object-dumper.tar.gz
  2. Extract required files. If all files should be extracted, path/inside/archive can be omitted. Files will be put in the directory restore within the current working directory.

    mkdir restore
    tar -C restore -xf syn-cluster-backup-object-dumper.tar.gz [path/inside/archive]
  3. Prepare files

    Depending on the restore requirements, the extracted files need to be altered before they can be applied to the cluster.

Apply objects

Apply the extracted and prepared objects to the target cluster.

Apply single file
kubectl --as cluster-admin apply -f <path/to/file>
Apply all files within a directory
kubectl --as cluster-admin apply -Rf <path/to/dir>

While it’s technically possible to restore all objects from a backup it’s not advisable to do so. Restoring objects that are managed by ArgoCD will prevent most ArgoCD-managed apps from successfully syncing. These objects would then need to be replaced manually. Try to limit the objects being restored to the necessary minimum.