Recover cluster objects from Azure Blob Storage backup

Prerequisites

Executables used in this guide:
- kubectl
- yq YAML processor (Version 4 or higher)
- restic
API access to the target cluster

General procedure

Collect configuration for restic
Identify and retrieve restic snapshot
Extract files containing the desired objects and prepare them
Apply objects to the cluster

Collect restic configuration

Restic requires the environment variables RESTIC_REPOSITORY, RESTIC_PASSWORD, AZURE_ACCOUNT_NAME and AZURE_ACCOUNT_KEY to be set. They can be obtained from the target cluster itself. They can also be obtained from the cluster catalog and Vault.

Obtaining restic configuration from cluster

export RESTIC_REPOSITORY=$(kubectl -n syn-cluster-backup get schedule objects -o jsonpath='azure:{.spec.backend.s3.bucket}:/')
export RESTIC_PASSWORD=$(kubectl -n syn-cluster-backup get secret objects-backup-password -o jsonpath='{.data.password}' | base64 --decode)
export AZURE_ACCOUNT_NAME=$(kubectl get secret admin-credentials-minio-k8up -n syn-minio -o jsonpath='{.data.accesskey}' | base64 --decode)
export AZURE_ACCOUNT_KEY=$(kubectl get secret admin-credentials-minio-k8up -n syn-minio -o jsonpath='{.data.secretkey}' | base64 --decode)

Obtaining restic configuration from catalog and vault

Obtain the repository URL of the cluster’s catalog and export it to REPO_URL

Get the URL from control.vshn.net/syn/lieutenantclusters. Alternatively, get it from the Lieutenant API or the Kubernetes API Lieutenant is running on.
Use the Lieutenant API
```
REPO_URL=$(curl -sH "${LIEUTENANT_AUTH}" "https://${LIEUTENANT_URL}/clusters/${CLUSTER_ID}" | jq -r .gitRepo.url)
```
Use the Kubernetes API
```
REPO_URL=$(kubectl -n ${LIEUTENANT_NS} get cluster -o jsonpath='{.spec.gitRepoURL}' ${CLUSTER_ID})
```

Download and extract the cluster catalog

mkdir catalog
git archive --remote ${REPO_URL} master | tar -xC catalog

export VAULT_ADDR=https://vault-prod.syn.vshn.net
vault login -method=ldap username=<your username>

Export restic configuration

export RESTIC_REPOSITORY=azure:$(yq e '. | select(.kind == "Schedule").spec.backend.s3.bucket' catalog/manifests/cluster-backup/05_schedule.yaml):/

PASSWORD_KEY="$(yq e '. | select(.kind == "Secret" and .metadata.name == "objects-backup-password").stringData.password' catalog/manifests/cluster-backup/05_schedule.yaml | cut -d: -f2)"
export RESTIC_PASSWORD=$(vault kv get -format json "clusters/kv/${PASSWORD_KEY%/*}" | jq -r ".data.data.${PASSWORD_KEY##*/}")

ID_KEY="$(yq e '. | select(.kind == "Secret" and .metadata.name == "object-backup-s3-credentials").stringData.username' catalog/manifests/cluster-backup/05_schedule.yaml | cut -d: -f2)"
export AZURE_ACCOUNT_NAME=$(vault kv get -format json "clusters/kv/${ID_KEY%/*}" | jq -r ".data.data.${ID_KEY##*/}")

SECRET_KEY="$(yq e '. | select(.kind == "Secret" and .metadata.name == "object-backup-s3-credentials").stringData.password' catalog/manifests/cluster-backup/05_schedule.yaml | cut -d: -f2)"
export AZURE_ACCOUNT_KEY=$(vault kv get -format json "clusters/kv/${SECRET_KEY%/*}" | jq -r ".data.data.${SECRET_KEY##*/}")

Identify and retrieve snapshot

List the available snapshots. Identify the one you do want to restore. Take note of its ID.
```
restic snapshots
```

Retrieve the backup archive

restic restore <ID> --target cluster-backup-object-restore-$(date +%F)

Extract and prepare files

Change to the restore directory

cd cluster-backup-object-restore-$(date +%F)

List files in the backup. Take note of the path containing the required files.
```
tar tvf syn-cluster-backup-object-dumper.tar.gz
```
Extract required files. If all files should be extracted, path/inside/archive can be omitted. Files will be put in the directory restore within the current working directory.
```
mkdir restore
tar -C restore -xf syn-cluster-backup-object-dumper.tar.gz [path/inside/archive]
```
Prepare files

Depending on the restore requirements, the extracted files need to be altered before they can be applied to the cluster.

Apply objects

Apply the extracted and prepared objects to the target cluster.

Apply single file

kubectl --as cluster-admin apply -f <path/to/file>

Apply all files within a directory

kubectl --as cluster-admin apply -Rf <path/to/dir>