Recover objects from backup

Prerequisites

Executables used in this guide:
- kubectl
- jq
- yq yq (version 4 or higher)
- restic
API access to the target cluster

General procedure

Collect configuration for restic
Identify and retrieve restic snapshot
Extract files containing the desired objects and prepare them
Apply objects to the cluster

Collect restic configuration

Restic requires the environment variables RESTIC_REPOSITORY, RESTIC_PASSWORD, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to be set. They can be obtained from the target cluster itself. They can also be obtained from the cluster catalog and Vault.

Obtaining restic configuration from cluster

export RESTIC_REPOSITORY=$(kubectl -n syn-cluster-backup get schedule objects -o jsonpath='s3:{.spec.backend.s3.endpoint}/{.spec.backend.s3.bucket}')
export RESTIC_PASSWORD=$(kubectl --as cluster-admin -n syn-cluster-backup get secret objects-backup-password -o jsonpath='{.data.password}' | base64 --decode)
export AWS_ACCESS_KEY_ID=$(kubectl --as cluster-admin -n syn-cluster-backup get secret objects-backup-s3-credentials -o jsonpath='{.data.username}' | base64 --decode)
export AWS_SECRET_ACCESS_KEY=$(kubectl --as cluster-admin -n syn-cluster-backup get secret objects-backup-s3-credentials -o jsonpath='{.data.password}' | base64 --decode)

Obtaining restic configuration from catalog and vault

Obtain the repository URL of the clusters catalog and export it to REPO_URL

Get the URL from control.vshn.net/syn/lieutenantclusters. Alternatively, get it from the Lieutenant API or the Kubernetes API Lieutenant is running on.
Use the Lieutenant API
```
REPO_URL=$(curl -sH "${LIEUTENANT_AUTH}" "https://${LIEUTENANT_URL}/clusters/${CLUSTER_ID}" | jq -r .gitRepo.url)
```
Use the Kubernetes API
```
REPO_URL=$(kubectl -n ${LIEUTENANT_NS} get cluster -o jsonpath='{.spec.gitRepoURL}' ${CLUSTER_ID})
```

Download and extract the cluster catalog

mkdir catalog
git archive --remote ${REPO_URL} master | tar -xC catalog

Export restic configuration

export RESTIC_REPOSITORY=s3:$(yq -o=json 'select(.kind == "Schedule")| .spec.backend.s3 | .endpoint + "/" + .bucket' catalog/manifests/cluster-backup/10_object.yaml)

PASSWORD_KEY="$(yq -o=json 'select(.kind == "Secret" and .metadata.name == "objects-backup-password") | .stringData.password' catalog/manifests/cluster-backup/10_object.yaml | cut -d: -f2)"
export RESTIC_PASSWORD=$(vault kv get -format json "clusters/kv/${PASSWORD_KEY%/*}" | jq -r ".data.data.${PASSWORD_KEY##*/}")

ID_KEY="$(yq -o=json 'select(.kind == "Secret" and .metadata.name == "objects-backup-s3-credentials") | .stringData.username' catalog/manifests/cluster-backup/10_object.yaml | cut -d: -f2)"
export AWS_ACCESS_KEY_ID=$(vault kv get -format json "clusters/kv/${ID_KEY%/*}" | jq -r ".data.data.${ID_KEY##*/}")

SECRET_KEY="$(yq -o=json 'select(.kind == "Secret" and .metadata.name == "objects-backup-s3-credentials") | .stringData.password' catalog/manifests/cluster-backup/10_object.yaml | cut -d: -f2)"
export AWS_SECRET_ACCESS_KEY=$(vault kv get -format json "clusters/kv/${SECRET_KEY%/*}" | jq -r ".data.data.${SECRET_KEY##*/}")

Identify and retrieve snapshot

List the available snapshots. Identify the one you do want to restore. Take note of its ID.
```
restic snapshots
```
Retrieve the backup archive
```
restic restore <ID> --target .
```

Extract and prepare files

List files in the backup. Take note of the path containing the required files.
```
tar tvf syn-cluster-backup-object-dumper.tar.gz
```
Extract required files. If all files should be extracted, path/inside/archive can be omitted. Files will be put in the directory restore within the current working directory.
```
mkdir restore
tar -C restore -xf syn-cluster-backup-object-dumper.tar.gz [path/inside/archive]
```
Prepare files

Depending on the restore requirements, the extracted files need to be altered before they can be applied to the cluster.

Apply objects

Apply the extracted and prepared objects to the target cluster.

Apply single file

kubectl --as cluster-admin apply -f <path/to/file>

Apply all files within a directory

kubectl --as cluster-admin apply -Rf <path/to/dir>

While it’s technically possible to restore all objects from a backup it’s not advisable to do so. Restoring objects that are managed by ArgoCD will prevent most ArgoCD-managed apps from successfully syncing. These objects would then need to be replaced manually. Try to limit the objects being restored to the necessary minimum.