Recover cluster objects from EKS backup

Prerequisites

Executables used in this guide:
- kubectl
- yq YAML processor (Version 4 or higher)
- restic
API access to the target cluster

General procedure

Collect configuration for restic
Identify and retrieve restic snapshot
Extract files containing the desired objects and prepare them
Apply objects to the cluster

Collect restic configuration

For certain clusters, public access to the S3 bucket is blocked. For such clusters, all the following steps need to be done from within the AWS account (e.g. jumphost) or the cluster itself.

Restic requires the environment variables RESTIC_REPOSITORY, RESTIC_PASSWORD, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to be set. They can be obtained from the target cluster itself. They can also be obtained from the cluster catalog and Vault.

Obtaining restic configuration from cluster

  export RESTIC_REPOSITORY=$(kubectl -n syn-cluster-backup get schedule objects -o jsonpath='s3:{.spec.backend.s3.endpoint}/{.spec.backend.s3.bucket}')
  export RESTIC_PASSWORD=$(kubectl -n syn-cluster-backup get secret objects-backup-password -o jsonpath='{.data.password}' | base64 --decode)
  export AWS_ACCESS_KEY_ID=$(kubectl -n syn-cluster-backup get secret objects-backup-s3-credentials -o jsonpath='{.data.username}' | base64 --decode)
  export AWS_SECRET_ACCESS_KEY=$(kubectl -n syn-cluster-backup get secret objects-backup-s3-credentials -o jsonpath='{.data.password}' | base64 --decode)

Obtaining restic configuration from catalog and vault

Obtain the repository URL of the cluster’s catalog and export it to REPO_URL

Get the URL from control.vshn.net/syn/lieutenantclusters. Alternatively, get it from the Lieutenant API or the Kubernetes API Lieutenant is running on.
Use the Lieutenant API
```
REPO_URL=$(curl -sH "${LIEUTENANT_AUTH}" "https://${LIEUTENANT_URL}/clusters/${CLUSTER_ID}" | jq -r .gitRepo.url)
```
Use the Kubernetes API
```
REPO_URL=$(kubectl -n ${LIEUTENANT_NS} get cluster -o jsonpath='{.spec.gitRepoURL}' ${CLUSTER_ID})
```

Download and extract the cluster catalog

mkdir catalog
git archive --remote ${REPO_URL} master | tar -xC catalog

export VAULT_ADDR=https://vault-prod.syn.vshn.net
vault login -method=ldap username=<your username>

Export restic configuration

export RESTIC_REPOSITORY=s3:$(yq read -cjd'*' catalog/manifests/cluster-backup/05_schedule.yaml | jq -r '.[] | select(.kind == "Schedule") | .spec.backend.s3 | .endpoint + "/" + .bucket')

PASSWORD_KEY="$(cat catalog/manifests/cluster-backup/* | yq read -cjd'*' - | jq -r '.[] | select(.kind == "Secret" and .metadata.name == "objects-backup-password") | .stringData.password' | cut -d: -f2)"
export RESTIC_PASSWORD=$(vault kv get -format json "clusters/kv/${PASSWORD_KEY%/*}" | jq -r ".data.data.${PASSWORD_KEY##*/}")

ID_KEY="$(cat catalog/manifests/cluster-backup/* | yq read -cjd'*' - | jq -r '.[] | select(.kind == "Secret" and .metadata.name == "object-backup-s3-credentials") | .stringData.username' | cut -d: -f2)"
export AWS_ACCESS_KEY_ID=$(vault kv get -format json "clusters/kv/${ID_KEY%/*}" | jq -r ".data.data.${ID_KEY##*/}")

SECRET_KEY="$(cat catalog/manifests/cluster-backup/* | yq read -cjd'*' - | jq -r '.[] | select(.kind == "Secret" and .metadata.name == "object-backup-s3-credentials") | .stringData.password' | cut -d: -f2)"
export AWS_SECRET_ACCESS_KEY=$(vault kv get -format json "clusters/kv/${SECRET_KEY%/*}" | jq -r ".data.data.${SECRET_KEY##*/}")

Identify and retrieve snapshot

List the available snapshots. Identify the one you do want to restore. Take note of its ID.
```
restic snapshots
```

Retrieve the backup archive

restic restore <ID> --target cluster-backup-object-restore-$(date +%F)

Extract and prepare files

Change to the restore directory

cd cluster-backup-object-restore-$(date +%F)

List files in the backup. Take note of the path containing the required files.
```
tar tvf syn-cluster-backup-object-dumper.tar.gz
```
Extract required files. If all files should be extracted, path/inside/archive can be omitted. Files will be put in the directory restore within the current working directory.
```
mkdir restore
tar -C restore -xf syn-cluster-backup-object-dumper.tar.gz [path/inside/archive]
```
Prepare files

Depending on the restore requirements, the extracted files need to be altered before they can be applied to the cluster.

Apply objects

Apply the extracted and prepared objects to the target cluster.

Apply single file

kubectl --as cluster-admin apply -f <path/to/file>

Apply all files within a directory

kubectl --as cluster-admin apply -Rf <path/to/dir>