Restore Vault from backup

This how-to guides you through restoring the vault-prod Vault instance from backup into a new cluster. It assumes that a new cluster already exists and is synthesized.

Prerequisites

Procedure

This is the VSHN-specific version of the component-vault restore procedure.

1. Set up new Vault instance

  • Add the vault application to your cluster configuration

  • Copy or reuse the component configuration from VSHN Synfra

  • Initially disable backups by setting .backups.enabled to false

  • Compile and push the cluster config and wait for Vault to start.

2. Retrieve the Vault snapshot

  • Access the Vault Prod backup credentials in Passbolt

  • Set up the restic credentials

    export AWS_ACCESS_KEY_ID="USERNAME_FROM_PASSBOLT"
export AWS_SECRET_ACCESS_KEY="PASSWORD_FROM_PASSBOLT"
export RESTIC_REPOSITORY="s3:https://objects.rma.cloudscale.ch/syn-vault-prod-backup" (1)
export RESTIC_PASSWORD="RESTIC_REPO_KEY" (2)
    1 This value can also be found in the URL field of the passbolt item
    2 You can find this in the description field of the passbolt item

  • Retrieve the latest Vault snapshot to your local disk

    mkdir restore
restic restore --target restore/ latest

  • Verify the snapshot file

    ls restore
# This should show a file named "syn-vault-prod-backup.snapshot"

3. Restore the snapshot

  • Expose the Vault pod

    kubectl port-forward -nsyn-vault-prod syn-vault-prod-0 8200

  • In a separate terminal, prepare the environment to access Vault

    # Get root token to log in
export VAULT_TOKEN="$(kubectl get secret -nsyn-vault-prod syn-vault-prod-seal -ojsonpath='{.data.vault-root}' | base64 -d)"
export VAULT_ADDR="http://127.0.0.1:8200"

  • Restore the backup

    vault operator raft snapshot restore -force restore/syn-vault-prod-backup.snapshot

4. Unseal Vault

If you were logged into the Vault UI, you should have gotten logged out now. This is expected.

  • Open your browser at localhost:8200

  • Retrieve the Vault-Prod unseal key from Passbolt

  • Use this unseal key when prompted in the UI to unseal Vault

  • Retrieve the Vault-Prod root token from Passbolt

  • Use this token to log in with the Token method

  • Verify that the restore worked, and secrets are now in Vault.

If Vault runs at the same URL as previously, OIDC login should work as well at this point

5. Update the Vault Secret

  • Encode the Vault credentials

    export VAULT_UNSEAL_KEY="UNSEAL_KEY_FROM_PASSBOLT"
export VAULT_ROOT_TOKEN="ROOT_TOKEN_FROM_PASSBOLT"

echo -n "$VAULT_UNSEAL_KEY" | base64 -w0
echo -n "$VAULT_ROOT_TOKEN" | base64 -w0

  • Update the Vault secret

    kubectl edit secret -nsyn-vault-prod syn-vault-prod-seal

  • Update the vault-root and vault-unseal-0 keys to reflect the values from Passbolt

  • Save the secret

  • Verify that auto-unseal works:

    • Restart all vault pods simultaneously:

      kubectl delete pod -nsyn-vault-prod syn-vault-prod-{0..2}

    • Expose the Vault UI

      kubectl port-forward -nsyn-vault-prod syn-vault-prod-0 8200

    • Verify that the Vault UI does not prompt you for the unseal key

6. Cleanup

  • Reenable backups.enabled in the component configuration