Restore Persistent Volumes (PVs) and Persistent Volume Claims (PVCs) from an Object Backup

Starting situation

  • You want to restore Persistent Volumes (PVs) and Persistent Volume Claims (PVCs)

    • The cluster manifests were deleted or corrupted

    • The underlying storage is a Cloudscale volume

    • The underlying volume still exists in the Cloudscale project

Prerequisites

  • You have extracted the manifest files according to Recover objects from backup

  • yq yq YAML processor (version 4 or higher - use the go version by mikefarah, not the jq wrapper by kislyuk)

  • kubectl set up to access the target cluster

PV and PVC recovery

  1. Identify the PVC in the backup and put it in a file called pvc.yaml

  2. Extract the PV name from the PVC

    PV_NAME=$(yq e '.spec.volumeName' pvc.yaml)
echo "$PV_NAME"

  3. Put the PV manifest in a file called pv.yaml

  4. Check the Cloudscale volume in the Cloudscale web interface to ensure it’s intact and available

  5. Detach the Cloudscale volume from any existing Kubernetes nodes if it’s still attached

    cloudscale volume detach

  6. Edit pv.yaml and remove the claimRef section to make the PV unbound

    yq -i '.spec.persistentVolumeReclaimPolicy = "Retain"' pv.yaml
yq -i 'del(.spec.claimRef)' pv.yaml

  7. Apply the modified PV manifest to the cluster

    kubectl apply -f pv.yaml

  8. Strip annotations from the PVC manifest so that the controller rebinds it to the existing PV

    yq -i 'del(.metadata.annotations)' pvc.yaml

  9. Apply the modified PVC manifest to the cluster

    kubectl apply -f pvc.yaml

  10. Verify that the PV is now bound to the PVC

    kubectl get pv "$PV_NAME" -oyaml | yq .status.phase