Backup and Restore

In our base setup, OpenShift is backed up on various stages: All nodes/masters are backed up, as well as all OpenShift resources. Backups are stored and can be accessed with burp, for further information like retention times and how to restore single files see Backup (BURP).

For a documentation on what is backed up see Backup and OpenShift 3 Backups.

Restore

The pre-backup script on each OpenShift master dumps and leaves all OpenShift objects into /var/lib/openshift-backup, so if you want to restore something that was deleted/changed since them, check there first.

  • The top level contains one JSON file for each object type (objects-*.json).

  • The split/ sub directory contains a folder for each OpenShift namespace, containing all objects for this namespace.

To restore the missing OpenShift objects, simply use oc apply or oc create to recreate the object.

oc apply -f /var/lib/openshift-backup/split/cust-app/objects/Route.json

Extracting single objects from backup files

In many cases, only a single object needs to be restored. Use jq to extract the one you want:

jq -r '.items[] | select(.metadata.name=="my-lost-secret")' \
  /var/lib/openshift-backup/split/cust-app/objects/Secret.json \
  > /root/restore-my-lost-secret.json
# or pipe to "oc apply -f-" to apply directly

Reattach PVs

If you restore a PVC from a backup, you must manually reattach its PV.

CAUTION

This method only works if the Gluster volume(s) in question has not been recycled and bound by another PV!

If the PV was already recycled, you’ll have to create a new Gluster volume and do a manual file restore from backup using burp.

Check whether the PV is in "Failed" state and therefore save to reattach:

oc get pv/gluster-pvXX
# NAME           CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM
# gluster-pvXX   1Gi        RWO,RWX        Recycle          Failed    cust-app/data

Get the UID from the new/restored PVC:

oc -n appuio-keycloak get cust-app/data -o json | jq -r .metadata.uid

Patch the PV to associate it with the new PVC:

oc patch pv/gluster-pvXX -p '{"spec":{"claimRef":{"uid": "<UID from last step>"}}}'
This only works if the PVC is in the same namespace and has the same name. If any of those changed, oc edit the PV instead and update spec.claimRef accordingly.