Change LUKS Key on an Encrypted Rook Managed Ceph OSD

This page describes how to change the LUKS encryption key of an OSD backing storage when using Rook to manage the Ceph cluster.

Information Gathering

  1. Export namespace of Ceph cluster, we’ll use this in later steps

    export CEPHCLUSTER_NAMESPACE=rook-ceph
export OSDID=X # select which OSD you want to operate on
export OSD_POD=$(kubectl -n "${CEPHCLUSTER_NAMESPACE}" get pods -l ceph-osd-id="${OSDID}" -o jsonpath='{.items[0].metadata.name}')

  2. Figure out the LUKS device which we’ll be working with by looking into the Pod spec to figure out the name of the PVC used

    kubectl -n ${CEPHCLUSTER_NAMESPACE} describe pod ${OSD_POD} | grep ClaimName

    The path on the node will then be:

    /var/lib/rook/${CEPHCLUSTER_NAMESPACE}/${PVCNAME}/ceph-${OSDID}/block-tmp

  3. Figure out the node the OSD Pod is running on

    kubectl -n ${CEPHCLUSTER_NAMESPACE} describe pod ${OSD_POD} | grep Node

  4. Get the current LUKS key

    kubectl -n ${CEPHCLUSTER_NAMESPACE} get secrets rook-ceph-osd-encryption-key-$PVCNAME -o jsonpath="{.data.dmcrypt-key}" | base64 -d

Change Key

  1. Login (SSH) to the node where the backing storage is located (check the OSD Pod spec)

  2. Add a new LUKS key, this needs the current key figured out above

    export LUKSDEV=/var/lib/rook/${CEPHCLUSTER_NAMESPACE}/${PVCNAME}/ceph-${OSDID}/block-tmp
cryptsetup luksAddKey $LUKSDEV

  3. Verify that the new key really works

    printf "THENEWKEY" | sudo cryptsetup luksOpen --test-passphrase $LUKSDEV && echo "There is a key available with this passphrase."

  4. Store the new key in the corresponding secret

    kubectl -n rook-ceph patch secret rook-ceph-osd-encryption-key-$PVCNAME -p='{"stringData":{"dmcrypt-key": "THENEWKEY"}}'

  5. Restart the OSD Pod to verify the change. It should come up as usual.

    kubectl -n ${CEPHCLUSTER_NAMESPACE} rollout restart deploy/ceph-osd-${OSDID}

  6. Figure out the Key slot of the old key

    cryptsetup -v luksOpen --test-passphrase $LUKSDEV

    It should print out Key slot X unlocked.

  7. Remove the old key

    cryptsetup -v luksKillSlot $LUKSDEV X

    For Enter any remaining passphrase enter the new key