Update Persistent Volume for a PVC

Fix a Persistent Volume binding to the wrong Persistent Volume

This run book will guide you through the process of fixing a Persistent Volume (PV) that has been incorrectly bound to the wrong Persistent Volume Claim (PVC).

This run book works for both ReadWriteMany and ReadWriteOnce Persistent Volumes (for example: NFS and Block Storage).

Back-up data

Since we are dealing with data, make sure you have created a back-up before modifing any persistent volume or persistent volume claim resource.

Follow the steps below to resolve the issue:

Situation

The PersistentVolumeClaim:

$ kubectl get pvc
app-shared                         Bound    pvc-98bd91e6-948d-4518-88d1-b9fed2779cc4    290Gi      RWX            nfs            23m

The PersistentVolumes:

$ kubectl get pv
pvc-09636c84-ad97-4921-9af5-8997bc96323b   290Gi      RWX            Retain           Released      app/app-shared                                           nfs                     154d
pvc-98bd91e6-948d-4518-88d1-b9fed2779cc4   290Gi      RWX            Retain           Bound         app/app-shared                                           nfs                     10h

The PVC is bound to PV pvc-98bd91e6-948d-4518-88d1-b9fed2779cc4 but needs to be bound to pvc-09636c84-ad97-4921-9af5-8997bc96323b.

Create new PVC yaml file

Create a new YAML file with the correct PVC, referencing the right PV in the last line. You can use the existing PVC output as a reference:

kubectl get pvc <pvc-name> -o yaml

For example:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    meta.helm.sh/release-name: app
    meta.helm.sh/release-namespace: app
  labels:
    app.kubernetes.io/managed-by: Helm
  name: app-shared
  namespace: app
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 290Gi
  storageClassName: nfs
  volumeMode: Filesystem
  volumeName: pvc-09636c84-ad97-4921-9af5-8997bc96323b

Remove Claim Reference from the PV

Edit the PV and remove the entire claimRef block, then save the file:

$ kubectl edit pv pvc-09636c84-ad97-4921-9af5-8997bc96323b

This looks something like below.

Remove this whole claimRef block and save file.

claimRef:
  apiVersion: v1
  kind: PersistentVolumeClaim
  name: app-shared
  namespace: app
  resourceVersion: "93464003"
  uid: 3a172e36-8652-43da-9244-eb8c2ee5aaa5

The new STATUS of the PV should now be Available instead of Released:

$ kubectl get pv
pvc-09636c84-ad97-4921-9af5-8997bc96323b   290Gi      RWX            Retain           Available     app/app-shared                                           nfs                     154d
pvc-98bd91e6-948d-4518-88d1-b9fed2779cc4   290Gi      RWX            Retain           Bound         app/app-shared                                           nfs                     10h

Remove the old PVC

The PVC that is pointing to the wrong PV needs to be removed. If this PVC is mounted into a pod, the pod must be stopped to free the PVC:

$ kubectl delete pvc app-shared

If the PVC remains in the Deleting status, it is likely still mounted. Stop the pod where it is mounted.

Create new PVC

Deploy the correct PVC you created in step 2. Ensure you are in the right namespace when deploying:

$ kubectl apply -f <create_new_pvc.yaml>

Verify the changes:

$ kubectl get pvc | grep shared
app-shared                         Bound    pvc-09636c84-ad97-4921-9af5-8997bc96323b   290Gi      RWX            nfs            42m
 
$ kubectl get pv | grep shared
pvc-09636c84-ad97-4921-9af5-8997bc96323b   290Gi      RWX            Retain           Bound      app/app-shared                                           nfs                     154d
pvc-98bd91e6-948d-4518-88d1-b9fed2779cc4   290Gi      RWX            Retain           Released   app/app-shared                                           nfs                     10h

The newly created PVC should now be bound to the correct PV. The original PV is released but still exists due to the "Retain" policy, ensuring that data is not lost until the PV is manually removed.

Start application and check data

Test the application by starting and verify the right data exists in the right place.