Update Persistent Volume for a PVC
How to fix a Persistent Volume binding to the wrong Persistent Volume
This run book will guide you through the process of fixing a Persistent Volume (PV) that has been incorrectly bound to the wrong Persistent Volume Claim (PVC).
This run book works for both
ReadWriteOnce Persistent Volumes (for example: NFS and Block Storage).
Since we are dealing with data, make sure you have created a back-up before modifing any persistent volume or persistent volume claim resource.
Follow the steps below to resolve the issue:
$ kubectl get pvc app-shared Bound pvc-98bd91e6-948d-4518-88d1-b9fed2779cc4 290Gi RWX nfs 23m
$ kubectl get pv pvc-09636c84-ad97-4921-9af5-8997bc96323b 290Gi RWX Retain Released app/app-shared nfs 154d pvc-98bd91e6-948d-4518-88d1-b9fed2779cc4 290Gi RWX Retain Bound app/app-shared nfs 10h
The PVC is bound to PV
pvc-98bd91e6-948d-4518-88d1-b9fed2779cc4 but needs to be bound to
Create new PVC yaml file
Create a new YAML file with the correct PVC, referencing the right PV in the last line. You can use the existing PVC output as a reference:
kubectl get pvc <pvc-name> -o yaml
apiVersion: v1 kind: PersistentVolumeClaim metadata: annotations: meta.helm.sh/release-name: app meta.helm.sh/release-namespace: app labels: app.kubernetes.io/managed-by: Helm name: app-shared namespace: app spec: accessModes: - ReadWriteMany resources: requests: storage: 290Gi storageClassName: nfs volumeMode: Filesystem volumeName: pvc-09636c84-ad97-4921-9af5-8997bc96323b
Remove Claim Reference from the PV
Edit the PV and remove the entire claimRef block, then save the file:
$ kubectl edit pv pvc-09636c84-ad97-4921-9af5-8997bc96323b
This looks something like below.
Remove this whole claimRef block and save file.
claimRef: apiVersion: v1 kind: PersistentVolumeClaim name: app-shared namespace: app resourceVersion: "93464003" uid: 3a172e36-8652-43da-9244-eb8c2ee5aaa5
The new STATUS of the PV should now be
Available instead of
$ kubectl get pv pvc-09636c84-ad97-4921-9af5-8997bc96323b 290Gi RWX Retain Available app/app-shared nfs 154d pvc-98bd91e6-948d-4518-88d1-b9fed2779cc4 290Gi RWX Retain Bound app/app-shared nfs 10h
Remove the old PVC
The PVC that is pointing to the wrong PV needs to be removed. If this PVC is mounted into a pod, the pod must be stopped to free the PVC:
$ kubectl delete pvc app-shared
If the PVC remains in the
Deleting status, it is likely still mounted. Stop the pod where it is mounted.
Create new PVC
Deploy the correct PVC you created in step 2. Ensure you are in the right namespace when deploying:
$ kubectl apply -f <create_new_pvc.yaml>
Verify the changes:
$ kubectl get pvc | grep shared app-shared Bound pvc-09636c84-ad97-4921-9af5-8997bc96323b 290Gi RWX nfs 42m $ kubectl get pv | grep shared pvc-09636c84-ad97-4921-9af5-8997bc96323b 290Gi RWX Retain Bound app/app-shared nfs 154d pvc-98bd91e6-948d-4518-88d1-b9fed2779cc4 290Gi RWX Retain Released app/app-shared nfs 10h
The newly created PVC should now be bound to the correct PV. The original PV is released but still exists due to the “Retain” policy, ensuring that data is not lost until the PV is manually removed.
Start application and check data
Test the application by starting and verify the right data exists in the right place.