Persistent Volume Filling Up
How to debug and resolve the PersistentVolumeFillingUp alert for Kubernetes Persistent Volumes
kubepersistentvolumefillingup alert is triggered when a Kubernetes persistent volume is running out of available space.
- The persistent volume is not large enough to accommodate the data being stored on it.
- The application running in the persistent volume is generating more data than expected.
- The persistent volume has been configured to use dynamic provisioning, but the underlying storage system is running out of space.
To diagnose the issue, you can run the following commands:
Check the usage of the persistent volume:
kubectl exec <pod-using-pv> -- df -h
Check the logs of the application running in the persistent volume to see if there are any error messages or unexpected data being generated.
You can also check metrics using Grafana Dashboards, to validate disk usage over time for the persistent volume.
To resolve the issue, follow these steps based on the findings from the diagnosis:
- If the persistent volume is not large enough, you will need to either resize the volume or migrate the data to a larger volume.
- If the application is generating more data than expected, you will need to investigate why and determine if there are ways to reduce the amount of data being generated.
Increasing the persistent volume size
In order to increase the size of the persistent volume being used, you first need to make sure your Kubernetes Storage Class supports Volume Expansion.
❯ kubectl get storageclass NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gp2 (default) ebs.csi.aws.com Retain WaitForFirstConsumer true 96d
ALLOWVOLUMEEXPANSION set to
If your persistent volume (PV) does not allow volume expansion, you need to copy your data to a new persistent volume that is increased in size.
When the storage class used for the persistent volume allows volume expansion, you can edit the persistent volume and modfiy the
kubectl edit pv <persistent-volume>
Edit the following section (
... spec: accessModes: - ReadWriteOnce capacity: storage: 10Gi
The Kubernetes Container Storage plugin (CSI) will automatically increase the size of the persistent volume / disk. Your pod in most cases does not need to restart or be recreated. You can verify this by performing
kubectl describe pv <persistent volume>. This will display events indicating the status of the volume expansion.
You can easily expand a persistent volume by using
acloud-toolkit storage resize.
resize command from
acloud-toolkit adjusts the size of a persistent volume claim (PVC). The command takes a PVC name as input along with an optional namespace parameter and a new size in gigabytes.
acloud-toolkit storage resize <persistent-volume-claim> [flags]
# Resize a PVC named 'data' in the default namespace to 20 gigabytes acloud-toolkit storage resize data --size 20G # Resize a PVC named 'data' in the 'prod' namespace to 50 gigabytes acloud-toolkit storage resize data --namespace prod --size 50G