Persistent Volume Filling Up

How to debug and resolve the PersistentVolumeFillingUp alert for Kubernetes Persistent Volumes

Situation

The kubepersistentvolumefillingup alert is triggered when a Kubernetes persistent volume is running out of available space.

Possible Causes

  • The persistent volume is not large enough to accommodate the data being stored on it.
  • The application running in the persistent volume is generating more data than expected.
  • The persistent volume has been configured to use dynamic provisioning, but the underlying storage system is running out of space.

Diagnosis

To diagnose the issue, you can run the following commands:

Check the usage of the persistent volume:

kubectl exec <pod-using-pv> -- df -h

Check the logs of the application running in the persistent volume to see if there are any error messages or unexpected data being generated.

You can also check metrics using Grafana Dashboards, to validate disk usage over time for the persistent volume.

Remediation

To resolve the issue, follow these steps based on the findings from the diagnosis:

  • If the persistent volume is not large enough, you will need to either resize the volume or migrate the data to a larger volume.
  • If the application is generating more data than expected, you will need to investigate why and determine if there are ways to reduce the amount of data being generated.

Increasing the persistent volume size

In order to increase the size of the persistent volume being used, you first need to make sure your Kubernetes Storage Class supports Volume Expansion.

❯ kubectl get storageclass
NAME            PROVISIONER       RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp2 (default)   ebs.csi.aws.com   Retain          WaitForFirstConsumer   true                   96d

Note the ALLOWVOLUMEEXPANSION set to true. If your persistent volume (PV) does not allow volume expansion, you need to copy your data to a new persistent volume that is increased in size.

When the storage class used for the persistent volume allows volume expansion, you can edit the persistent volume and modfiy the capacity field.

kubectl edit pv <persistent-volume>

Edit the following section (spec.capacity.storage):

...
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 10Gi

The Kubernetes Container Storage plugin (CSI) will automatically increase the size of the persistent volume / disk. Your pod in most cases does not need to restart or be recreated. You can verify this by performing kubectl describe pv <persistent volume>. This will display events indicating the status of the volume expansion.

using acloud-toolkit

You can easily expand a persistent volume by using acloud-toolkit storage resize.

The resize command from acloud-toolkit adjusts the size of a persistent volume claim (PVC). The command takes a PVC name as input along with an optional namespace parameter and a new size in gigabytes.

acloud-toolkit storage resize <persistent-volume-claim> [flags]

Some examples:

# Resize a PVC named 'data' in the default namespace to 20 gigabytes
acloud-toolkit storage resize data --size 20G

# Resize a PVC named 'data' in the 'prod' namespace to 50 gigabytes
acloud-toolkit storage resize data --namespace prod --size 50G
← DNS issues
Pod Evicted →