As a DevOps engineer, effectively managing Kubernetes clusters requires being mindful of avoiding unnecessary cloud resource usage. One particular area to address is the accumulation of unused Kubernetes Persistent Volumes (PVs). This situation commonly occurs when undeploying stateful sets, testing new applications, or in non-production clusters.
Cleaning up these unused PVs manually can be a tedious and error-prone task. It involves considering the Persistent Volume's Retain value, and if there are many PVs, it can take a significant amount of time. Moreover, it's a monotonous and repetively task.
To simplify and speed up the process, our CLI tool, acloud-toolkit, provides a convenient storage prune subcommand. This functionality allows you to quickly remove any unused persistent volume claims (PVCs). By utilizing the acloud-toolkit, you can automate the cleanup, ensuring efficient use of cloud resources and reducing unnecessary waste associated with unused PVs.
Consider a real-world scenario where we have a Lab cluster dedicated to one of our team members who has been actively working on Helm charts and deployments for Dependency Track. Over time, we noticed that the number of persistent volumes being utilized in the cluster has been steadily increasing. While the storage occupied may not be significant, it is important to address the issue of accumulating unused resources, as it is wasteful to retain them unnecessarily.
Here are the persistent volumes within the cluster:
As you can see, a significant number of persistent volumes are categorized as Released in the STATUS field. This indicates that the corresponding persistent volume claims have been deleted, while the actual volumes still exist in the cloud. Adding to the complexity, the RECLAIM POLICY is set to Retain" Therefore, if you were to simply delete these persistent volumes using the kubectl delete command, they would remain orphaned in the cloud infrastructure. Consequently, you would still incur costs for these volumes, despite them no longer being referenced or used within your cluster.
Once installed, we can start using the storage prune sub command to gather an overview of any volume that is unused and can be removed. By default, the command runs in dry-run mode. This is to avoid accidently removing persistent volumes that you did not intend to delete.
We can dubble check the volumes it is going to delete for us. We also see the total amount of storage that will be deleted by performing this action. In this case, 79Gi.
When we confirm we are happy with the volumes it will delete, we can run the same command using --dry-run=false.
The executed command accomplishes the following actions:
Modifies the reclaim policy of the persistent volumes from "Retain" to "Delete" specifically for volumes marked as Released. It's important to note that this command does not affect any persistent volumes with Available or Bound statuses.
Removes the persisted volumes from the Kubernetes cluster. The associated CSI (Container Storage Interface) provider, such as AWS EBS, Ceph, or others, will handle the deletion of the volumes within the underlying storage backend.
Following these steps allows for efficient cleanup of unused persistent volumes across your Kubernetes clusters. By executing this procedure, you can effectively manage resources and optimize storage utilization within your environment.