Using Prometheus Agent Mode for better memory utilization
Posted June 13, 2023 by Avisi Cloud ‐ 2 min read
Reducing memory utilization of our monitoring addon
Using Prometheus Agent Mode we have reduced Prometheus memory utilization by up to 20% for most of our production clusters. Combined with new Enhancements from Prometheus, this offers significant operational improvements and optmizations within the monitoring plugin for your Avisi Cloud Kubernetes cluster.
Prometheus Enhancements
Prometheus has recently rolled out their version 2.43 which includes a feature called String Labels Optimization. If your Prometheus instance handles lots of series labels, this feature will improve its memory usage. Following this update, Prometheus 2.44.0 comes with the stringlabels change enabled by default.
Prometheus is an open-source systems monitoring and alerting toolkit, used by Avisi Cloud to provide monitoring on Kubernetes out of the box.
Moreover, we’ve set up the Prometheus instance within the cluster to operate in Agent mode. In this setup, Prometheus is dedicated to scraping metrics and pushing these data to a remote write endpoint. Avisi Cloud leverages the remote write feature of Prometheus to store metrics in our long-term storage solution, which you can access through the Avisi Cloud Console and various endpoints like Grafana.
Effects of the updated Prometheus StringLabels & Agent Mode
We’ve seen some good results since deploying the updated monitoring addon with Prometheus Agent Mode on our production clusters. Memory usage dropped by roughly 400MiB for most clusters.
We’re witnessing similar memory reductions across all our clusters, with decreases ranging from 100MiB to as much as 600MiB. The exact amount varies, depending on the size of the cluster and the number of endpoints that Prometheus is monitoring.
Available on cluster upgrade
We’re excited to share that these memory-saving enhancements are now available to all our customers. When you upgrade to the latest Kubernetes patches, Prometheus agent mode will be automatically configured, no extra steps needed.
Available versions:
- v1.27.2-u-ame.3
- v1.26.5-u-ame.3
- v1.25.10-u-ame.3