Default alertsCopy Anchor Link

AME Kubernetes comes with a set of default alerts for Traefik. This page serves as a reference for when one of these alerts fires within your cluster. Each alert gives a brief description of what it means, a list of possible causes and suggestions on how to resolve the issue.

CertificatesCopy Anchor Link

AME reads certificate expiry from the TLS certificates that Traefik serves. The Traefik default certificate is excluded.

CertificateExpiryWithin30Days [info]Copy Anchor Link

The certificate for <cn> is expiring within 30 days. Automatic renewal may have failed.

A certificate that Traefik serves expires in less than 30 days. Check whether it is still renewed automatically before it reaches the one-week threshold.

Possible causes

• cert-manager cannot complete the ACME challenge, for example because the DNS record does not point at the cluster load balancer
• The Certificate, or the Ingress annotation that requests it, was removed, so nothing renews the certificate
• The certificate is stored in a Secret that you manage yourself and has not been replaced

Find the certificate and inspect its renewal status:

kubectl get certificates -A
kubectl describe certificate <name> -n <namespace>
kubectl get certificaterequests,orders,challenges -n <namespace>

CertificateExpiryWithinOneWeek [warning]Copy Anchor Link

The certificate for <cn> is expiring within one week. Automatic renewal may have failed.

A certificate that Traefik serves expires in less than a week. Clients start rejecting connections once it expires, so resolve this before then. Follow the same steps as for CertificateExpiryWithin30Days and check the cert-manager logs for errors about the certificate.

TraefikCopy Anchor Link

TraefikServiceHighHttp5xxRate [critical]Copy Anchor Link

Traefik high HTTP 5xx error rate (<value>%) for service <service>.

More than 1% of the requests to a service return a 5xx status code, sustained for 5 minutes.

Possible causes

• A service is experiencing an outage
• The service has no ready endpoints, for example because its pods are crashing or failing their readiness probes
• Cluster / node failure
• Network partition

Investigate the service named in the alert, starting with its pods and their logs. Additional alerts usually fire alongside this one when the backend itself is failing.

TraefikServiceRequestRateOutsideNormalRange [info]Copy Anchor Link

Requests for service <service> are outside expected operating parameters (Z-Score > 3)

The request rate for a service deviates more than three standard deviations from the rate AME predicts, sustained for 15 minutes. AME bases the prediction on the traffic for the same hour in the previous week. The alert covers both unusually high and unusually low traffic.

AME only evaluates services that receive more than one request per second, and needs two weeks of traffic history for a service before the alert can fire. Traefik internal services and cert-manager HTTP-01 solvers are excluded.

Possible causes

• A marketing campaign, release or other planned event changed the traffic pattern
• A client or crawler is sending an unusual amount of traffic
• An upstream outage, DNS change or client failure means requests no longer reach the service
• A public holiday or other seasonal change in usage

Compare the traffic with the previous week in Grafana. If the change is expected, no action is needed.