Kubernetes Node Recycling

You can manually trigger the lifecycle of an individual Kubernetes node by applying special annotations. This is useful when you want to proactively replace a node (e.g. suspected hardware issue, stuck kubelet, underlying VM degradation) or mark it for prioritized removal.

Annotations

We currently support the following node annotations:

When to use recycle vs delete

Recycle is the recommended way to proactively replace a node. It minimizes disruption by ensuring extra capacity arrives before the old node is removed. The pool size stays the same throughout the process.

Delete is useful when you want a specific node removed the next time the pool scales down. It does not trigger immediate removal — it marks the node so it is chosen first during scale-down. Use this when you can tolerate waiting for the autoscaler or a manual pool size reduction to actually remove the node.

If you need a specific node gone immediately without waiting for a scale-down, use recycle — it replaces the node in-place, keeping the pool size the same.

How to request a recycle

kubectl annotate node/<node-name> k8s.avisi.cloud/node-recycle-requested=true

This will:

1. Provision a new node (surge) and wait until it is Ready.
2. Cordon & drain the old node.
3. Delete the old node.

How to request a delete

kubectl annotate node/<node-name> k8s.avisi.cloud/node-delete-requested=true

This will:

1. Mark the node for prioritized removal during the next scale-down.
2. When the pool scales down, this node will be selected first, cordoned, drained, and deleted.

The node remains running and schedulable until a scale-down occurs. To also prevent new pods from being scheduled on it in the meantime, cordon it manually:

kubectl cordon <node-name>

Scale-down priority

When the platform needs to remove nodes from a pool (e.g. autoscaler scale-down), it selects nodes in the following priority order:

1. Nodes marked by the cluster autoscaler (ToBeDeletedByClusterAutoscaler taint)
2. Nodes with the node-delete-requested annotation
3. Unreachable nodes
4. NotReady nodes
5. Unschedulable nodes
6. Healthy nodes (oldest first)

Within the same priority level, nodes are ordered by timestamp (oldest first).

Cancelling a request

Recycle: Once the platform begins processing (a replacement node starts provisioning), removing the annotation will not stop the operation. If the annotation is removed before the next reconciliation picks it up (~2 minutes), the recycle is canceled.

Delete: Since this annotation only marks the node for prioritized removal during scale-down, removing the annotation at any time before the scale-down occurs will cancel the request.

kubectl annotate node/<node-name> k8s.avisi.cloud/node-recycle-requested-
# or
kubectl annotate node/<node-name> k8s.avisi.cloud/node-delete-requested-

Notes and Limitations

• These annotations do NOT support Bring Your Own Node (BYON) nodes; recycling/deletion must be handled manually for those nodes.
• Draining respects PodDisruptionBudgets; pods protected by strict PDBs may delay completion.
• Ensure critical workloads have appropriate redundancy before recycling or deleting a node.