Kubernetes Node Recycling

Feature state:

stable

Manually request replacement or deletion of a node using annotations.

You can manually trigger the lifecycle of an individual Kubernetes node by applying special annotations. This is useful when you want to proactively replace a node (e.g. suspected hardware issue, stuck kubelet, underlying VM degradation) or force its deletion.

Annotations

We currently support the following node annotations:

Annotation	Action
`k8s.avisi.cloud/node-recycle-requested`	Recycle the node: a new node is provisioned and joins the cluster first; after it becomes Ready and passes health checks, the annotated node is cordoned, drained and deleted.
`k8s.avisi.cloud/node-delete-requested`	Delete the node immediately: the node is cordoned, drained, then deleted. If the node pool desired size still requires a node, a replacement will be created afterwards by normal reconciliation.

When to use recycle vs delete

Recycling is the recommended way to proactively replace a node. It minimizes disruption by ensuring extra capacity arrives before the old node is removed.

Use delete only when you explicitly want the node removed as soon as possible (for example an irrecoverable state) and can tolerate a brief capacity dip until the autoscaler / reconciler restores the desired size.

How to request a recycle

kubectl annotate node/<node-name> k8s.avisi.cloud/node-recycle-requested=true

This will:

Provision a new node (surge) and wait until it is Ready.
Cordon & drain the old node.
Delete the old node.

How to request a delete

kubectl annotate node/<node-name> k8s.avisi.cloud/node-delete-requested=true

This will:

Cordon & drain the node.
Delete the node.
(If the desired size still allows) a new node will be launched afterwards by the normal node pool reconciliation.

Idempotency & clearing requests

Setting the annotation to true queues the action once. After processing begins (the platform has acknowledged and scheduled the recycle/delete), removal of the annotation will NOT stop the in‑flight operation.

If the platform has not yet picked up the annotation, you can cancel by removing it; otherwise allow the procedure to finish.

kubectl annotate node/<node-name> k8s.avisi.cloud/node-recycle-requested-
# or
kubectl annotate node/<node-name> k8s.avisi.cloud/node-delete-requested-

Notes and Limitations

Bring Your Own Node (BYON) nodes are NOT supported by these annotations; recycling/deletion must be handled manually for those nodes.
Draining respects PodDisruptionBudgets; pods protected by strict PDBs may delay completion.
Ensure critical workloads have appropriate redundancy before recycling or deleting a node.