Scale Node Pools to Zero

When you scale every node pool in a cluster to zero nodes, AME enters a two-phase shutdown flow to guarantee the cluster can reach an absolute zero state. This behaviour is available from cluster-controller v2.29.0 (AME releases v1.34.1-u-ame.4, v1.33.5-u-ame.4, and v1.32.9-u-ame.4).

Two-phase eviction workflow

• If at least one node pool keeps a desired size greater than zero, AME performs the standard drain behaviour and respects Pod Disruption Budgets (PDBs) and grace periods.
• When every node pool targets zero nodes, AME enables force eviction mode to make progress even if pods cannot be evicted normally.

Flow overview

1. Start a node pool scale-down.
2. If at least one node pool keeps a desired size above zero, AME follows the standard drain process and the workflow ends.
3. If all node pools target zero nodes, AME enables force eviction mode and enters Phase 1.
4. Phase 1 performs graceful evictions for up to five minutes.
5. After the timeout, AME checks if pods remain:
   • If no pods remain, the nodes and backing machines are deleted and the workflow completes.
   • If pods remain, AME moves to Phase 2 and force deletes the remaining pods before removing the nodes and machines.

Phase details

Phase 1: graceful eviction

AME attempts a normal drain for five minutes:

• Uses the eviction API so PDBs and graceful termination periods continue to apply.
• Retries failed pod evictions until the timeout is reached.

Phase 2: force cleanup

If pods remain after the five-minute window, AME:

• Deletes remaining pods with a zero-second grace period.
• Bypasses the eviction API to prevent PDBs from blocking termination.
• Deletes the backing machines so the cluster reaches zero provisioned nodes.

This ensures automation and scheduled shutdowns can reliably deprovision an entire cluster without manual cleanup.