New Nodes Require a Reboot After Joining

This runbook covers two problems that share the same root cause: nodes join your cluster with OS security patches still pending.

Situation

Nodes are provisioned from a base image. Those images are rebuilt on their own schedule, so between two image releases the OS packages in the image fall behind on security patches. A node that joins your cluster runs on the packages from that image until the automated unattended security upgrades run on the node itself and install the pending updates.

When those updates include a new kernel, the node writes /var/run/reboot-required and needs to be rebooted before the new kernel is actually in use. With automatic node reboots enabled, the node is then drained and rebooted, sometimes only hours after it joined.

Problem: new nodes immediately require a reboot after joining

Symptom

• A node that joined recently reports the NeedsReboot condition with status True.
• The node is drained and rebooted shortly after it joined.
• The node carries the weave.works/kured-most-recent-reboot-needed annotation with a timestamp close to the moment it joined.

AME nodes carry a custom NeedsReboot node condition. It is set to True with reason SystemRebootRequired as soon as /var/run/reboot-required is present on the node, and reports False with reason NoSystemRebootRequired when no reboot is pending.

kubectl describe node <node-name>

Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                 ------  -----------------                 ------------------                ------                       -------
  NeedsReboot          True    Tue, 18 Aug 2026 21:16:58 +0200   Tue, 07 Jul 2026 08:19:49 +0200   SystemRebootRequired         System reboot required

The condition is also shown in the Avisi Cloud Console, in the node pool details view of your cluster.

To find every node in the cluster that is waiting for a reboot:

kubectl get nodes -o custom-columns='NODE:.metadata.name,NEEDS_REBOOT:.status.conditions[?(@.type=="NeedsReboot")].status'

Cause

The node joined with the packages from its base image. The unattended security upgrades that run on the node installed a newer kernel afterwards, which marked the node as requiring a reboot. The reboot is a consequence of patching a node that was already serving workloads, not of a problem with the node itself.

Resolution

Enable security updates on join on the node pool and set it to INSTALL_AND_REBOOT. AME then installs the OS security updates during node bring-up and reboots the node before it joins, so the node arrives fully patched and does not need a reboot right after joining. This is the recommended setting for most node pools, and it will become the default value once the feature moves out of beta.

{
  "securityUpdatesOnJoin": "INSTALL_AND_REBOOT"
}

The setting applies to nodes that join after the change. Nodes that are already part of the cluster are not patched or rebooted by it, so expect one more reboot round for the existing nodes, or replace them.

Problem: the cluster autoscaler keeps adding unpatched nodes that are rebooted daily

Symptom

• Node pools scaled by the cluster autoscaler show a continuous stream of drains and reboots.
• Several nodes report the NeedsReboot condition with status True at the same time, and new ones keep appearing.
• Every scale-up is followed, within a day, by a reboot of the new node.
• With automatic node reboots enabled on an autoscaled pool, nodes appear to be recycled every day in a loop.

Cause

Autoscaling and automatic node reboots feed each other on the same node pool:

1. A node joins the cluster unpatched, because its base image lags behind on security updates.
2. The next morning, the unattended security upgrades install the pending updates on that node, and the node is flagged as requiring a reboot.
3. Automatic node reboots drains the node so that it can reboot. The pods that are evicted become unschedulable.
4. The cluster autoscaler reacts to those pending pods by adding a node, which is unpatched again.
5. The rebooted node comes back empty. It is no longer needed, so the autoscaler removes it.

Step 4 produces exactly the unpatched node that step 1 starts from, so the cycle continues. Every node in the pool is replaced this way, every day, without anything being wrong with the nodes themselves.

Resolution

Set security updates on join to INSTALL_AND_REBOOT on the autoscaled node pools. Each node the autoscaler adds installs its security updates and reboots while it is still empty, before joining the cluster, so it does not need to be drained and rebooted afterwards. That breaks the cycle at step 1: nodes no longer join unpatched, so there is nothing for the reboot in step 3 to be triggered by, and it is safe to run autoscaling and automatic node reboots on the same pool.

Related

• Security Updates on Join
• Node Pools: Automatic Node Reboots
• Autoscaling