Security Updates on Join

Nodes boot from a base image, and that image can lag behind on OS security patches. A node that joins your cluster today may therefore already be missing security updates, and it will only pick them up later through the automated unattended security upgrades that run on the node. When those updates include a new kernel, the node is flagged as requiring a reboot shortly after it joined.

Security updates on join closes that gap. When enabled, AME installs the OS security updates during node bring-up, before the node joins the cluster, so the node joins fully patched.

How it works

During bring-up, and before the node runs kubeadm join, AME runs the OS security updates on the machine using unattended-upgrade. Depending on the configured value, the node then either joins directly or reboots first when the installed updates require it (/var/run/reboot-required is present). After the reboot, the node joins the cluster.

Values

Security updates on join is configured per node pool, through the securityUpdatesOnJoin setting.

Each node pool has its own setting, so you can enable it for the pools that benefit from it and leave it off for pools where node bring-up time matters most.

When it applies

The setting applies only when a node first joins a cluster. That includes:

• newly provisioned nodes in a node pool,
• replacement nodes, for example after node auto healing, node recycling or a replace upgrade strategy,
• nodes added by the cluster autoscaler.

It does not patch or reboot nodes that are already part of your cluster, and it is not applied during in-place upgrades of existing nodes. To bring existing nodes to the same state, replace them or use automatic node reboots.

Autoscaling combined with automatic node reboots

Enabling autoscaling and automatic node reboots on the same node pool is a poor combination as long as its nodes join unpatched. A node joins, is patched the next morning, and is drained so that it can reboot. The evicted pods make the autoscaler add another unpatched node, and the rebooted node comes back empty and is scaled down again. The pool then recycles all of its nodes every day.

Setting security updates on join to INSTALL_AND_REBOOT removes the cause: nodes join fully patched, so they are not flagged for a reboot shortly after joining, and the two features can be used together safely.

See the runbook on new nodes that require a reboot after joining for the full cycle and how to recognise it.

Trade-off: node bring-up time

Installing security updates takes time, and rebooting the node adds to that. Nodes in a pool with INSTALL or INSTALL_AND_REBOOT therefore take longer before they are Ready and can accept workloads.

Keep this in mind for node pools that are scaled by the cluster autoscaler: a slower bring-up means the cluster reacts slower to a sudden increase in demand. If autoscaling reaction time is critical for a pool, consider leaving the setting OFF for that pool, or scaling out earlier.

Enabling security updates on join

Security updates on join is configured per node pool, so you set it on each node pool that should use it.

Avisi Cloud Console

Open your cluster in the Avisi Cloud Console and go to the Node Pools tab. The Security Updates On Join field is available when you add a node pool and when you modify an existing one, with the options Off, Install security updates and Install security updates and reboot if required.

acloud CLI

The acloud CLI exposes the setting as --security-updates-on-join, which accepts off, install and install-and-reboot.

Set it when you create a node pool:

acloud node-pools create \
  --name=workers-a \
  --cluster=example-cluster \
  --node-type="t3.large" \
  --node-count=2 \
  --security-updates-on-join=install-and-reboot

Or change it on node pools that already exist:

# set the value on a single node pool, by node pool id
acloud node-pools update 1001 --security-updates-on-join install

# set the value on every node pool in a cluster
acloud node-pools update -c example-cluster --security-updates-on-join install-and-reboot

API

Set the securityUpdatesOnJoin field when you create or update a node pool. The API uses the value names OFF, INSTALL and INSTALL_AND_REBOOT, where the CLI uses their lowercase equivalents.

{
  "name": "workers",
  "nodeSize": "t3.large",
  "securityUpdatesOnJoin": "INSTALL_AND_REBOOT"
}

When the field is omitted on create, the node pool defaults to OFF. When the field is omitted on update, the node pool keeps its current value.

See the API reference for creating a node pool and updating a node pool.

Bring Your Own Node

Security updates on join is supported for managed node pools on all cloud providers and for Bring Your Own Node (BYON) node pools. For BYON pools, the generated join and install script honors the node pool setting, so nodes you provision yourself install their security updates before joining as well.