Security Updates on Join

Beta. Security updates on join is a beta feature. The setting, its values and its defaults can still change while we gather feedback.

Today we are introducing a new node pool setting: security updates on join. When enabled, AME installs the OS security updates on a node during bring-up, before that node joins your cluster.

The problem: fresh nodes with pending security patches

Nodes are provisioned from a base image, and those images are rebuilt and published on their own schedule. Between two image releases, the OS packages in the image fall behind on security patches.

The result is familiar if you run clusters that scale or replace nodes regularly: a node that joined a few minutes ago is already missing security updates. It picks them up later, through the automated unattended security upgrades that run on the node itself. When those updates include a new kernel, the node is flagged as requiring a reboot, and it has to be drained and rebooted again, shortly after it joined.

For node pools that are scaled by the cluster autoscaler, this repeats every time a node is added. Worse, it can turn into a loop that never settles. A node joins unpatched, gets patched the next morning and is drained so that it can reboot. The pods that are evicted make the autoscaler add another node, which is unpatched again, and the rebooted node comes back empty and is scaled down. The pool ends up recycling all of its nodes, every day, while nothing is actually wrong with them.

What we built

Security updates on join moves that work to the front of a node's life, when it is still empty and nothing is running on it yet.

During bring-up, and before the node joins the cluster, AME runs the OS security updates on the machine. When configured to do so, the node also reboots first if the installed updates require a reboot, and only joins the cluster afterwards. Your workloads land on a node that is already patched and already rebooted.

The setting is configured per node pool and has three values:

Value	What happens during node bring-up
`OFF`	Nothing. The node joins with the packages from its base image. This is the default.
`INSTALL`	The OS security updates are installed, then the node joins.
`INSTALL_AND_REBOOT`	The OS security updates are installed, the node reboots when the updates require it, then it joins.

We recommend INSTALL_AND_REBOOT for most node pools, and it will become the default value once this feature moves out of beta. Until then you have to set it explicitly, since node pools default to OFF.

It applies when a node first joins the cluster: newly provisioned nodes, replacement nodes, and nodes added by the cluster autoscaler. It does not patch or reboot the nodes that are already running in your cluster.

Security updates on join works for managed node pools on all cloud providers, and for Bring Your Own Node pools, where the generated join and install script honors the node pool setting.

Enabling it

You can set it per node pool in the Avisi Cloud Console, with the acloud CLI, or through the API.

In the console, the Security Updates On Join field is available when you add or modify a node pool. With the CLI:

acloud node-pools update -c example-cluster --security-updates-on-join install-and-reboot

And through the API, by setting the securityUpdatesOnJoin field on a node pool:

{
  "name": "workers",
  "nodeSize": "t3.large",
  "securityUpdatesOnJoin": "INSTALL_AND_REBOOT"
}

Omitting the field on create gives you the default, OFF. Omitting it on update keeps the current value of the node pool.

One trade-off to plan for

Installing security updates takes time, and a reboot adds to that. Nodes in a pool with INSTALL or INSTALL_AND_REBOOT take longer to become Ready. For node pools driven by the cluster autoscaler, that directly affects how fast the cluster reacts to a spike in demand, so it is worth deciding per node pool rather than switching it on everywhere at once.

Availability and feedback

Security updates on join is available in AME v1.34.7-u-ame.0 and newer, v1.35.4-u-ame.0 and newer, and all v1.36 versions.

Because the feature is in beta, we are keen to hear how it behaves in your clusters, in particular how much it adds to your node bring-up times. Let us know through our support desk.