Kubernetes Cluster Node Pool

Feature state: stable

AME Kubernetes Cluster Node Pool functionality

A Node Pool represents a set of machines that will be provisioned to join a Kubernetes cluster as a Node. Avisi Cloud provisions these machines within the Cloud your cluster runs in.

Terminology

All Nodes within a pool will receive the same configuration. This includes;

The Type (machine or instance type of the Virtual Machine within a Cloud Provider). This represents the size (i.g. 8GB memory) of a virtual machine.
Name of the node pool, which is used to configure the Kubernetes node role label for each Node within this Pool.
Version of Kubernetes and the container runtime running on the Node.
Amount of nodes / size of the node pool. This is the number of nodes that are provisioned within the pool.
Node Labels & Annotations. These are applied to the node within Kubernetes and can be used for scheduling decisions or automation tasks/scripts.
Node Taints. These are applied to the nodes within Kubernetes and can be used to restrict the node to only running certain workloads.

Nodes within a pool are gracefully replaced during upgrades using a surge strategy. This means Avisi Cloud makes sure a new node has successfully joined and is healthy before starting to drain and delete an outdated node. This process is called rolling updates.

See our help article Updating your cluster for more information.

Auto Healing

Automatic healing of unhealthy nodes within a Node Pool.

Autoscaling

Scale your node pool based on utilization. See our help article about node pool scaling for more information.

Automatic Node Reboots

Nodes that require a reboot (e.g., after a kernel or system update) are automatically rebooted in a controlled manner. Only one node is rebooted at a time, and nodes are safely drained before rebooting to minimize workload disruption. Automatic node reboots must be enabled per node pool. The reboot window (update days, start time, end time, and time zone) can be configured on the cluster's Patching page under System Reboots.

Autoscaling combined with automatic node reboots

On a node pool that also has autoscaling enabled, automatic node reboots can recycle every node in the pool daily: nodes join unpatched, are patched and drained for a reboot, the evicted pods make the autoscaler add another unpatched node, and the rebooted node is scaled down again while empty. Enable security updates on join so that nodes join fully patched, or see the runbook.

Blocking reboots with pod labels

You can prevent a node from rebooting while specific pods are running on it. Any pod with the label k8s.avisi.cloud/block-reboot: "true" will block the node it runs on from being rebooted until the pod is removed or the label is removed.

This is useful for protecting long-running jobs, batch workloads, or other critical processes that should not be interrupted by a reboot.

Lock timeout

Supported on AME versions containing addon-controller v0.29.3 or newer.

A node reboot lock is automatically released after 1 hour if the reboot process has not completed. This prevents a stuck or failed reboot from blocking all other nodes in the cluster from rebooting.

Node annotations

The following annotations are automatically added to nodes during the reboot process:

Annotation	Description
`weave.works/kured-reboot-in-progress`	Set on a node that is currently being rebooted.
`weave.works/kured-most-recent-reboot-needed`	Timestamp of the most recent reboot requirement detected on the node.
`k8s.avisi.cloud/kured-node-lock`	Used internally to coordinate reboots across the cluster, ensuring only one node reboots at a time.

These annotations can be used for monitoring or automation purposes.

Security Updates on Join

beta

Nodes boot from a base image that can lag behind on OS security patches, which means a freshly joined node can immediately be flagged as requiring a reboot. With security updates on join, AME installs the OS security updates during node bring-up, before the node joins the cluster, and optionally reboots the node first when the updates require it.

The setting is configured per node pool through securityUpdatesOnJoin and supports three values: OFF (default), INSTALL and INSTALL_AND_REBOOT. It applies only when a node first joins, so it does not patch or reboot existing nodes. Installing the updates does make node bring-up take longer.

We recommend INSTALL_AND_REBOOT for most node pools. It will become the default value once the feature moves out of beta.

See Security Updates on Join for the full reference.