Updating an AME cluster

Update your AME Kubernetes cluster to the next version

AME Kubernetes clusters can be upgrade to a new version through the Avisi Cloud console. Upgrades are currently only available on demand, and have to be activated by clicking the upgrade button for a cluster.

Release notes

The release notes for new AME Kubernetes versions are published at AME Kubernetes - Release notes.

Upgrade impact

Each release describes the upgrade impact when upgrading from the previous version to the next. A few rules will always apply, and thus will not be written for each release. The upgrade impact column only describes ADDITIONAL impact expections.

  • When upgrading between different kubernetes patch versions (e.g. v1.20.4 to v1.20.5), will result in a node recycle.
  • When upgrading between Kubernetes minor versions (e.g. v1.20.x to v1.21.x), a node recycle will also occur, in addition to an occasional brief period of API downtime due to an etcd upgrade.
  • Upgrading is only supported from one minor version to the next.
  • Downgrading is only supported between patches of the same minor version. A full rollback (e.g. from v1.21 to v1.20) is supported, but ONLY right after an upgrade of the minor version, unless stated otherwise in the release notes.

Lifecycle & version policy

Please read our lifecycle policy for more information.


Upgrading your cluster

How to start an upgrade

First navigate to the cluster detail page. You can find your cluster’s detail page, by (in your organisation) navigating to the environments view (console.avisi.cloud/environments), and selecting the environment in which your cluster is provisioned.

Click on your environment, and you will see a table with all clusters available in this environment.

Rectangle
Overview of all clusters in your environment

Click on the cluster’s name. This will open the detail page of the cluster.

Detail page

The detail page shows the following information:

  • Currently running version of the cluster
  • Region the cluster is provisioned in
  • Size of the cluster
  • Some cluster specific settings, such as whether or not auto upgrades are enabled (currently not supported)
Rectangle
Cluster detail view

Starting the upgrade

Before starting

Make sure your pods can safely shutdown. For example, with some PostgreSQL deployments, Postgres will not shutdown when you still have active connections. For those cases it is recommended to scale down these statefulsets or deployments in order to speed up node recycles.

Release Channels

Avisi Cloud makes new versions available within an update or release channel. This allows you to stay on a specific Kubernetes minor release (v1.23) and easily find the next Stable Release within this serie.

Select channel version
Update cluster
Select channel version
Manual versions

Next to the cluster’s version, there is an option to select “upgrade cluster”. When clicking this, a new modal will open up in which you can select the next version to upgrade to, with a confirmation on starting the upgrade process.

Select version
Select the next Kubernetes version

When clicking on the button in this modal, Upgrade cluster, the upgrade will start. This process will take several minutes while the control plane is replaced, followed by a replacement of all nodes in your cluster.

Depending on the version you upgrade towards, nodes may not be recycled. To be safe, always assume nodes will be replaced during an upgrade, or throughly read the release notes before starting the upgrade.

Notes

  • The upgrade of a control plane takes around 2 minutes on average.
  • The upgrade of nodes in your cluster can take anywhere between a minute to 5 minutes, depending on the cloud provider used.
  • AME will automatically retry on failed upgrades.

In the cluster

In the cluster, the nodes will be replaced. The upgrade has been completed once all nodes show the new Kubernetes version.

You can validate this by using the following command:

kubectl get nodes -o wide

You can validate the version of your control plane by using the following command:

kubectl version

AME uses a surge style upgrade strategy. Meaning before a node is replaced, a new one will be provisioned.

Node replacement flow
Node replacement

Troubleshooting

There are a few edge cases in which an upgrade may take a long time to complete;

  • When running a workload such as postgres, that uses a graceful shutdown. e.g. waiting until all client connections have disconnected before exiting.
  • Standalone pods, deployed without making use of either a StatefulSet, Deployment or DaemonSet. These pods will be removed during a node replacement and will not be recreated.

To speed up this process, it may be necessary to shutdown some applications manually.