Updating an AME cluster
How to Upgrade an AME Cluster to a new version
AME Kubernetes clusters can be upgrade to a new version through the Avisi Cloud console. Upgrades are currently only available on demand, and have to be activated by clicking the upgrade button for a cluster.
Release notes
The release notes for new AME Kubernetes versions are published at AME Kubernetes - Release notes.
Upgrade impact
Each release describes the upgrade impact when upgrading from the previous version to the next. A few rules will always apply, and thus will not be written for each release. The upgrade impact column only describes ADDITIONAL impact expections.
- When upgrading between different kubernetes patch versions (e.g. v1.20.4 to v1.20.5), will result in a node recycle.
- When upgrading between Kubernetes minor versions (e.g. v1.20.x to v1.21.x), a node recycle will also occur, in addition to an occasional brief period of API downtime due to an etcd upgrade.
- Upgrading is only supported from one minor version to the next.
- Downgrading is only supported between patches of the same minor version. A full rollback (e.g. from v1.21 to v1.20) is supported, but ONLY right after an upgrade of the minor version, unless stated otherwise in the release notes.
Lifecycle & version policy
Please read our lifecycle policy for more information.
Upgrading your cluster
How to start an upgrade
First navigate to the cluster detail page. You can find your cluster's detail page, by (in your organisation) navigating to the environments view (console.avisi.cloud/environments), and selecting the environment in which your cluster is provisioned.
Click on your environment, and you will see a table with all clusters available in this environment.
Click on the cluster's name. This will open the detail page of the cluster.
Detail page
The detail page shows the following information:
- Currently running version of the cluster
- Region the cluster is provisioned in
- Size of the cluster
- Some cluster specific settings, such as whether or not auto upgrades are enabled (currently not supported)
Starting the upgrade
Before starting
Upgrading clusters is only possible if they are running. For stopped clusters, please ensure they are started and verified as running correctly before proceeding with the upgrade. Perform upgrades one minor version at a time, ensuring all nodes and pods are functioning properly before advancing to the next minor version.
Make sure your pods can safely shutdown. For example, with some PostgreSQL deployments, Postgres will not shutdown when you still have active connections. For those cases it is recommended to scale down these statefulsets or deployments in order to speed up node recycles.
Release Channels
Avisi Cloud makes new versions available within an update or release channel. This allows you to stay on a specific Kubernetes minor release (v1.23) and easily find the next Stable Release within this serie.
Manual versions
Next to the cluster's version, there is an option to select "upgrade cluster". When clicking this, a new modal will open up in which you can select the next version to upgrade to, with a confirmation on starting the upgrade process.
When clicking on the button in this modal, Upgrade cluster, the upgrade will start. This process will take several minutes while the control plane is replaced, followed by a replacement of all nodes in your cluster.
Depending on the version you upgrade towards, nodes may not be recycled. To be safe, always assume nodes will be replaced during an upgrade, or throughly read the release notes before starting the upgrade.
Notes
- The upgrade of a control plane takes around 2 minutes on average.
- The upgrade of nodes in your cluster can take anywhere between a minute to 5 minutes, depending on the cloud provider used.
- AME will automatically retry on failed upgrades.
In the cluster
In the cluster, the nodes will be replaced. The upgrade has been completed once all nodes show the new Kubernetes version.
You can validate this by using the following command:
You can validate the version of your control plane by using the following command:
AME uses a surge style upgrade strategy. Meaning before a node is replaced, a new one will be provisioned.
Troubleshooting
There are a few edge cases in which an upgrade may take a long time to complete;
- When running a workload such as postgres, that uses a graceful shutdown. e.g. waiting until all client connections have disconnected before exiting.
- Standalone pods, deployed without making use of either a StatefulSet, Deployment or DaemonSet. These pods will be removed during a node replacement and will not be recreated.
To speed up this process, it may be necessary to shutdown some applications manually.
Last updated on