Upgrade from v1.3.1 to v1.3.2
General information
An Upgrade button appears on the Dashboard screen whenever a new Harvester version that you can upgrade to becomes available. For more information, see Start an upgrade.
For air-gapped environments, see Prepare an air-gapped upgrade.
Known issues
1. Two-node cluster upgrade stuck after the first node is pre-drained
Shut down all workload VMs before upgrading two-node clusters to prevent data loss.
The worker node can falsely transition to a not-ready state when RKE2 is upgraded on the management node. Consequently, the existing pods on the worker node are evicted and new pods cannot be scheduled on any nodes. These ultimately cause a chained failure in the whole cluster and prevent completion of the upgrade process.
Check the cluster status when the following occur:
- The upgrade process becomes stuck for some time.
- You are unable to access the Harvester UI and receive an HTTP 503 error.
Check the conditions and node statuses of the latest
Upgradecustom resource.Proceed to the next step if the following conditions are met:
SystemServicesUpgradedis set toTrue, indicating that the system services upgrade is completed.- In
nodeStatuses, the state of the management node is eitherPre-drainedorWaiting Reboot. - In
nodeStatuses, the state of the worker node isImages preloaded.
Example:
# Find out the latest Upgrade custom resource
$ kubectl -n harvester-system get upgrades.harvesterhci -l harvesterhci.io/latestUpgrade=true
NAME AGE
hvst-upgrade-szlg8 48m
# Check the conditions and node statuses
$ kubectl -n harvester-system get upgrades hvst-upgrade-szlg8 -o yaml
apiVersion: harvesterhci.io/v1beta1
kind: Upgrade
metadata:
...
labels:
harvesterhci.io/latestUpgrade: "true"
harvesterhci.io/upgradeState: UpgradingNodes
name: hvst-upgrade-szlg8
namespace: harvester-system
...
spec:
image: ""
logEnabled: false
version: v1.3.2-rc2
status:
conditions:
- status: Unknown
type: Completed
- lastUpdateTime: "2024-09-02T11:57:04Z"
message: Upgrade observability is administratively disabled
reason: Disabled
status: "False"
type: LogReady
- lastUpdateTime: "2024-09-02T11:58:01Z"
status: "True"
type: ImageReady
- lastUpdateTime: "2024-09-02T12:02:31Z"
status: "True"
type: RepoReady
- lastUpdateTime: "2024-09-02T12:18:44Z"
status: "True"
type: NodesPrepared
- lastUpdateTime: "2024-09-02T12:31:25Z"
status: "True"
type: SystemServicesUpgraded
- status: Unknown
type: NodesUpgraded
imageID: harvester-system/hvst-upgrade-szlg8
nodeStatuses:
harvester-c6phd:
state: Pre-drained
harvester-jkqhq:
state: Images preloaded
previousVersion: v1.3.1
...Check the node status.
Proceed to the next step if the following conditions are met:
- The status of the worker node is
NotReady. - The status of the management node is
Ready,SchedulingDisabled.
Example:
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
harvester-c6phd Ready,SchedulingDisabled control-plane,etcd,master 174m v1.28.12+rke2r1
harvester-jkqhq NotReady <none> 166m v1.27.13+rke2r1- The status of the worker node is
Check the pods on the worker node.
The issue exists in the cluster if the status of most pods is
Terminating.Example:
# Assume harvester-jkqhq is the worker node
$ kubectl get pods -A --field-selector spec.nodeName=harvester-jkqhq
NAMESPACE NAME READY STATUS RESTARTS AGE
cattle-fleet-local-system fleet-agent-6779fb5dd9-dkpjz 1/1 Terminating 0 18m
cattle-fleet-system fleet-agent-86db8d9954-qgcpq 1/1 Terminating 2 (18m ago) 61m
cattle-fleet-system fleet-controller-696d4b8878-ddctd 1/1 Terminating 1 (19m ago) 29m
cattle-fleet-system gitjob-694dd97686-s4z68 1/1 Terminating 1 (19m ago) 29m
cattle-provisioning-capi-system capi-controller-manager-6f497d5574-wkrnf 1/1 Terminating 0 20m
cattle-system cattle-cluster-agent-76db9cf9fc-5hhsx 1/1 Terminating 0 20m
cattle-system cattle-cluster-agent-76db9cf9fc-dnr6m 1/1 Terminating 0 20m
cattle-system harvester-cluster-repo-7458c7c69d-p982g 1/1 Terminating 0 27m
cattle-system rancher-7d65df9bd4-77n7w 1/1 Terminating 0 31m
cattle-system rancher-webhook-cfc66d5d7-fd6gm 1/1 Terminating 0 28m
harvester-system harvester-85ff674986-wxkl4 1/1 Terminating 0 26m
harvester-system harvester-load-balancer-54cd9754dc-cwtxg 1/1 Terminating 0 20m
harvester-system harvester-load-balancer-webhook-c8699b786-x6clw 1/1 Terminating 0 20m
harvester-system harvester-network-controller-manager-b69bf6b69-9f99x 1/1 Terminating 0 178m
harvester-system harvester-network-controller-vs4jg 1/1 Running 0 178m
harvester-system harvester-network-webhook-7b98f8cd98-gjl8b 1/1 Terminating 0 20m
harvester-system harvester-node-disk-manager-tbh4b 1/1 Running 0 26m
harvester-system harvester-node-manager-7pqcp 1/1 Running 0 178m
harvester-system harvester-node-manager-webhook-9cfccc84c-68tgp 1/1 Running 0 20m
harvester-system harvester-node-manager-webhook-9cfccc84c-6bbvg 1/1 Running 0 20m
harvester-system harvester-webhook-565dc698b6-np89r 1/1 Terminating 0 26m
harvester-system hvst-upgrade-szlg8-apply-manifests-4rmjw 0/1 Completed 0 33m
harvester-system virt-api-6fb7d97b68-cbc5m 1/1 Terminating 0 20m
harvester-system virt-api-6fb7d97b68-gqg5c 1/1 Terminating 0 23m
harvester-system virt-controller-67d8b4c75c-5qz9x 1/1 Terminating 0 24m
harvester-system virt-controller-67d8b4c75c-bdf8w 1/1 Terminating 2 (18m ago) 23m
harvester-system virt-handler-xw98h 1/1 Running 0 24m
harvester-system virt-operator-6c98db546-brgnx 1/1 Terminating 2 (18m ago) 26m
kube-system harvester-snapshot-validation-webhook-b75f94bcb-95zlb 1/1 Terminating 0 20m
kube-system harvester-snapshot-validation-webhook-b75f94bcb-xfrmf 1/1 Terminating 0 20m
kube-system harvester-whereabouts-tdr5g 1/1 Running 1 (178m ago) 178m
kube-system helm-install-rke2-ingress-nginx-4wt4j 0/1 Terminating 0 15m
kube-system helm-install-rke2-metrics-server-jn58m 0/1 Terminating 0 15m
kube-system kube-proxy-harvester-jkqhq 1/1 Running 0 178m
kube-system rke2-canal-wfpch 2/2 Running 0 178m
kube-system rke2-coredns-rke2-coredns-864fbd7785-t7k6t 1/1 Terminating 0 178m
kube-system rke2-coredns-rke2-coredns-autoscaler-6c87968579-rg6g4 1/1 Terminating 0 20m
kube-system rke2-ingress-nginx-controller-d4h25 1/1 Running 0 178m
kube-system rke2-metrics-server-7f745dbddf-2mp5j 1/1 Terminating 0 20m
kube-system rke2-multus-fsp94 1/1 Running 0 178m
kube-system snapshot-controller-65d5f465d9-5b2sb 1/1 Terminating 0 20m
kube-system snapshot-controller-65d5f465d9-c264r 1/1 Terminating 0 20m
longhorn-system backing-image-manager-c16a-7c90 1/1 Terminating 0 54m
longhorn-system csi-attacher-5fbd66cf8-674vc 1/1 Terminating 0 20m
longhorn-system csi-attacher-5fbd66cf8-725mn 1/1 Terminating 0 20m
longhorn-system csi-attacher-5fbd66cf8-85k5d 1/1 Terminating 0 20m
longhorn-system csi-provisioner-5b6ff8f4d4-97wsf 1/1 Terminating 0 20m
longhorn-system csi-provisioner-5b6ff8f4d4-cbpm9 1/1 Terminating 0 20m
longhorn-system csi-provisioner-5b6ff8f4d4-q7z58 1/1 Terminating 0 19m
longhorn-system csi-resizer-74c5555748-6rmbf 1/1 Terminating 0 20m
longhorn-system csi-resizer-74c5555748-fw2cw 1/1 Terminating 0 20m
longhorn-system csi-resizer-74c5555748-p4nph 1/1 Terminating 0 20m
longhorn-system csi-snapshotter-6bc4bcf4c5-6858b 1/1 Terminating 0 20m
longhorn-system csi-snapshotter-6bc4bcf4c5-cqkbw 1/1 Terminating 0 20m
longhorn-system csi-snapshotter-6bc4bcf4c5-mkqtg 1/1 Terminating 0 20m
longhorn-system engine-image-ei-b0369a5d-2t4k4 1/1 Running 0 178m
longhorn-system instance-manager-a5bd20597b82bcf3ba9d314620b7e670 1/1 Terminating 0 178m
longhorn-system longhorn-csi-plugin-x6bdg 3/3 Running 0 178m
longhorn-system longhorn-driver-deployer-85cf4b4849-5lc52 1/1 Terminating 0 20m
longhorn-system longhorn-loop-device-cleaner-hhvgv 1/1 Running 0 178m
longhorn-system longhorn-manager-5h2zw 1/1 Running 0 178m
longhorn-system longhorn-ui-6b677889f8-hrg8j 1/1 Terminating 0 20m
longhorn-system longhorn-ui-6b677889f8-w5hng 1/1 Terminating 0 20m
To resolve the issue, you must restart the rke2-agent service on the worker node.
# On the worker node
sudo systemctl restart rke2-agent.service
The upgrade should resume after the rke2-agent service is fully restarted.
This issue occurs because the agent load balancer on the worker node is unable to connect to the API server on the management node after the rke2-server service is restarted. Because the rke2-server service can be restarted multiple times when nodes are upgraded, the upgrade process is likely to become stuck again. You may need to restart the rke2-agent service multiple times.
To determine if the agent load balancer is functioning, run the following commands:
# On the management node, check if the `rke2-server` service is running.
sudo systemctl status rke2-server.service
# On the worker node, check if the agent load balancer is functioning.
sudo /var/lib/rancher/rke2/bin/kubectl --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig get nodes
If the kubectl command does not return a response, the kubelet is unable to access the API server via the agent load balancer. You must restart the rke2-agent service.
For more information, see Issue #6432.