Skip to main content
Version: v1.5 (dev)

Upgrade from v1.3.2 to v1.4.0

General information​

An Upgrade button appears on the Dashboard screen whenever a new Harvester version that you can upgrade to becomes available. For more information, see Start an upgrade.

For air-gapped environments, see Prepare an air-gapped upgrade.

Known issues​

1. A VM with a container disk can't be migrated which makes the upgrade stuck in pre-drain status​

tip

Manually stop the VMs to continue the upgrade process.

When upgrading from v1.3.2 to v1.4.0, the upgrade process may become stuck if a VM with a container disk cannot be migrated. There is some limitation of live migration.

For more information, see Issue #7005.

2. Upgrade stuck on waiting for Harvester bundle​

When upgrading from v1.3.2 to v1.4.0, the upgrade process may become stuck on waiting for the Harvester bundle to become ready. This issue is caused by a race condition when the Fleet agent (fleet-agent) is redeployed.

The following error messages indicate that the issue exists.

> kubectl get bundles -n fleet-local
NAME                                          BUNDLEDEPLOYMENTS-READY   STATUS
mcc-harvester                                 0/1                       ErrApplied(1) [Cluster fleet-local/local: encountered 2 deletion errors. First is: admission webhook "validator.harvesterhci.io" denied the request: Internal error occurred: no route match found for DELETE /v1, Kind=Secret harvester-system/sh.helm.release.v1.harvester.v2]
mcc-harvester-crd                             0/1                       ErrApplied(1) [Cluster fleet-local/local: admission webhook "validator.harvesterhci.io" denied the request: Internal error occurred: no route match found for DELETE /v1, Kind=Secret harvester-system/sh.helm.release.v1.harvester-crd.v1]

You can run the following script to fix the issue.

#!/bin/bash

patch_fleet_bundle() {
  local bundleName=$1
  local generation=$(kubectl get -n fleet-local bundle ${bundleName} -o jsonpath='{.spec.forceSyncGeneration}')
  local new_generation=$((generation+1))
  patch_manifest="$(mktemp)"
  cat > "$patch_manifest" <<EOF
{
  "spec": {
    "forceSyncGeneration": $new_generation
  }
}
EOF
  echo "patch bundle to new generation: $new_generation"
  kubectl patch -n fleet-local bundle ${bundleName}  --type=merge --patch-file $patch_manifest
  rm -f $patch_manifest
}

echo "removing harvester validating webhook"
kubectl delete validatingwebhookconfiguration harvester-validator

for bundle in mcc-harvester-crd mcc-harvester
do
  patch_fleet_bundle ${bundle}
done

echo "removing longhorn services"
kubectl delete svc longhorn-engine-manager -n longhorn-system --ignore-not-found=true
kubectl delete svc longhorn-replica-manager -n longhorn-system --ignore-not-found=true

3. Upgrade stuck on waiting for Fleet​

When upgrading from v1.3.2 to v1.4.0, the upgrade process may become stuck on waiting for Fleet to become ready. This issue is caused by a race condition when Rancher is redeployed.

Check the Harvester logs and Fleet history for the following indicators:

  • The manifest pod is stuck in the deployed status.
  • The upgrade is pending with a chart version that has been deployed.

Example:

> kubectl logs -n harvester-system -l harvesterhci.io/upgradeComponent=manifest
wait helm release cattle-fleet-system fleet fleet-104.0.2+up0.10.2 0.10.2 deployed

> helm history -n cattle-fleet-system fleet
REVISION    UPDATED                     STATUS          CHART                   APP VERSION DESCRIPTION
26          Tue Dec 10 03:09:13 2024    superseded      fleet-103.1.5+up0.9.5   0.9.5       Upgrade complete
27          Sun Dec 15 09:26:54 2024    superseded      fleet-103.1.5+up0.9.5   0.9.5       Upgrade complete
28          Sun Dec 15 09:27:03 2024    superseded      fleet-103.1.5+up0.9.5   0.9.5       Upgrade complete
29          Mon Dec 16 05:57:03 2024    deployed        fleet-103.1.5+up0.9.5   0.9.5       Upgrade complete
30          Mon Dec 16 05:57:13 2024    pending-upgrade fleet-103.1.5+up0.9.5   0.9.5       Preparing upgrade

You can run the following command to fix the issue.

helm rollback fleet -n cattle-fleet-system <last-deployed-revision>