Skip to main content
Version: v1.9 (Dev)

Upgrade from v1.7.x to v1.7.y

General Information​

An Upgrade button appears on the Dashboard screen whenever a new Harvester version that you can upgrade to becomes available. For more information, see Start an upgrade.

For information about upgrading Harvester in air-gapped environments, see Prepare an air-gapped upgrade.

Known Issues​

1. Upgrade Is Stuck in the "Post-draining" State​

During the node draining process, the upgrade-repo deployment may get stuck when its Longhorn volume remains in the "Attaching" state. This causes the upgrade flow to stall in the "Post-draining" phase, since the post-drain jobs wait for the upgrade-repo deployment to become ready before proceeding.

The workaround is to delete the Longhorn replica of the upgrade-repo volume on the drained node. This allows the volume to attach and the upgrade flow to continue. Note that any node being drained during the upgrade may encounter this issue, so this workaround may need to be applied whenever it occurs.

Related issues: #9597 and #12226

2. Unnecessary live-migrations during the upgrade​

Harvester v1.6.x enables CPU and memory hot-plugging for virtual machines through KubeVirt's LiveMigrate workload update strategy. However, when the KubeVirt operator is upgraded, this feature triggers simultaneous live-migration of all running VMs to update their virt-launcher pods immediately. This mass migration can overwhelm cluster resources and cause performance degradation.

To prevent this issue, you can temporarily disable the LiveMigrate workload update method before the upgrade and re-enable it after the upgrade completes. VMs will migrate naturally during node upgrades, allowing the virt-launcher image to be updated gradually.

note

Starting from v1.8.0, this process is handled automatically. The workaround described below is only necessary when upgrading to a version before v1.8.0.

Please see the instruction on this page.

3. Upgrade Stalls During Image Preloading Due to System-Upgrade-Controller Failing to Retry a Plan​

During Phase 2 (Preload Container Images) of the upgrade, Harvester creates a system-upgrade-controller (SUC) plan for each node to preload the container images required for the new release. If SUC fails to reschedule a plan job after a transient failure, the affected node's plan remains stuck in the applying state, stalling the upgrade indefinitely.

This is an intermittent issue: once a plan job fails and is deleted (by the default job TTL of 900 seconds), SUC may stop rescheduling it for the affected node.

note

Although the prepare stage is the most common upgrade stage where this issue manifests, the same SUC behavior can also affect other stages that rely on SUC plans.

Symptoms​

  • The upgrade has shown no progress for an extended period (typically more than 30 minutes) while in the image preloading phase.

  • No job has been created for the affected node's prepare plan in the cattle-system namespace:

    kubectl get jobs -n cattle-system | grep prepare

  • One or more SUC prepare plans are stuck in the applying state:

    kubectl get plans.upgrade.cattle.io -n cattle-system | grep prepare

    Check the status of the stuck plan (replace <plan-name> with the actual plan name from the previous command):

    kubectl get plans.upgrade.cattle.io <plan-name> -n cattle-system -o yaml | yq .status

    If the issue is present, the affected node appears under applying, and the Complete condition is False:

    applying:
    - <node-name>
    conditions:
    - lastUpdateTime: "..."
      reason: SyncJob
      status: "False"
      type: Complete

Workaround​

Restart the system-upgrade-controller deployment to force SUC to reconcile all plans and reschedule any stuck jobs:

kubectl rollout restart deployment/system-upgrade-controller -n cattle-system

After the restart, SUC reschedules the plan job for the affected node. The upgrade should resume automatically within a few minutes.

Related issue: #9880

4. Upgrade Stuck in Crash Loop After CDI Importer Pod Is OOM-Killed​

During Phase 1 (Provision an Upgrade Repository Virtual Machine), the Containerized Data Importer (CDI) downloads the target ISO file and converts it to a raw disk image using qemu-img convert -t writeback, which buffers converted data in memory. On slow destination storage, this buffer can grow until it exceeds the CDI importer pod's memory limit, causing the pod to be OOM-killed.

This issue stems from the CDI configuration running on the source cluster rather than the target release. Clusters running v1.7.0 or v1.7.1 use the default importer pod memory limit of 600M, which is prone to this failure. The limit was raised to 2G in v1.7.2, but slow destination storage and large ISO images can still drive memory consumption past this threshold.

After the importer pod is OOM-killed, its /data PVC is not cleaned up automatically (unlike the /scratch PVC). The partially converted disk image remains on the volume, causing subsequent retries to miscalculate available storage space and fail immediately, leaving the pod crash-looping indefinitely.

Symptoms​

  • The importer-prime-* pod in the harvester-system namespace is in a CrashLoopBackOff state.

  • Node kernel logs indicate an oom-kill event for the virt-cdi-import and qemu-img processes, typically occurring during the initial crash:

    Memory cgroup out of memory: Killed process ... (virt-cdi-import) ...
    Memory cgroup out of memory: Killed process ... (qemu-img) ...

  • Subsequent restarts do not trigger an OOM-kill. Instead, the pod fails with an error message similar to the following:

    Unable to convert source data to target format: virtual image size <X> is larger than the reported available storage <Y>. A larger PVC is required

Workaround​

  1. Stop the ongoing upgrade.

    This action deletes the Upgrade CR along with its associated DataVolume and PVCs, clearing the stale /data content.

  2. Edit the harvester ManagedChart resource.

    kubectl edit managedchart.management.cattle.io harvester -n fleet-local

  3. Increase the CDI importer pod's memory limit beyond the default value.

    Under spec.values, configure a higher value for the cdi.spec.config.podResourceRequirements.limits.memory field based on available node memory. Slow storage backends and large images may require a higher allocation.

    If the cdi key (or any part of its nested path) does not exist under spec.values, add the missing structure.

    spec:
      values:
        cdi:
          spec:
            config:
              podResourceRequirements:
                limits:
                  memory: 4G
    caution

    Only modify the cdi.spec.config.podResourceRequirements.limits.memory field. Do not modify or delete any other existing fields in the harvester ManagedChart resource.

    The YAML snippet is an excerpt, not a full resource manifest.

  4. Verify that the CDI CR reflects the change:

    kubectl get cdi cdi -o jsonpath='{.spec.config.podResourceRequirements.limits.memory}{"\n"}'

  5. Restart the upgrade.

  6. After the upgrade completes successfully, remove the podResourceRequirements override you added to the harvester ManagedChart resource in step 3.

    The version you upgraded to already includes the memory-limit fix, so the override is no longer needed.

Related issues: #11143 and #10056