Version: v1.1

Upgrade from v1.0.2 to v1.0.3

General information

The Harvester GUI Dashboard page should have an upgrade button to perform an upgrade. For more details please refer to start an upgrade.

For the air-gap env upgrade, please refer to prepare an air-gapped upgrade.

Known issues

1. Fail to download the upgrade image

Description

Downloading the upgrade image can't be complete or fails with an error.

Related issues

[BUG] failed to create upgrade image

Workaround

Delete the current upgrade and start over. Please see "Start over an upgrade".

2. An upgrade is stuck, a node is in "Pre-drained" state (case 1)

Description

Users might see a node is stuck at the Pre-drained state for a while (> 30 minutes).

This might be caused by instance-manager-r-* pod on node harvester-z7j2g can’t be drained. To verify the above case:

Check rancher server logs:

kubectl logs deployment/rancher -n cattle-system

Example output:

error when evicting pods/"instance-manager-r-10dd59c4" -n "longhorn-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod longhorn-system/instance-manager-r-10dd59c4
error when evicting pods/"instance-manager-r-10dd59c4" -n "longhorn-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod longhorn-system/instance-manager-r-10dd59c4
error when evicting pods/"instance-manager-r-10dd59c4" -n "longhorn-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod longhorn-system/instance-manager-r-10dd59c4
error when evicting pods/"instance-manager-r-10dd59c4" -n "longhorn-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.

Verify the pod longhorn-system/instance-manager-r-10dd59c4 is on the stuck node:

kubectl get pod instance-manager-r-10dd59c4 -n longhorn-system -o=jsonpath='{.spec.nodeName}'

Example output:

harvester-z7j2g

Check degraded volumes:

kubectl get volumes -n longhorn-system

Example output:

NAME                                       STATE      ROBUSTNESS   SCHEDULED   SIZE          NODE              AGE
pvc-08c34593-8225-4be6-9899-10a978df6ea1   attached   healthy      True        10485760      harvester-279l2   3d13h
pvc-526600f5-bde2-4244-bb8e-7910385cbaeb   attached   healthy      True        21474836480   harvester-x9jqw   3d1h
pvc-7b3fc2c3-30eb-48b8-8a98-11913f8314c2   attached   healthy      True        10737418240   harvester-x9jqw   3d
pvc-8065ed6c-a077-472c-920e-5fe9eacff96e   attached   healthy      True        21474836480   harvester-x9jqw   3d
pvc-9a40e5b9-543a-4c90-aafd-ac78b05d7599   attached   degraded     True        10737418240   harvester-x9jqw   2d23h
pvc-9a6539b8-44e5-430e-9b24-ea8290cb13b7   attached   healthy      True        53687091200   harvester-x9jqw   3d13h

Here we can see volume pvc-9a40e5b9-543a-4c90-aafd-ac78b05d7599 is degraded.

note

The user needs to check all degraded volumes one by one.

Check degraded volume’s replica state:

kubectl get replicas -n longhorn-system --selector longhornvolume=pvc-9a40e5b9-543a-4c90-aafd-ac78b05d7599 -o json | jq '.items[] | {replica: .metadata.name, healthyAt: .spec.healthyAt, nodeID: .spec.nodeID, state: .status.currentState}'

Example output:

{
  "replica": "pvc-9a40e5b9-543a-4c90-aafd-ac78b05d7599-r-15e31246",
  "healthyAt": "2022-07-25T07:33:16Z",
  "nodeID": "harvester-z7j2g",
  "state": "running"
}
{
  "replica": "pvc-9a40e5b9-543a-4c90-aafd-ac78b05d7599-r-22974d0f",
  "healthyAt": "",
  "nodeID": "harvester-279l2",
  "state": "running"
}
{
  "replica": "pvc-9a40e5b9-543a-4c90-aafd-ac78b05d7599-r-bc6f7fa5",
  "healthyAt": "",
  "nodeID": "harvester-x9jqw",
  "state": "stopped"
}

Here the only healthy replica is pvc-9a40e5b9-543a-4c90-aafd-ac78b05d7599-r-15e31246 and it’s on node harvester-z7j2g. So we can confirm the instance-manager-r-* pod resides on node harvester-z7j2g and avoids the drain.

Related issues

[BUG] Upgrade: longhorn-system can't be evicted

Workaround

We need to start the “Stopped” replica, from the previous example, the stopped replica’s name is pvc-9a40e5b9-543a-4c90-aafd-ac78b05d7599-r-bc6f7fa5.

Check the Longhorn manager log, we should see a replica waiting for the backing image. First, we need to get the manager's name:

kubectl get pods -n longhorn-system --selector app=longhorn-manager --field-selector spec.nodeName=harvester-x9jqw

Example output:

NAME                     READY   STATUS    RESTARTS   AGE
longhorn-manager-zmfbw   1/1     Running   0          3d10h

Get pod log:

kubectl logs longhorn-manager-zmfbw -n longhorn-system | grep pvc-9a40e5b9-543a-4c90-aafd-ac78b05d7599-r-bc6f7fa5

Example output:

(...)
time="2022-07-28T04:35:34Z" level=debug msg="Prepare to create instance pvc-9a40e5b9-543a-4c90-aafd-ac78b05d7599-r-bc6f7fa5"
time="2022-07-28T04:35:34Z" level=debug msg="Replica pvc-9a40e5b9-543a-4c90-aafd-ac78b05d7599-r-bc6f7fa5 is waiting for backing image harvester-system-harvester-iso-n7bxh downloading file to node harvester-x9jqw disk 3830342d-c13d-4e55-ac74-99cad529e9d4, the current state is in-progress" controller=longhorn-replica dataPath= node=harvester-x9jqw nodeID=harvester-x9jqw ownerID=harvester-x9jqw replica=pvc-9a40e5b9-543a-4c90-aafd-ac78b05d7599-r-bc6f7fa5
time="2022-07-28T04:35:34Z" level=info msg="Event(v1.ObjectReference{Kind:\"Replica\", Namespace:\"longhorn-system\", Name:\"pvc-9a40e5b9-543a-4c90-aafd-ac78b05d7599-r-bc6f7fa5\", UID:\"c511630f-2fe2-4cf9-97a4-21bce73782b1\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"632926\", FieldPath:\"\"}): type: 'Normal' reason: 'Start' Starts pvc-9a40e5b9-543a-4c90-aafd-ac78b05d7599-r-bc6f7fa5"

Here we can determine the replica is waiting for the backing image harvester-system-harvester-iso-n7bxh.

Get the disk file map from the backing image:

kubectl describe backingimage harvester-system-harvester-iso-n7bxh -n longhorn-system

Example output:

(...)
Disk File Status Map:
    3830342d-c13d-4e55-ac74-99cad529e9d4:
      Last State Transition Time:  2022-07-25T08:30:34Z
      Message:
      Progress:                    29
      State:                       in-progress
    3aa804e1-229d-4141-8816-1f6a7c6c3096:
      Last State Transition Time:  2022-07-25T08:33:20Z
      Message:
      Progress:                    100
      State:                       ready
    92726efa-bfb3-478e-8553-3206ad34ce70:
      Last State Transition Time:  2022-07-28T04:31:49Z
      Message:
      Progress:                    100
      State:                       ready

The disk file with UUID 3830342d-c13d-4e55-ac74-99cad529e9d4 has the state in-progress.

Next, we need to find backing-image-manager that contains this disk file:

kubectl get pod -n longhorn-system --selector=longhorn.io/disk-uuid=3830342d-c13d-4e55-ac74-99cad529e9d4

Example output:

NAME                              READY   STATUS    RESTARTS   AGE
backing-image-manager-c00e-3830   1/1     Running   0          3d1h

Restart the backing-image-manager by deleting its pod:

kubectl delete pod -n longhorn-system backing-image-manager-c00e-3830

3. An upgrade is stuck, a node is in "Pre-drained" state (case 2)

Description

Users might see a node is stuck at the Pre-drained state for a while (> 30 minutes).

Here are some steps to verify this issue has happened:

Visit the Longhorn GUI: https://{{VIP}}/k8s/clusters/local/api/v1/namespaces/longhorn-system/services/http:longhorn-frontend:80/proxy/#/volume (replace VIP with an appropriate value) and check degraded volumes. The degraded volume might contain a healthy replica only (with blue background) and the healthy replica resides on the "Pre-drained" node:
Hover the mouse to the red scheduled icon, the reason is toomanysnapshots:

Related issues

[BUG] Upgrade is stuck in "Pre-drained" state (Volume has too many system snapshots)

Workaround

In the "Snapshots and Backup" panel, toggle the "Show System Hidden" switch and delete the latest system snapshot (which is just before the "Volume Head").
The volume will continue rebuilding to resume the upgrade.

General information​

Known issues​

1. Fail to download the upgrade image​

2. An upgrade is stuck, a node is in "Pre-drained" state (case 1)​

3. An upgrade is stuck, a node is in "Pre-drained" state (case 2)​

General information

Known issues

1. Fail to download the upgrade image

2. An upgrade is stuck, a node is in "Pre-drained" state (case 1)

3. An upgrade is stuck, a node is in "Pre-drained" state (case 2)