Skip to main content
Version: v1.9 (Dev)

Longhorn V2 Data Engine

The Longhorn V2 Data Engine harnesses the power of the Storage Performance Development Kit (SPDK) to significantly reduce I/O latency while boosting IOPS and throughput. The result is a high-performance storage solution that is capable of meeting diverse workload demands.

caution

The Longhorn V2 Data Engine is a Technical Preview feature. Explore the feature extensively before using in production environments.

Prerequisites​

Every node with an active Longhorn V2 Data Engine requires the following dedicated resources:

  • 1 CPU core for use by the Longhorn instance-manager pod
  • 2 GiB RAM (allocated as 1024 × 2 GiB huge pages)
  • At least one local NVMe disk for data storage

Limitations​

The Longhorn V2 Data Engine currently does not support the following operations:

  • Backing image creation and usage
  • Volume encryption

Using the Longhorn V2 Data Engine​

The Longhorn V2 Data Engine is only available for newly created volumes and images. Existing volumes, virtual machine images and virtual machine root volumes will continue to use the V1 Data Engine.

  1. On the Harvester UI, go to Advanced > Settings.

  2. Set longhorn-v2-data-engine-enabled to true.

Harvester automatically loads the kernel modules required by the Longhorn V2 Data Engine, and attempts to allocate 1024 × 2 MiB-sized huge pages (for example, 2 GiB of RAM) on all nodes.

Changing this setting automatically restarts RKE2 on all nodes but does not affect running virtual machine workloads.

tip

If you encounter error messages that include the phrase "not enough hugepages-2Mi capacity", allow some time for the error to be resolved. If the error persists, reboot the affected nodes.

If you do not plan to use hugepages or want to allocate a custom memory size, adjust the longhorn-v2-data-engine-hugepage-enabled and longhorn-v2-data-engine-memory-size settings, preferably before enabling the Longhorn V2 Data Engine. Disabling hugepages is currently supported only for Longhorn V2 disks that use the AIO bdev driver (the default in Harvester v1.9.0). Do not disable hugepages when using the NVMe or virtio bdev driver.

To disable the Longhorn V2 Data Engine on specific nodes (for example, nodes with less processing and memory resources), go to the Hosts screen and add the following label to the target nodes:

  • label: node.longhorn.io/disable-v2-data-engine
  • value: true
  1. Go to the Hosts screen, and then add extra disks to each node as described in Multi-disk Management.

Set the Provisioner of each extra disk to Longhorn V2 (CSI).

important

The default Longhorn disk driver for newly added Longhorn V2 disks in Harvester is aio. This driver uses a Linux block device path and avoids the SPDK NVMe VFIO path.

  1. Go to Advanced > Storage Classes, and then add a new StorageClass as described in Creating a StorageClass.

Set the Provisioner to Longhorn V2 (CSI).

  1. Use the new StorageClass:
    • When creating new volumes (either on the Volumes screen or during virtual machine creation)
    • When creating images on the Images screen

Volumes and images created using the new StorageClass are backed by the Longhorn V2 Data Engine.

Migrating Longhorn V2 Disks from NVMe to AIO​

Before migrating a disk, verify that all affected volumes are healthy and that the remaining Longhorn V2 disks have sufficient free space to accommodate rebuilt replicas during disk removal.

note

In Harvester v1.9.0 and later, newly created Longhorn V2 disks use the aio disk driver by default. Existing Longhorn V2 disks retain their current driver configuration until you manually remove and add them again.

To migrate a disk, perform the following steps:

  1. Remove the affected Longhorn V2 disk.

  2. Add the Longhorn V2 disk again.

  3. Verify that the migrated disk uses aio.

    kubectl -n longhorn-system get blockdevices.harvesterhci.io BLOCKDEVICE_NAME -o yaml

    The matching BlockDevice custom resource should have the following values:

    spec:
      provisioner:
        longhorn:
          diskDriver: aio
          engineVersion: LonghornV2
    status:
      provisionPhase: Provisioned
      state: Active

Known Issues​

I/O Operations May Stall on ARM64 Platforms​

On ARM64 platforms, Longhorn V2 disks using the SPDK NVMe bdev driver can experience stalled I/O operations under active workloads. For more information, see issue #10710.

For information about the workaround, see Migrating Longhorn V2 Disks from NVMe to AIO.

Upgrading from Harvester v1.4.x​

In Harvester v1.4 (which uses Longhorn v1.7), V2 volumes did not support live migration, nor could the V2 data engine be used for virtual machine images, which meant VM boot volumes could not use the V2 Data Engine.

Starting with Harvester v1.5.0 and Longhorn v1.8.1, these limitations are removed, but only for volumes and images that are created after the system is upgraded. Any V2 StorageClass created with Harvester v1.4.0 will have the migratable option set to "false", and like other StorageClass properties, this cannot be changed once set. Similarly, any existing V2 volumes will remain non-migratable after the upgrade. If you have used the V2 data engine on Harvester v1.4, and later upgrade to Harvester v1.5, you will need to create a new V2 StorageClass, which will default to having migratable set to "true". Volumes and images created using this Storage Class will be live-migratable.

important

  • If you are using the SPDK AIO bdev driver (i.e. if your disks were added using /dev/sd* device paths), V2 volumes created before the upgrade will be unusable after upgrading, and cannot be recovered. For more details see https://github.com/longhorn/longhorn/issues/10461.

  • If you are using the SPDK NVMe bdev driver (i.e. your disks were added using /dev/nvme* device paths), V2 volumes created before the upgrade will function after the upgrade, but will continue to use the Longhorn v1.7.x engine. As mentioned above, these volumes will remain non-migratable, but it is possible to export the data and create new migratable volumes (see below for details).

  • All virtual machines with V2 volumes attached need to be stopped before upgrading to Harvester v1.5.0. If there are any V2 volumes active during the upgrade, the process will stall part way through "upgrading system services". The logs of the apply-manifests pod will show repeated messages similar to the following:

    instance-manager (aio)(v2) (image=longhornio/longhorn-instance-manager:v1.8.1) state is not running on node harvester-node-0, will retry...

    Stopping all Virtual Machines that are using V2 volumes will allow the upgrade to proceed.

If you have existing virtual machines with V2 non-migratable volumes attached, and you are using the SPDK NVMe bdev driver (i.e. your disks were added using /dev/nvme* device paths), it's possible to transition to live-migratable volumes as follows:

  1. Stop the Virtual Machine
  2. For each V2 volume attached to the Virtual Machine, use the Export Image option to export that volume to an image that uses your new V2 StorageClass (with migratable set to "true"). This may take a while, depending on how much data needs to be copied.
  3. Once complete, edit the Virtual Machine and on the Volumes tab:
    • Remove the existing V2 volume(s).
    • Use the "Add VM Image" button to add the image(s) that were exported in the previous step.
  4. Start the VM. Again, this may take a while depending on how much data needs to be copied.
  5. Delete the original volume(s) and the exported image(s) as these should no longer be necessary to keep around.