Available as of v0.3.0
Harvester has provided a built-in monitoring integration using Prometheus. Monitoring is automatically enabled during the Harvester installations.
Dashboard page, users can view the cluster metrics and top 10 most used VM metrics respectively.
Also, users can click the Grafana dashboard link to view more dashboards on the Grafana UI.
Only admin users are able to view the cluster dashboard metrics.
VM Detail Metrics¶
For VMs, you can view VM metrics by clicking on the
VM details page > VM Metrics.
Memory Usage is calculated based on
(1 - free/total) * 100%, not
(used/total) * 100%.
For example, in a Linux OS, the
free -h command outputs the current memory statistics as follows
$ free -h total used free shared buff/cache available Mem: 7.7Gi 166Mi 4.6Gi 1.0Mi 2.9Gi 7.2Gi Swap: 0B 0B 0B
Memory Usage is
(1 - 4.6/7.7) * 100%, roughly
How to Configure Monitoring Settings¶
Available as of v1.0.1
Monitoring has several components that help to collect and aggregate metric data from all Nodes/Pods/VMs. The resources required for monitoring depend on your workloads and hardware resources. Harvester sets defaults based on general use cases, and you can change them accordingly.
Resources Settings can be configured for the following components:
- Prometheus Node Exporter(UI configurable as of v1.0.2)
Advanced Settings page, you can view and change the resources settings as follows:
Navigate to settings page, find
Show harvester-monitoringto view the current values.
⋮ > Edit Settingto set a new value.
Monitoringresource will be restarted within a few seconds. Please be aware that the reboot can take some time to reload previous data.
The most frequently used option is the memory setting:
Requested Memoryis the minimum memory required by the
Monitoringresource. The recommended value is about 5% to 10% of the system memory of one single management node. A value less than 500Mi will be denied.
Memory Limitis the maximum memory that can be allocated to a
Monitoringresource. The recommended value is about 30% of the system's memory for one single management node. When the
Monitoringreaches this threshold, it will automatically restart.
Depending on the available hardware resources and system loads, you may change the above settings accordingly.
If you have multiple management nodes with different hardware resources, please set the value of Prometheus based on the smaller one.
When an increasing number of VMs get deployed on one node, the
prometheus-node-exporter pod might get killed due to OOM(out of memory). In that case, you should increase the value of
To update those values, you can also use the CLI command with:
$kubectl edit managedchart rancher-monitoring -n fleet-local.
For Harvester version
>= v1.0.1, the related path and default value are:
# Prometheus configs spec.values.prometheus.prometheusSpec.resources.limits.cpu: 1000m spec.values.prometheus.prometheusSpec.resources.limits.memory: 2500Mi spec.values.prometheus.prometheusSpec.resources.requests.cpu: 750m spec.values.prometheus.prometheusSpec.resources.requests.memory: 1750Mi --- # node exporter configs spec.values.prometheus-node-exporter.resources.limits.cpu: 200m spec.values.prometheus-node-exporter.resources.limits.memory: 180Mi spec.values.prometheus-node-exporter.resources.requests.cpu: 100m spec.values.prometheus-node-exporter.resources.requests.memory: 30Mi
<= v1.0.0, the related path and default value are not specified in the
managedchart rancher-monitoring, you need to add them accordingly.
For Monitoring support and troubleshooting, please refer to the troubleshooting page .