fix grafana dashboard and clarify dashboard usage more clearly.

[jiangsanyin]

Signed-off-by: jiangsanyin [email protected]

What type of PR is this?
/kind bug

What this PR does / why we need it:
fix grafana dashboard and clarify dashboard usage more clearly. Thanks "fangfenghuang (https://github.com/fangfenghuang)" for your help

Which issue(s) this PR fixes:
Fixes #498 #468

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

[yxl]

Should "nvidia.com/vgpu" be "nvidia.com/gpu"?

[jiangsanyin]

Should "nvidia.com/vgpu" be "nvidia.com/gpu"?

I forgot to explain it, it depends on our own case.
In order to distinguish from “nvidia.com/gpu” in nvidia-device-plugin, I used resourceName parameter and setted it's value to "nvidia.com/vgpu", such as: helm install hami hami-charts/hami --set resourceName=nvidia.com/vgpu --set scheduler.kubeScheduler.imageTag=v1.23.10 -n kube-system

[wawa0210]

@fangfenghuang Can you help review this pr?

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Flag	Coverage Δ
unittests	27.09% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

[fangfenghuang]

fix some http url

[fangfenghuang]

It is best to place the referenced images in the ../imgs/

[jiangsanyin]

ok, changes has been made.

[Nimbus318]

@jiangsanyin
I have followed the installation instructions as described in the documentation, but encountered a minor issue, which I also mentioned previously in Issue #498. By default, the dcgm-exporter only includes the Hostname label. To match the current Grafana dashboard configuration, it's necessary to add a node_name relabeling configuration when installing dcgm-exporter

https://github.com/NVIDIA/dcgm-exporter/blob/b97b7633e3f39f7a537bd77561cc0ec0c2dca3f5/deployment/values.yaml#L117C3-L117C18

This relabeling should be consistent with the configurations for hami-device-plugin-svc-monitor and hami-scheduler-svc-monitor

It would be helpful to include this information in the documentation, as users unfamiliar with the Prometheus stack may struggle to configure everything correctly on the first attempt

[jiangsanyin]

@jiangsanyin I have followed the installation instructions as described in the documentation, but encountered a minor issue, which I also mentioned previously in Issue #498. By default, the dcgm-exporter only includes the Hostname label. To match the current Grafana dashboard configuration, it's necessary to add a node_name relabeling configuration when installing dcgm-exporter

https://github.com/NVIDIA/dcgm-exporter/blob/b97b7633e3f39f7a537bd77561cc0ec0c2dca3f5/deployment/values.yaml#L117C3-L117C18

This relabeling should be consistent with the configurations for hami-device-plugin-svc-monitor and hami-scheduler-svc-monitor

It would be helpful to include this information in the documentation, as users unfamiliar with the Prometheus stack may struggle to configure everything correctly on the first attempt

Have you created and applied the ServiceMonitor as depicted in dashboard.md or dashboard_cn.md？node_name is added after this is done.
#Create the file hami-device-plugin-svc-monitor.yaml
root@controller01:~# cat hami-device-plugin-svc-monitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: hami-device-plugin-svc-monitor
namespace: kube-system
spec:
selector:
matchLabels:
app.kubernetes.io/component: hami-device-plugin
namespaceSelector:
matchNames:
- "kube-system"
endpoints:

• path: /metrics
  port: monitorport
  interval: "15s"
  honorLabels: false
  relabelings:
  • sourceLabels: [__meta_kubernetes_endpoints_name]
    regex: hami-.*
    replacement: $1
    action: keep
  • sourceLabels: [__meta_kubernetes_pod_node_name]
    regex: (.*)
    targetLabel: node_name
    replacement: ${1}
    action: replace
  • sourceLabels: [__meta_kubernetes_pod_host_ip]
    regex: (.*)
    targetLabel: ip
    replacement: $1
    action: replace

#apply the file hami-device-plugin-svc-monitor.yaml
root@controller01:~# kubectl apply -f hami-device-plugin-svc-monitor.yaml

[Nimbus318]

@jiangsanyin
Both are correct. What I meant is that you might have forgotten to include the explanation for the relabel configuration of dcgm-exporter. By default, dcgm-exporter only includes the Hostname label

It’s important to document this configuration to ensure it aligns with the relabeling setup for hami-device-plugin-svc-monitor. Without this explanation, users may miss adding the necessary node_name relabeling when setting up dcgm-exporter