Argo-workflows not outputting all default metrics

[johnfountainsomos]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

Describe the bug

I've recently installed argo workflows via the official helm chart here: https://github.com/argoproj/argo-helm/tree/argo-workflows

The documentation here: https://argo-workflows.readthedocs.io/en/release-3.6/metrics/#queue_depth_gauge says that there is a metric called queue_depth_gauge that does not seem to exist in the metrics output.

example output:

# curl http://10.101.2.230:9090/metrics | grep queue
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 44117    0 44117    0     0  9945k      0 --:--:-- --:--:--# HELP argo_workflows_queue_longest_running Longest running worker https://argo-workflows.readthedocs.io/en/release-3.6/metrics/#queue_longest_running
# TYPE argo_workflows_queue_longest_running gauge
argo_workflows_queue_longest_running{queue_name="cron_wf_queue"} 0
argo_workflows_queue_longest_running{queue_name="pod_cleanup_queue"} 0
argo_workflows_queue_longest_running{queue_name="workflow_archive_queue"} 0
argo_workflows_queue_longest_running{queue_name="workflow_queue"} 0
argo_workflows_queue_longest_running{queue_name="workflow_ttl_queue"} 0
# HELP argo_workflows_queue_unfinished_work Unfinished work time https://argo-workflows.readthedocs.io/en/release-3.6/metrics/#queue_unfinished_work
# TYPE argo_workflows_queue_unfinished_work gauge
argo_workflows_queue_unfinished_work{queue_name="cron_wf_queue"} 0
argo_workflows_queue_unfinished_work{queue_name="pod_cleanup_queue"} 0
argo_workflows_queue_unfinished_work{queue_name="workflow_archive_queue"} 0
argo_workflows_queue_unfinished_work{queue_name="workflow_queue"} 0
argo_workflows_queue_unfinished_work{queue_name="workflow_ttl_queue"} 0
 --:--:-- argo_workflows_workers_busy_count{worker_type="cron_wf_queue"} 0
14argo_workflows_workers_busy_count{worker_type="pod_cleanup_queue"} 0
.0argo_workflows_workers_busy_count{worker_type="workflow_archive_queue"} 0
Margo_workflows_workers_busy_count{worker_type="workflow_queue"} 0

Related helm chart

argo-workflows

Helm chart version

latest

To Reproduce

Install argo workflows via helm chart

collect metrics at /metrics endpoint

Expected behavior

the output should contain the metric: queue_depth_gauge

Screenshots

No response

Additional context

No response

Version(s)

latest

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

unrelated to workflow manifest. only metrics are affected.

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded