COS / More alerts

[gustavosr98]

Steps to reproduce

Follow tutorial for COS integration

Expected behavior

A few alerts I would like to see

• Machine is down
• Machine is up but service is down
• Cluster is not writable
• Cluster will not be writable if I lose one more node
• Query latency goes over X ms
• Number of connections is close to max connections limit

And any similar related to degradation of to prevent the system stops working as expected

Actual behavior

No errors in the alerts, just two alerts

Versions

Juju 3.5.4

data-integrator                    active      1  data-integrator           latest/stable   41  no
grafana-agent                      active      3  grafana-agent             latest/edge    299  no       tracing: off
mongodb                            active      3  mongodb                   6/stable       199  no
self-signed-certificates           active      1  self-signed-certificates  latest/stable  155  no

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/DPE-6073.

This message was autogenerated

[MiaAltieri]

Hi @gustavosr98 I have just opened a PR(#521) for these desired alerts. Thanks for taking the time to outline your wishlist! <3

I've gone ahead and added most of them. The ones I did not add in that PR are:

• Machine is down
• Machine is up but service is down
  and
• Query latency goes over X ms

The first two I did not add since they are already there (screenshots 1+2) the third I do not believe is possible since I don't think grafana supports alerts based on user provided input (i.e. X), if you have a specific latency that you have in mind let me know and I can implement that for you ASAP

Please note that #1 will be further improved on o11y end since they are currently undergoing work for it