Prometheus Alerting Display and Templates

[CannonLock]

Pelican Service:

• Director
• Origin
• Cache

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Create a UI item to display and provide templated Alerts for prometheus that can be enabled. Each template will have a description as well as the actual alert trigger equation. If it is easy add a way to adjust the actual alert tolerances/triggers so that they can be more broadly available.

Currently thinking of things like % of failures, # of threads, GBs transferred, slow request #... Going to reach out to system admins to get more ideas of things they saw that correlates with issues.

[CannonLock]

One thing I want to stress is a v1 goal for this will be easy tweaking of the triggers so that people don't automatically turn them off if they are constantly triggering and potentially adding some guidance for what values are good.

Another idea is that for the first round we have smart triggers that are determined by previous context. So no less "Trigger as 40mbps transfers" and more "Trigger at 2 deviations above existing max load average".