Expose Prometheus Endpoint and Alertmanager Service on Admin Gateway

[joelmccoy]

Is your feature request related to a problem? Please describe.

As an SRE/Infra admin I would like to be able to easily reach the Alertmanager and Prometheus GUIs. As an SRE I should be able to view and silence alerts in Alertmanager via the GUI, and I also would like to see the health of prometheus scrapers in the GUI. As I understand it currently, the only way to do this is to run a kubectl port forward to the respective service (but this requires write creds to the kubernetes cluster). Both of these services feel like they should be exposed on the Admin gateway. I suppose this might have not been done initially due to auth reasons? But now that we have authservice, I think it is reasonable to expose these services by default on the Admin gateway behind Authservice.

Even if we chose to not expose these by default in uds-core, it could at least made configurable by overrides (which I don't think is possible here right now)

Describe the solution you'd like

• Given A deploy of default UDS Core
• When when a user with access to the Admin gateway and appropriate permissions
• Then can reach the alertmanager GUI at (alerts.uds.dev) and the GUI at prometheus (prometheus.uds.dev). (open to other names for these endpoints)

Describe alternatives you've considered

• kubectl port-forward to the service (this is ok for now, but requires appropriate kubernetes permissions and feels a bit heavy handed)

Additional context

Happy to toss up a PR for this if we believe that this is an ok route to take (either expose these by default or allow them to be configured).

[joelmccoy]

A tangential issue to this as I am trying to setup alertmanager to send alerts to out to slack. Currently the netpols do not allow this. I believe it would also be beneficial to have an optional value to set up these custom netpols for this particular package to account for scenarios like this.