DataDog Dashboards

[gatesn]

Wanted to put this up to get comments on structure before spending time on tests :)

[changelog-app]

Generate changelog in changelog/@unreleased

Type

• Feature
• Improvement
• Fix
• Break
• Deprecation
• Manual task
• Migration

Description

DataDog Dashboards

Wanted to put this up to get comments on structure before spending time on tests :)

Check the box to generate changelog(s)

• Generate changelog entry

[carterkozak]

We took a look at something like this, but decided to start looking at implementing alerts first since they're less complex, and there's more value to gain by making sure they're valid.
If you can make dashbaords work well, I'm all for it (note: I haven't reviewed this PR yet, I might not be able to put time into it until break or next year)! If we add support for them, we should make sure to support both datadog and prometheus dashboards from the same schema, which may require us to make design trade-offs for a reasonable support burden.

[gatesn]

I'm not convinced that alerts would be any less complex - in particular, expressing a query API that can be made compatible with both DataDog and Prometheus is the hard piece of both Dashboards and Monitoring.

Just added very basic Prom+Grafana support to this PR. There's a lot of cleaning up to do, but I think the idea could work.

[gatesn]

TODO: maybe look at re-using this Grafana client instead of building our own immutables model: https://github.com/appnexus/grafana-api-java-client

[glynternet]

I've done a fair chunk of work regarding dashboards in datadog and grafana/prometheus.
We have a some tooling to generate grafana dashboards that I'm hoping to eventually use the dashboard schema as the intermediate stage of.
@gatesn, it would be good to sync on this to check we are aligning and not duplicating work.