Update metrics documentation

[binarycrayon]

Motivation

Update metrics documentation with an updated grafana example dashboard, and add some debugging tips, for issue #2677

Modifications

Changed sample production_metrics, added note to run server with --enable-metrics flag, also added debugging tips

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.

[zhaochenyang20]

This looks nice. I will ask ziliang for checking it. And will fix the style later. @shuaills Could you also review this? Thanks!

[zhaochenyang20]

@ziliangpeng @shuaills Thanks for reviewing this.

[ziliangpeng]

thanks for the change. generally looks good! i haven't spun up a server and containers to test, but nothing looks obviously wrong to me.

also, i recommend further automate metrics set up for users to improve experience. i bet most sglang users are ml engineers or ai researchers that might not have time to figure out prometheus / grafana like an infra engineer.
maybe we can have a mini project for the end to end automation for this.

[ziliangpeng]

is there a way to prepare a config file (or list of commands in Dockerfile) that gets the datasource set up, then supply it to the grafana container, and get the entire thing ready to use end to end?

I feel that requiring a manual step would be error prone and difficult for the team to troubleshoot for users. we're targeting users with no metrics knowledge (or no time to to spend on metrics set up) so "ready-to-use"ness is important.

[binarycrayon]

let me see what I can do with a grafana.ini

[ziliangpeng]

again, i think there's way to automate this by either grafana iac or making api calls, which we can get them done in a dockerfile for the grafana image, and make it ready-to-use.

creating variables could be even more non-trivial than adding datasource, for users who have no grafana experience. would be really nice to have this automated.

[binarycrayon]

In this case I feel it's better to be more explicit and less automated. Setting up variables and seeing previewed values is a also good indication that prometheus is actively receiving metrics data from sglang