[Feature] Throughput-aware speculative decoding

[vkc1vk]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

Since speculative decoding techniques such as Eagle are mostly effective in low-throughput scenarios, I was wondering if it makes sense to tone down the size of draft trees or to shut off speculative decoding completely in a throughput-aware manner.

What do you think?

Related resources

No response

[zhyncs]

You're talking about Automatic Speculative Decoding, which is feasible.