Development Roadmap (2024 Q4)

[Ying1123]

Here is the development roadmap for 2024 Q4. Contributions and feedback are welcome (Join Bi-weekly Development Meeting). Previous 2024 Q3 roadmap can be found in #634.

Performance

• Hide CPU overhead with overlapped scheduler (Faster overlap mode scheduler #1738, Enable overlap by default #2067)
• Support speculative decoding
  • Eagle Speculative EAGLE2. New PR #2150
  • Reference-based. Reference speculative decoding #270
  • Medusa head [Feature] plan to support medusa? #859
  • Draft model based.
• Sparse Attention Support double sparsity #1459
• Faster grammar parsing library for constrained decoding [Performance] Support both xgrammar and outlines for constrained decoding #1752
• Multi-layer radix cache (GPU/CPU/Disk) @xiezhq-hermann
• Improve the performance of mixed chunked prefill. see a draft Rewrite mixed chunked prefill #1383
• Integrate CuDNN paged attention kernels

Parallelism

• Support sequence parallelism [Feature] Add initial support for sequence parallelism #1436. Related paper
• Support pipeline parallelism.
• Support expert parallelism + data parallelism for DeepSeek/MoE models. @ispobock
  • Data parallelism Support DP MLA #1970
  • Expert parallelism # [Feature] Expert parallelism support #1435
• Implement a better cache-aware load balancer for data parallelism. [router] cache-aware load-balancing router v1 #2114 [Feature] Cache-aware Data Parallel Router #1732 @ByronHsu @yichuan520030910320
• Overlap communication in tensor parallelsim. @ZhuohaoL
• Support disaggregated serving to separate prefill and decoding.

Hardware Coverage

• AMD optimizations. cc @HaiShaw
  • CK kernels
  • Setup CI (accuracy/performance) for AMD
• Intel XPU support.
  • [Feature, Hardware] Enable SGLang on XPU GPUs via PyTorch #1480
  • Add initial support for intel Gaudi accelerators #2121

Model Coverage

• Multi-modal models
  • Llama 3.2 Vision Llama3.2 vision model support #1551
  • QWen2-VL Support qwen2 vl model #1546
  • GLM 4V Add GLM-4v Multimodal Model support for SGLang #1641
  • VILA
  • Phi-vision
  • FishSpeech audio model support
  • Ultravox
• Language models
  • Mamba models
• Reward models
  • [Feature] Support reward model LxzGordon/URM-LLaMa-3.1-8B #1525
  • Gemma2 reward model support #1954

New features

• Integrate with LMCache https://github.com/LMCache/LMCache
• A padded batch mode to make results more deterministic

  sglang/docs/references/faq.md

  Line 3 in 8912b76

  ## The results are not deterministic, even with a temperature of 0

• Performance optimizations for multi-LoRA serving [LoRA, Performance] Add gemm expand triton kernel for multi-LoRA #1728

Quantization

@HaiShaw @zhyncs @ispobock

• Torchao integration Add llama implementation with no tensor parallel linears #1561
• Turbomind operators integration
• More CUTLASS mixed precision gemm integration
• KV cache quantization (more formats + scaling factor)

Server API

• Support directly taking embedding as inputs. [Feature] Generation Inputs: input_embeds #745
• Add APIs for using the inference engine in a single script without launching a separate server. See also examples.
  • Provide an offline engine API #1567
• Support endpoint other than OpenAI (Anthropic, Mistral) in the language frontend.
• Better APIs to support RL trainers, including https://github.com/huggingface/trl and https://github.com/OpenRLHF/OpenRLHF @zhaochenyang20
• Support generalized reward API (adding linear layers to any Causal LM to get the reward) https://github.com/OpenRLHF/OpenRLHF @zhaochenyang20

Observability

• Integrate Grafana / Prometheus
  • support prometheus metrics #1853 [WIP] Prometheus Metrics #1461

Others

• Notebook-style interactive tutorials. @zhaochenyang20
• Compiler mode optimizations for the language (e.g. support sending a full serialized SGL program to the server). @hnyls2002
• Memory pool refactor to better support mixing different attention layers (e.g., interleaved window attention). @Ying1123
• Make vLLM an optional dependency. @zhyncs @ByronHsu @yizhang2077 [Feature] Make vLLM optional in model code #1673

[fengyang95]

Are there any plans to optimize long context latency?

[lumiere-ml]

Hi，can I help for Multi-layer radix cache (GPU/CPU/Disk)？ Really insterested in that.

[tanzelin430]

Are there any plans to optimize long context latency?

I am interested in contributing to P-D split inference architechure and I have machines that support me to develop the architechure, if you guys got any related develop plans please let me know. Thank you @Ying1123 @zhyncs @fengyang95

[merrymercy]

@lumiere-ml @tanzelin430 Are you in the slack channel? We can follow up on that.

[zhyncs]

@lumiere-ml @tanzelin430 Are you in the slack channel? We can follow up on that.

@lumiere-ml @tanzelin430 Welcome to join our slack channel https://join.slack.com/t/sgl-fru7574/shared_invite/zt-2ngly9muu-t37XiH87qvD~6rVBTkTEHw

[tanzelin430]

@lumiere-ml @tanzelin430 Are you in the slack channel? We can follow up on that.

@lumiere-ml @tanzelin430 Welcome to join our slack channel https://join.slack.com/t/sgl-fru7574/shared_invite/zt-2ngly9muu-t37XiH87qvD~6rVBTkTEHw

thanks for invitation, I am in slack now. forward to collaberate with you

[lumiere-ml]

@lumiere-ml @tanzelin430 Are you in the slack channel? We can follow up on that.

@lumiere-ml @tanzelin430 Welcome to join our slack channel https://join.slack.com/t/sgl-fru7574/shared_invite/zt-2ngly9muu-t37XiH87qvD~6rVBTkTEHw

Thanks for your invitation！

[Edenzzzz]

@lumiere-ml @tanzelin430 Are you in the slack channel? We can follow up on that.

@lumiere-ml @tanzelin430 Welcome to join our slack channel https://join.slack.com/t/sgl-fru7574/shared_invite/zt-2ngly9muu-t37XiH87qvD~6rVBTkTEHw

Thanks for your invitation！

@lumiere-ml @zhyncs I'm also very interested, could you share which channel you're using to discuss?
Perhaps we can combine radix tree prefix matching with P-D disaggregation similar to Mooncake?

[mfdj2002]

If no one is actively working on supporting pipeline parallelism, I'm down to help