Skip to content

Commit

Permalink
update readme
Browse files Browse the repository at this point in the history
  • Loading branch information
AmineDiro committed Aug 5, 2023
1 parent 1c19db9 commit aabd88c
Showing 1 changed file with 4 additions and 2 deletions.
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,11 +146,13 @@ The only thing left to do is to change `Cargo.toml` file to
- [x] Implement streaming completions SSE
- [x] Cleanup cargo features with llm
- [x] Support MacOS Metal
- [ ] Merge completions / completion_streaming routes in same endpoint
- [x] Merge completions / completion_streaming routes in same endpoint
- [ ] Setup good tracing
- [ ] Implement `/embeddings` route
- [ ] Implement route `chat/completions`
- [ ] Implement route `/chat/completions`
- [ ] Implement streaming chat completions SSE
- [ ] GPU use should be optional ?
- [ ] Metrics ??
- [ ] Batching requests(ala iouring):
- For each response put an entry in a ringbuffer queue with : Entry(Flume mpsc (resp_rx,resp_tx))
- Spawn a model in separate task reading from ringbuffer, get entry and put each token in response
Expand Down

0 comments on commit aabd88c

Please sign in to comment.