Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Introduce automatic LLM routing #3454

Open
jacopo-chevallard opened this issue Nov 4, 2024 — with Linear · 1 comment
Open

Introduce automatic LLM routing #3454

jacopo-chevallard opened this issue Nov 4, 2024 — with Linear · 1 comment
Assignees
@jacopo-chevallard
Labels
area: backend Related to backend functionality or under the /backend directory enhancement New feature or request rag: retrieval

Comments

@jacopo-chevallard Linear
Copy link
Collaborator

jacopo-chevallard commented Nov 4, 2024 •
edited
Loading

Currently, we manually configure LLMs, and associate the same LLM to the different nodes of a workflow.

Implementing automatic routing, for instance using https://github.com/lm-sys/RouteLLM, will allow us to cut costs and latency, while maintaining performances largely unaltered.

See also:
@jacopo-chevallard jacopo-chevallard added area: backend Related to backend functionality or under the /backend directory rag: retrieval labels Nov 4, 2024 — with Linear
@linear Linear
Copy link

linear bot commented Nov 4, 2024

CORE-282 Introduce automatic LLM routing

@dosubot dosubot bot added the enhancement New feature or request label Nov 4, 2024
@jacopo-chevallard jacopo-chevallard self-assigned this Nov 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jacopo-chevallard jacopo-chevallard

Labels
area: backend Related to backend functionality or under the /backend directory enhancement New feature or request rag: retrieval
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jacopo-chevallard