Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Automatic fallback to CPU in case of GPU loading failure #805

Closed
the-vindicar opened this issue Mar 11, 2024 · 1 comment
Closed
Labels
category: hardware management Related to hardware & compute P1: important Important feature / fix

Comments

@the-vindicar
Copy link

Problem
If GPU acceleration is enabled, Jan appears to follow "all or nothing" strategy, with model failing to activate completely if there is not enough vRAM, for example.

Success Criteria
A much better approach would be "graceful degradation", with model activating using CPU instead. Perhaps with a UI warning to notify user what has happened. That way at least the model would still respond, even if more slowly. Additionally, it'd allow accelerating small models and still working with larger ones.

An ideal approach would be to implement partial model offloading. That way it'd be possible to make a guess at how many layers can be safely offloaded into vRAM, so the model is accelerated as much as possible with the given hardware.

Additional context
I think LMStudio and GPT4All implement partial model offloading, so it's something that's possible to do. However, they just stick a slider into UI and leave it to user to find out how many layers can be loaded into vRAM.

@the-vindicar the-vindicar added the type: feature request A new feature label Mar 11, 2024
@Van-QA Van-QA transferred this issue from janhq/jan Jul 1, 2024
@imtuyethan imtuyethan removed the type: feature request A new feature label Aug 28, 2024
@imtuyethan imtuyethan removed the status in Menlo Aug 28, 2024
@imtuyethan imtuyethan moved this to Icebox in Menlo Sep 2, 2024
@freelerobot freelerobot added P1: important Important feature / fix category: hardware management Related to hardware & compute labels Sep 6, 2024
@dan-menlo
Copy link
Contributor

Closing this issue into #1165

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category: hardware management Related to hardware & compute P1: important Important feature / fix
Projects
Archived in project
Development

No branches or pull requests

6 participants