-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
GPU Not Utilized When Using llm-rs with CUDA Version #27
Comments
Currently only I will probably create some sort of table in the We are planning to bring cuda acceleration too |
I appreciate the plan to create a table in the rustformers/llm repository, showing which architectures are supported with acceleration on specific platforms. That will definitely help avoid confusion in the future. Thanks again for the explanation. |
I have installed the llm-rs library with the CUDA version, However, even though I have set
use_gpu=True
in theSessionConfig
, the GPU is not utilized when running the code. Instead, the CPU usage remains at 100% during execution.Additional Information:
I am using the "RedPajama Chat 3B" model from Rustformers. The model can be found at the following link: RedPajama Chat 3B Model.
Terminal output:
Code:
The text was updated successfully, but these errors were encountered: