Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature] Support QLoRA weights #1826

Open
zzh-www opened this issue Oct 28, 2024 Discussed in #1777 · 2 comments
Open

[Feature] Support QLoRA weights #1826

zzh-www opened this issue Oct 28, 2024 Discussed in #1777 · 2 comments
Assignees
@Ying1123
Labels
enhancement New feature or request

Comments

@zzh-www
Copy link

zzh-www commented Oct 28, 2024

Does sgl support qlora? Could you provide some instructions on how to use it?
@zzh-www
Copy link
Author

zzh-www commented Oct 29, 2024

Hi! I run it by python -m sglang.bench_latency --model-path TheBloke/TinyLlama-1.1B-Chat-v0.3-GPTQ --batch 1 --inp ut-len 1 --output-len 512 --trust-remote-code --lora-paths jashing/tinyllama-colorist-lora.
It seems no problem, is there any way to help me verify that this is the correct thing to do?

@zzh-www
Copy link
Author

zzh-www commented Oct 30, 2024

I verified the results of QLoRA on my personal fine-tuning task (deployed as a service using the above command), and concluded that QLoRA did not actually take effect. I believe it's better to throw an error to alert users and avoid misunderstandings. @Ying1123

@merrymercy merrymercy changed the title sgl & qlora [Feature] Support QLoRA weights Nov 1, 2024
@merrymercy merrymercy added the enhancement New feature or request label Nov 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Ying1123 Ying1123

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Ying1123 @merrymercy @zzh-www