-
Notifications
You must be signed in to change notification settings - Fork 445
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Encountering RuntimeError: Cannot convert token
� (29333) to bytes: �
for some model vocabularies when using llama.cpp
#820
Comments
This is also true of gemma models (eg. gemma-2b-it) |
I'm seeing this with llama3 instruct as well. |
Temp workaround: install |
Experiencing this as well on Llama 3 8b instruct, as well as various other models. |
Can someone confirm that the problem was introduced in |
@rlouf can confirm. When testing the following script: from outlines import models, generate
model = models.llamacpp("./Meta-Llama-3-8B-Instruct.Q8_0.gguf")
generator = generate.choice(model, ["skirt", "dress", "pen", "jacket"])
answer = generator("Pick the odd word out: skirt, dress, pen, jacket")
print(answer) Running fine under |
Thank you! The bug was most likely introduced by #738. I'll try to understand why. |
Hi @rlouf, is there any update on this? I'm using Outlines version 0.0.41 with Llama-3-8B and it keeps failing. When using version 0.0.36 the inference takes too long with any Llama-3 model compared to other models such as mixtral-7x[8|22]B. |
This issue appears to be a result of You can resolve this issue with the following steps:
|
I'm still getting this issue with |
@pmbaumgartner could you please ensure you're loading via
|
Describe the issue as clearly as possible:
Ran into this tokenizer issue in
regex.py
with some models (Qwen1.5
,Phi-2
) but not others (OpenHermes-2.5-Mistral-7B
) when using llama.cpp.This honestly might be my fault with something I'm doing in the
llama-cpp-python
but I'm not familiar enough with the outlines codebase to tell.Steps/code to reproduce the bug:
Expected result:
`skirt`
Error message:
Outlines/Python version information:
Version information
Context for the issue:
No response
The text was updated successfully, but these errors were encountered: