You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
and model.resize_token_embeddings(32001)
Are they needed for the fine-tuned model with extended context length to work properly? For example, I finetuned the orignal Llama2 model to get a new context length of 16k, do I still need the settings for the model during inference? This is important since it will save us the hassle of writing custom inference code when using certain model-serving frameworks. We just tell the framework the model's save location.
The text was updated successfully, but these errors were encountered:
spring1915
changed the title
Configs in inference.py necessary for context length expansion?
Configs in inference.py necessary for context length expansion in model serving?
Dec 13, 2023
In inference.py, there're two settings
and
model.resize_token_embeddings(32001)
Are they needed for the fine-tuned model with extended context length to work properly? For example, I finetuned the orignal Llama2 model to get a new context length of 16k, do I still need the settings for the model during inference? This is important since it will save us the hassle of writing custom inference code when using certain model-serving frameworks. We just tell the framework the model's save location.
The text was updated successfully, but these errors were encountered: