the current text generation call will exceed the model's predefined maximum length (4096)

[waleyW]

Thanks for providing this project.
I am using interface.py to infer the model Llama-2-13b-chat-longlora-32k-sft. I input a text of 32,000 tokens and received the following warning:

This is a friendly reminder: the current text generation call will exceed the model's predefined maximum length (4096). Depending on the model, you may observe exceptions, performance degradation, or nothing at all.

here is my input:

python inference.py
--base_model "Llama-2-13b-chat-longlora-32k-sft"
--question "Please summarize the most important 30 question and answer pairs based on this article."
--context_size 32768
--max_gen_len 32768
--flash_attn True
--material "part_1.txt"

I don't know how to solve this warning. If I run LongAlpaca-7B, it has the same warning.

[cauchy221]

Same question here.
[Edit] Find an explanation to this question: #80

[waleyW]

Same question here. [Edit] Find an explanation to this question: #80

I reset the config, but it still had the warning"the current text generation call will exceed the model's predefined maximum length (4096)"

[cauchy221]

Same question here. [Edit] Find an explanation to this question: #80

I reset the config, but it still had the warning"the current text generation call will exceed the model's predefined maximum length (4096)"

I also did and it still shows this warning. But I think it's not an error, it's just a warning. Also from my experimental results, my model can correctly understand and generate output with 32k input even with the warning showing up.

[waleyW]

Same question here. [Edit] Find an explanation to this question: #80

I reset the config, but it still had the warning"the current text generation call will exceed the model's predefined maximum length (4096)"

I also did and it still shows this warning. But I think it's not an error, it's just a warning. Also from my experimental results, my model can correctly understand and generate output with 32k input even with the warning showing up.

I tested with a long text, and it indeed behaves as you described. Thank you for your response.