How to fix the token length of the model input?

[lonleyodd]

When testing the dataset like MMLU , the input token length varies for each inference. How can I fix the token length of the model input?

[baberabb]

You can pass in max_length to constrain max sequence length, and it will left truncate any inputs greater than that

[sorobedio]

i have the same problem
CUDA_VISIBLE_DEVICES=0 lm_eval --model hf \ --model_args pretrained=openai-community/gpt2,max_length=1024,dtype="bfloat16"\ --tasks leaderboard_musr\ --device cuda:0 \ --num_fewshot 0\ --batch_size 8

and here is the error id does not disapeared. i am using v0.4.3 same forked version as lm-leaderboard

[evaluator.py:433] Running loglikelihood requests
Token indices sequence length is longer than the specified maximum sequence length for this model (1139 > 1024). Running this sequence through the model will result in indexing errors

[lonleyodd]

You can pass in max_length to constrain max sequence length, and it will left truncate any inputs greater than that

@baberabb well I mean how to have a fixed input length for each inference, taking 2048 as an example, where the input length is always 2048. When the actual input content is less than 2048, padding is added to reach this length, rather than simply limiting the maximum input length to no more than 2048.

[lonleyodd]

i have the same problem CUDA_VISIBLE_DEVICES=0 lm_eval --model hf \ --model_args pretrained=openai-community/gpt2,max_length=1024,dtype="bfloat16"\ --tasks leaderboard_musr\ --device cuda:0 \ --num_fewshot 0\ --batch_size 8

and here is the error id does not disapeared. i am using v0.4.3 same forked version as lm-leaderboard

[evaluator.py:433] Running loglikelihood requests Token indices sequence length is longer than the specified maximum sequence length for this model (1139 > 1024). Running this sequence through the model will result in indexing errors

have you tried set max_length=2048?

[baberabb]

@baberabb well I mean how to have a fixed input length for each inference, taking 2048 as an example, where the input length is always 2048. When the actual input content is less than 2048, padding is added to reach this length, rather than simply limiting the maximum input length to no more than 2048.

yeah, the inputs are padded to the largest sequence length in a batch (logliklehood , generations), so not fixed. I think the simplest way to handle this would be to modify the _model_call (add right padding, and trim it from the outputs) and model_generate (add left padding) methods.

[CPA872]

@baberabb Jumping in for a side question. I notice that the seqlen in the model decreases as the evaluation goes along. For example, with batch 64, the first iteration input is (64, 720), second iteration (64, 718) , until the very end it becomes (64, 30). Something like this. Is there any query sorting mechanism (based on seqlen) implemented in lm-eval or huggingface dataset that is used? If you happen to know about this, could you provide a pointer to where this sorting is implemented in the codebase?

Thanks in advance.

[baberabb]

@baberabb Jumping in for a side question. I notice that the seqlen in the model decreases as the evaluation goes along. For example, with batch 64, the first iteration input is (64, 720), second iteration (64, 718) , until the very end it becomes (64, 30). Something like this. Is there any query sorting mechanism (based on seqlen) implemented in lm-eval or huggingface dataset that is used? If you happen to know about this, could you provide a pointer to where this sorting is implemented in the codebase?

Thanks in advance.

Hi! yeah, the sequences are sorted by token length (using this utility function).