Context length management

[noelo]

Hi,
I've just being trying out the stack and it looks great. I have an 3.1-8b parameter mode running on a remote VLLM instance.

I'm setting the vllm max_token using " --env VLLM_MAX_TOKENS=6048" but when I run any of the llama-stack-apps I keep getting the following error "openai.BadRequestError: Error code: 400 - {'object': 'error', 'message': "This model's maximum context length is 6048 tokens. However, you requested 6616 tokens (568 in the messages, 6048 in the completion). Please reduce the length of the messages or completion.", 'type': 'BadRequestError', 'param': None, 'code': 400}

I've tried to find if there's some parameter to change to avoid overfilling the models context but I've not been able to solve it. Any ideas to help with this?

[terrytangyuan]

You can change this when starting your vLLM server via --max_model_len=6048

[noelo]

thanks for replying, I appreciate the help. Yea I've done that hence the "This model's maximum context length is 6048 tokens" output but I think the llama-stack is not respecting that but I can't find where that it. Now it may be on the llama-stack-app demo that I'm running and I've tried to set the context length there but it's still ignored

[noelo]

I managed to sort this out at the client level. The AgentConfig class has a max_tokens field in the sampling_params config.

    agent_config = AgentConfig(
        model=selected_model,
        sampling_params={"max_tokens":4096,
            "strategy": {"type": "top_p", "temperature": 1.0, "top_p": 0.9},
        },

I wonder if this should be something passed back by the framework, similar to the model name.