New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

inference OOM #154

Open

PharMolix opened this issue Dec 9, 2023 · 0 comments

PharMolix commented Dec 9, 2023

你好！
我使用repo提供的环境和inference.py代码，在4*80G A100上进行inference，结果遇到了OOM的问题

python inference.py  \
        --base_model merge_model/7b \
        --question "介绍一下这本书的内容" \
        --context_size 32768 \
        --max_gen_len 512 \
        --flash_attn True \
        --material "materials/chinese_medical_long.txt"

请问可能是什么原因呢？

PharMolix closed this as not planned

PharMolix reopened this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment