-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cannot convert neox model to HF #1231
Labels
bug
Something isn't working
Comments
If you look at |
This error occurs when your config doesn't include |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
I get an error while converting neox model to HF
the error that i get
Traceback (most recent call last): File "/media/sid/WDInternal/stability_ai/gpt-neox/tools/convert_sequential_to_hf.py", line 318, in <module> hf_model = convert(args.input_dir, loaded_config, args.output_dir) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/media/sid/WDInternal/stability_ai/gpt-neox/tools/convert_sequential_to_hf.py", line 156, in convert hf_config = create_config(loaded_config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/media/sid/WDInternal/stability_ai/gpt-neox/tools/convert_sequential_to_hf.py", line 108, in create_config tokenizer = build_tokenizer(args) ^^^^^^^^^^^^^^^^^^^^^ File "/media/sid/WDInternal/stability_ai/gpt-neox/megatron/tokenizer/tokenizer.py", line 37, in build_tokenizer if args.tokenizer_type.lower() == "GPT2BPETokenizer".lower(): ^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'lower'
To Reproduce
python tools/ckpts/convert_neox_to_hf.py --input_dir checkpoints/global_step5000/ --config_fil checkpoints/global_step5000/configs/125M.yml --output_dir hf/
Expected behavior
The model should be converted to HF
Environment (please complete the following information):
The text was updated successfully, but these errors were encountered: