Adding support for latest OLMo architectures #331
Conversation
| @@ -0,0 +1,115 @@ | |||
| ARG CUDA | |||
There was a problem hiding this comment.
do we want to add documentation about building with this image or nah? We had it before.
| model_name_or_path: ai2-adapt-dev/OLMo-medium-peteish7-anneal-from-928646-50B-nowup-dclm07-flan | ||
| model_revision: main | ||
| use_flash_attn: false | ||
| tokenizer_name: ai2-adapt-dev/OLMo-medium-peteish7-anneal-from-928646-50B-nowup-dclm07-flan |
There was a problem hiding this comment.
I would double check with luca that this is the right tokenizer. Sometimes conversions from the olmo repo and tokenizers get jumbled...
open_instruct/dpo_tune.py
Outdated
| @@ -672,7 +679,7 @@ def load_model(): | |||
| 0, | |||
| 1, | |||
| ], "LlamaTokenizer should only add one special token - the pad_token, or no tokens if pad token present." | |||
| elif isinstance(tokenizer, GPTNeoXTokenizerFast): | |||
| elif isinstance(tokenizer, GPTNeoXTokenizerFast) or isinstance(tokenizer, OLMoTokenizerFast): | |||
There was a problem hiding this comment.
wont this error if olmo is not imported? Maybe we should do:
elif isinstance(tokenizer, GPTNeoXTokenizerFast) or (check_hf_olmo_availability() and isinstance(tokenizer, OLMoTokenizerFast)):
open_instruct/finetune.py
Outdated
| @@ -643,7 +650,7 @@ def main(args: FlatArguments): | |||
| 0, | |||
| 1, | |||
| ], "LlamaTokenizer should only add one special token - the pad_token, or no tokens if pad token present." | |||
| elif isinstance(tokenizer, GPTNeoXTokenizerFast): | |||
| elif isinstance(tokenizer, GPTNeoXTokenizerFast) or isinstance(tokenizer, OLMoTokenizerFast): # noqa | |||
There was a problem hiding this comment.
same comment as above. elif isinstance(tokenizer, GPTNeoXTokenizerFast) or (check_hf_olmo_availability() and isinstance(tokenizer, OLMoTokenizerFast)):
| if package_exists: | ||
| try: | ||
| # Primary method to get the package version | ||
| package_version = importlib.metadata.version(pkg_name) |
There was a problem hiding this comment.
curious why doing this over like:
try
import hf_olmo
except ImportError
...is preferable. Don't mind it, just curious.
|
Yup @vwxyzjn have trained a couple models. Most recent one https://wandb.ai/ai2-llm/open_instruct_internal/runs/ny1m6zwx?nw=nwusernatolambert (plus another that already had eval scores, see slack) |
# Conflicts: # open_instruct/mix_data.py # requirements.txt
No description provided.