Converting exported models to GGUF

[cemremengu]

I am trying to convert my models/experiments to GGUF format. I first push checkpoints to hugging face then use instructions from llama.cpp to convert the pushed model to GGUF. However I always get some errors like could not parse ModelProto from tokenizer.model.

Am I doing something wrong or this is not supported?

[psinger]

Is this happening for all models or only specific ones?

It seems this is a known issue for llama3:
ggerganov/llama.cpp#6747 (comment)

Best to research directly in llama.cpp as it does not seem related to LLM Studio.

But seems like just adding --vocab-type bpe to the convert script might solve it.

[cemremengu]

It turned out to be a PBCAK. I forgot to enable git LFS so was not cloning the whole repo. Sorry for wasting your time.

For future comers:

git clone https://github.com/ggerganov/llama.cpp.git

pip install -r llama.cpp/requirements.txt

python llama.cpp/convert.py your-model --outfile your-model.gguf --outtype f16 

[sunilswain]

Not working for me
I have fine tuned my model using llmstudio cli and I have my model checkpoint file(.pth), And using the above code gives me this

error
Traceback (most recent call last):
File "/content/llama.cpp/convert.py", line 1714, in
main()
File "/content/llama.cpp/convert.py", line 1630, in main
model_plus = load_some_model(args.model)
File "/content/llama.cpp/convert.py", line 1474, in load_some_model
models_plus.append(lazy_load_file(path))
File "/content/llama.cpp/convert.py", line 1020, in lazy_load_file
return lazy_load_torch_file(fp, path)
File "/content/llama.cpp/convert.py", line 976, in lazy_load_torch_file
model = unpickler.load()
File "/content/llama.cpp/convert.py", line 965, in find_class
return self.CLASSES[(module, name)]
KeyError: ('torch', 'CharStorage')

here's my config file for finetuning

architecture:
    backbone_dtype: int8
    force_embedding_gradients: false
    gradient_checkpointing: true
    intermediate_dropout: 0.0
    pretrained: true
    pretrained_weights: ''
augmentation:
    neftune_noise_alpha: 0.0
    random_parent_probability: 0.0
    skip_parent_probability: 0.0
    token_mask_probability: 0.0
dataset:
    add_eos_token_to_answer: true
    add_eos_token_to_prompt: true
    add_eos_token_to_system: true
    answer_column: output_prompt
    chatbot_author: Esspl
    chatbot_name: EssplGPT
    data_sample: 1.0
    data_sample_choice:
    - Train
    - Validation
    limit_chained_samples: true
    mask_prompt_labels: true
    parent_id_column: parent_id
    personalize: false
    prompt_column:
    - input_prompt
    system_column: None
    text_answer_separator: <|answer|>
    text_prompt_start: <|prompt|>
    text_system_start: <|system|>
    train_dataframe: data/EssplTravelPolicyQA/qa_train.csv
    validation_dataframe: None
    validation_size: 0.01
    validation_strategy: automatic
environment:
    compile_model: false
    deepspeed_allgather_bucket_size: 1000000
    deepspeed_method: ZeRO2
    deepspeed_reduce_bucket_size: 1000000
    deepspeed_stage3_param_persistence_threshold: 1000000
    deepspeed_stage3_prefetch_bucket_size: 1000000
    find_unused_parameters: false
    gpus:
    - '0'
    - '1'
    huggingface_branch: main
    mixed_precision: false
    number_of_workers: 1
    seed: -1
    trust_remote_code: false
    use_deepspeed: false
experiment_name: EssplTravelPolicyFT
llm_backbone: h2oai/h2ogpt-4096-llama2-7b
logging:
    logger: None
    neptune_project: Zoo/h2o-llm
output_directory: EssplTravelPolicyFTExperiment
prediction:
    batch_size_inference: 0
    do_sample: false
    max_length_inference: 256
    max_time: 120.0
    metric: BLEU
    metric_gpt_model: gpt-3.5-turbo-0301
    metric_gpt_template: general
    min_length_inference: 2
    num_beams: 1
    num_history: 4
    repetition_penalty: 1.1
    stop_tokens: ''
    temperature: 0.0
    top_k: 0
    top_p: 1.0
problem_type: text_causal_language_modeling
tokenizer:
    add_prompt_answer_tokens: false
    max_length: 512
    max_length_answer: 256
    max_length_prompt: 256
    padding_quantile: 1.0
    use_fast: true
training:
    batch_size: 2
    differential_learning_rate: 1.0e-05
    differential_learning_rate_layers: []
    drop_last_batch: true
    epochs: 1
    evaluate_before_training: false
    evaluation_epochs: 1.0
    grad_accumulation: 1
    gradient_clip: 0.0
    learning_rate: 0.0001
    lora: true
    lora_alpha: 16
    lora_dropout: 0.05
    lora_r: 4
    lora_target_modules: ''
    loss_function: TokenAveragedCrossEntropy
    optimizer: AdamW
    save_best_checkpoint: false
    schedule: Cosine
    train_validation_data: false
    use_flash_attention_2: false
    warmup_epochs: 0.0
    weight_decay: 0.0

[cemremengu]

Not sure if this script works on checkpoints. I first exported my model to huggingface before doing the conversion (disabling safetensors format). After you need to clone that repo (make sure git lfs is enabled) and run the script on that directory.