Problem in loading models

[myrainbowandsky]

model=TheBloke_Llama-2-13B-GPTQ/model.safetensors, I also tried: Wizard-Vicuna-7B-Uncensored-GPTQ-4bit-128g.no-act-order.safetensors, same problem

Loading model ...
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[36], line 5
      1 # MODEL_PATH = '/home/quang/working/LLMs/oobabooga_linux/text-generation-webui/models/TheBloke_Wizard-Vicuna-7B-Uncensored-GPTQ'
      2 # CHECKPOINT_PATH = '/home/quang/working/LLMs/oobabooga_linux/text-generation-webui/models/TheBloke_Wizard-Vicuna-7B-Uncensored-GPTQ/Wizard-Vicuna-7B-Uncensored-GPTQ-4bit-128g.no-act-order.safetensors'
      4 DEVICE = torch.device('cuda:0')
----> 5 model, tokenizer = load_model_main(MODEL_PATH, CHECKPOINT_PATH, DEVICE)

File /mnt/data/generativeAgent_LLM/server/model.py:56, in load_model_main(model_para, checkpoint_para, device)
     55 def load_model_main(model_para, checkpoint_para, device):
---> 56     model = load_quant(model_para, checkpoint_para, 4, 128)
     57     model.to(device)
     58     tokenizer = AutoTokenizer.from_pretrained(model_para)

File /mnt/data/generativeAgent_LLM/server/model.py:47, in load_quant(model, checkpoint, wbits, groupsize)
     45 if checkpoint.endswith('.safetensors'):
     46     from safetensors.torch import load_file as safe_load
---> 47     model.load_state_dict(safe_load(checkpoint))
     48 else:
     49     model.load_state_dict(torch.load(checkpoint))

File ~/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py:1671, in Module.load_state_dict(self, state_dict, strict)
   1666         error_msgs.insert(
   1667             0, 'Missing key(s) in state_dict: {}. '.format(
   1668                 ', '.join('"{}"'.format(k) for k in missing_keys)))
   1670 if len(error_msgs) > 0:
-> 1671     raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
   1672                        self.__class__.__name__, "\n\t".join(error_msgs)))
   1673 return _IncompatibleKeys(missing_keys, unexpected_keys)

RuntimeError: Error(s) in loading state_dict for LlamaForCausalLM:
	Unexpected key(s) in state_dict: "model.layers.0.self_attn.rotary_emb.inv_freq", "model.layers.0.self_attn.k_proj.g_idx", "model.layers.0.self_attn.o_proj.g_idx", "model.layers.0.self_attn.q_proj.g_idx", "model.layers.0.self_attn.v_proj.g_idx", "model.layers.0.mlp.down_proj.g_idx", "model.layers.0.mlp.gate_proj.g_idx", "model.layers.0.mlp.up_proj.g_idx", "model.layers.1.self_attn.rotary_emb.inv_freq", "model.layers.1.self_attn.k_proj.g_idx", "model.layers.1.self_attn.o_proj.g_idx", "model.layers.1.self_attn.q_proj.g_idx", "model.layers.1.self_attn.v_proj.g_idx", "model.layers.1.mlp.down_proj.g_idx", "model.layers.1.mlp.gate_proj.g_idx", "model.layers.1.mlp.up_proj.g_idx", "model.layers.2.self_attn.rotary_emb.inv_freq", "model.layers.2.self_attn.k_proj.g_idx", "model.layers.2.self_attn.o_proj.g_idx", "model.layers.2.self_attn.q_proj.g_idx", "model.layers.2.self_attn.v_proj.g_idx", "model.layers.2.mlp.down_proj.g_idx", "model.layers.2.mlp.gate_proj.g_idx",

[en-ashay]

model=TheBloke_Llama-2-13B-GPTQ/model.safetensors, I also tried: Wizard-Vicuna-7B-Uncensored-GPTQ-4bit-128g.no-act-order.safetensors, same problem

Loading model ...
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[36], line 5
      1 # MODEL_PATH = '/home/quang/working/LLMs/oobabooga_linux/text-generation-webui/models/TheBloke_Wizard-Vicuna-7B-Uncensored-GPTQ'
      2 # CHECKPOINT_PATH = '/home/quang/working/LLMs/oobabooga_linux/text-generation-webui/models/TheBloke_Wizard-Vicuna-7B-Uncensored-GPTQ/Wizard-Vicuna-7B-Uncensored-GPTQ-4bit-128g.no-act-order.safetensors'
      4 DEVICE = torch.device('cuda:0')
----> 5 model, tokenizer = load_model_main(MODEL_PATH, CHECKPOINT_PATH, DEVICE)

File /mnt/data/generativeAgent_LLM/server/model.py:56, in load_model_main(model_para, checkpoint_para, device)
     55 def load_model_main(model_para, checkpoint_para, device):
---> 56     model = load_quant(model_para, checkpoint_para, 4, 128)
     57     model.to(device)
     58     tokenizer = AutoTokenizer.from_pretrained(model_para)

File /mnt/data/generativeAgent_LLM/server/model.py:47, in load_quant(model, checkpoint, wbits, groupsize)
     45 if checkpoint.endswith('.safetensors'):
     46     from safetensors.torch import load_file as safe_load
---> 47     model.load_state_dict(safe_load(checkpoint))
     48 else:
     49     model.load_state_dict(torch.load(checkpoint))

File ~/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py:1671, in Module.load_state_dict(self, state_dict, strict)
   1666         error_msgs.insert(
   1667             0, 'Missing key(s) in state_dict: {}. '.format(
   1668                 ', '.join('"{}"'.format(k) for k in missing_keys)))
   1670 if len(error_msgs) > 0:
-> 1671     raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
   1672                        self.__class__.__name__, "\n\t".join(error_msgs)))
   1673 return _IncompatibleKeys(missing_keys, unexpected_keys)

RuntimeError: Error(s) in loading state_dict for LlamaForCausalLM:
	Unexpected key(s) in state_dict: "model.layers.0.self_attn.rotary_emb.inv_freq", "model.layers.0.self_attn.k_proj.g_idx", "model.layers.0.self_attn.o_proj.g_idx", "model.layers.0.self_attn.q_proj.g_idx", "model.layers.0.self_attn.v_proj.g_idx", "model.layers.0.mlp.down_proj.g_idx", "model.layers.0.mlp.gate_proj.g_idx", "model.layers.0.mlp.up_proj.g_idx", "model.layers.1.self_attn.rotary_emb.inv_freq", "model.layers.1.self_attn.k_proj.g_idx", "model.layers.1.self_attn.o_proj.g_idx", "model.layers.1.self_attn.q_proj.g_idx", "model.layers.1.self_attn.v_proj.g_idx", "model.layers.1.mlp.down_proj.g_idx", "model.layers.1.mlp.gate_proj.g_idx", "model.layers.1.mlp.up_proj.g_idx", "model.layers.2.self_attn.rotary_emb.inv_freq", "model.layers.2.self_attn.k_proj.g_idx", "model.layers.2.self_attn.o_proj.g_idx", "model.layers.2.self_attn.q_proj.g_idx", "model.layers.2.self_attn.v_proj.g_idx", "model.layers.2.mlp.down_proj.g_idx", "model.layers.2.mlp.gate_proj.g_idx",

@myrainbowandsky @QuangBK I was also getting the same error but it was specific to this model, I later tried with microsoft phi 2 and mixtral and those two worked fine.