BitNet_Llama_model_test_huggingface_GPU.ipynb

[DewEfresh]

I was testing in Colab and when I ran "model.model.layers[0].mlp.gate_proj.weight". I recieved very different results from yours. You got:
Parameter containing:
tensor([[ 0.0032, -0.0339, 0.0150, ..., 0.0041, -0.0048, 0.0061],
[-0.0105, -0.0049, -0.0586, ..., -0.0092, 0.0188, -0.0084],
[-0.0383, -0.0109, 0.0031, ..., -0.0410, 0.0211, 0.0223],
...,
[ 0.0131, -0.0259, 0.0034, ..., 0.0233, -0.0281, -0.0131],
[ 0.0062, 0.0198, 0.0085, ..., 0.0129, -0.0205, 0.0050],
[ 0.0292, 0.0152, -0.0175, ..., 0.0256, 0.0276, 0.0082]],
device='cuda:0', dtype=torch.bfloat16, requires_grad=True)

I got:
tensor([[ 0.0007, 0.0007, 0.0007, ..., 0.0007, -0.0007, 0.0007],
[ 0.0007, 0.0007, 0.0007, ..., 0.0007, 0.0007, -0.0007],
[ 0.0007, 0.0007, -0.0007, ..., -0.0007, -0.0007, 0.0007],
...,
[ 0.0007, -0.0007, 0.0007, ..., -0.0007, -0.0007, 0.0007],
[ 0.0007, 0.0007, 0.0007, ..., -0.0007, -0.0007, -0.0007],
[ 0.0007, -0.0007, 0.0007, ..., 0.0007, 0.0007, 0.0007]],
device='cuda:0', dtype=torch.bfloat16)

[Beomi]

Thanks for notice!

Could you provide the colab code you tried to run?
it would be helpful to check the issue😄

[DewEfresh]

https://colab.research.google.com/drive/1nvzhy_PCBZ_r6dlvQv3GfweJsGlZHrNJ?usp=sharing