"Hallucinating" after first successfull QA

[Stef1519]

Hi,

i 'm not very deep into finetuning LLMs (two Days, to be precise) and i have a rather "simple" Question.

After feeding a .csv with QA-Pairs about a certain programming language, using h2oai/h2o-danube3-500m-base (can´t use more because iḿ limited to two GPUS with 6 GB each), Metric BLEU, 512 Tokens, LORA, Token Averaged Cross Entropy, Batch Size 2, up to 30 Epochs, the Model (running in "chat") initially gives a correct answer after the first question, but wrong answers for the following questions and mixes up the first correct answer with the new answer, means its hallucinating. I can only prevent this by closing the chat entirely and open a new one.

Can anybody explain this behavior and a possible cause (wrong Training parameters, "works as intended")?

Also, could it be possible with LLM Studio to train - not - on GPU, but on CPU (two Xeons, 128 GB RAM) or automatically swap the load off to the CPU(s)? Performing slower wouldn´t be the issue for me, but it would open the possibility to use larger (better) models for finetuning.

Best regards & thank you !

[pascal-pfeiffer]

Thank you for the detailed question. From how you describe the issue, I can see at least the following possible reason:

As you are only training on pairs, meaning not a multi-turn conversation but only single question + single answer, the inference your are performing in the chat tab with multi-turn conversations is completely out of distribution.

You might want to chain multiple questions and answers together if you want the model to also work well with this setup.

Your dataset would look like this:

id	parent_id	question	answer
0	None	what is XY	XY is ZX
1	0	What is YZ	YZ is 123
2	None	What is AB	AB is IU
3	2	What is CC	CC is great

of course can chain more Q/A pairs together or mix up the number of the chains a bit to add diversity to your train dataset.

Regarding your second question:
Training very small models might be technically possible on CPU with quite a few limitations such as missing support for quantization, but H2O LLM Studio officially only supports training on GPU as CPU training is prohibitively slow.

[Stef1519]

Thank you for your explanation. Multi-turn isn´t precisely the issue and the dataset looks good after uploading the file and inspecting it.

The problem is, again: If i ask a question -exactly as it is provided in the original dataset (csv)- the first initial Chat-Answer to my question is correct. If i ask another question, again -exactly as it is provided in the dataset-, the LLM starts to hallucinate and mixes up the answer with the first initial answer - more than that, the next answer has nothing to do with my question . If i close the chat and reopen it, it will correctly answer my second question, but for the next question the same weird behaviour starts again.

So i cannot check that the model works as intended or if it is a training error - which i doubt, because if it would be the case, i wouldn´t get any correct answer. This is what makes me wonder why this happens and if this is a fault from my side or a fault of the chat engine.

[psinger]

There is a setting num_history that is by default 4, meaning a certain number of previous questions/answers will be put into context in the chat tab. You should be able to change this to one if you do not want to have any history there. This is also exposed in the UI.