GPT4ALL basically all the small models reply mess up the parser

[AvengerMoJo]

[nlp]
lang = "en"
pipeline = ["llm"]
batch_size = 8192

[components.llm]
factory = "llm"

[components.llm.task]
@llm_tasks = "spacy.TextCat.v3"
labels = ["COMPLIMENT", "INSULT"]

[components.llm.task.label_definitions]
"COMPLIMENT" = "a polite expression of praise or admiration.",
"INSULT" = "a disrespectful or scornfully abusive remark or act."

[components.llm.model]
@llm_models = "langchain.GPT4All.v1"
name = "gemma-2b-it-q4_k_m.gguf"
context_length = 8192
config = { "temp": 0.0 }

doc = nlp("You look gorgeous!")
print(doc.cats)

doc = nlp("This is non-sense")
print(doc.cats)

doc = nlp("You are horrible stupid")
print(doc.cats)

['You are an expert Text Classification system. Your task is to accept Text as input\nand provide a category for the text based on the predefined labels.\n\nClassify the text below to any of the following labels: COMPLIMENT, INSULT\n\nOnly reply with the labels nothing else.\nThe task is non-exclusive, so you can provide more than one label as long as\nthey're comma-delimited. For example: Label1, Label2, Label3.\nDo not put any other text in your answer, only one or more of the provided labels with nothing before or after.\nIf the text cannot be classified into any of the provided labels, answer ==NONE==.\n\nBelow are definitions of each label to help aid you in correctly classifying the text.\nAssume these definitions are written by an expert and follow them closely.\n\n"COMPLIMENT": "a polite expression of praise or admiration.",\n"INSULT": a disrespectful or scornfully abusive remark or act.\n\n\nHere is the text that needs classification\n\n\nText:\n'''\nYou look gorgeous!\n'''']
LLM response for doc: You look gorgeous!
['\n\nThe category for this text would be INSULT, as it is a disrespectful and scornfully abusive remark or act.']
defaultdict(<function reduce_shards_to_doc.. at 0x130882e80>, {'COMPLIMENT': 0.0, 'INSULT': 0.0})
Generated prompt for doc: This is non-sense
['You are an expert Text Classification system. Your task is to accept Text as input\nand provide a category for the text based on the predefined labels.\n\nClassify the text below to any of the following labels: COMPLIMENT, INSULT\n\nOnly reply with the labels nothing else.\nThe task is non-exclusive, so you can provide more than one label as long as\nthey're comma-delimited. For example: Label1, Label2, Label3.\nDo not put any other text in your answer, only one or more of the provided labels with nothing before or after.\nIf the text cannot be classified into any of the provided labels, answer ==NONE==.\n\nBelow are definitions of each label to help aid you in correctly classifying the text.\nAssume these definitions are written by an expert and follow them closely.\n\n"COMPLIMENT": "a polite expression of praise or admiration.",\n"INSULT": a disrespectful or scornfully abusive remark or act.\n\n\nHere is the text that needs classification\n\n\nText:\n'''\nThis is non-sense\n'''']
LLM response for doc: This is non-sense
['\n\nAnswer: ==NONE==']
defaultdict(<function reduce_shards_to_doc.. at 0x130883380>, {'COMPLIMENT': 0.0, 'INSULT': 0.0})
Generated prompt for doc: You are horrible stupid
['You are an expert Text Classification system. Your task is to accept Text as input\nand provide a category for the text based on the predefined labels.\n\nClassify the text below to any of the following labels: COMPLIMENT, INSULT\n\nOnly reply with the labels nothing else.\nThe task is non-exclusive, so you can provide more than one label as long as\nthey're comma-delimited. For example: Label1, Label2, Label3.\nDo not put any other text in your answer, only one or more of the provided labels with nothing before or after.\nIf the text cannot be classified into any of the provided labels, answer ==NONE==.\n\nBelow are definitions of each label to help aid you in correctly classifying the text.\nAssume these definitions are written by an expert and follow them closely.\n\n"COMPLIMENT": "a polite expression of praise or admiration.",\n"INSULT": a disrespectful or scornfully abusive remark or act.\n\n\nHere is the text that needs classification\n\n\nText:\n'''\nYou are horrible stupid\n'''']
LLM response for doc: You are horrible stupid
['\n\nAnswer: INSULT']
defaultdict(<function reduce_shards_to_doc.. at 0x1308832e0>, {'COMPLIMENT': 0.0, 'INSULT': 0.0})

---

You can see the reply from the LLM is correct but the format is not good.

[rmitsch]

Hi @AvengerMoJo, thanks for reporting this. Smaller and older models are prone to not always picking up the expected response format correctly. We are currently looking into ways to improve this.

[AvengerMoJo]

I understand that. However my issue is focus more about the parser return part. I try to change the format to the correct reply with nothing more and nothing less. The example print doc.cats still print like defaultdict(<function reduce_shards_to_doc.. at 0x1308832e0>, {'COMPLIMENT': 1.0, 'INSULT': 0.0})
but print doc.cats['COMPLIMENT'] does work
Meaning if I force the LLM to replay "INSULT" or "COMPLIMENT", it will get the correct result but still print doc.cats still look wrong.