Response generated with Mistral 7B Instruct for NER task contains only one named entity at the most.

[siddharthtumre]

I have run spacy-llm for ner task to identify organisation and product entities with the Mistral 7B instruct model.

In the LLM response I can see that only a single named entity is being identified at most. I have checked this with multiple examples and the result is same.

Example:

LLM response for doc: Mike Iandoli is a self-confessed car sicko. He's executive director of the Larz Anderson Auto Museum in Brookline, and the owner of -- count 'em -- six cars: 1969 Jaguar Roadster, 1961 Jaguar Mark II Sedan, 1995 Land Rover Defender, 1950 Willys Jeepster, 2007 Audi S4, and a 2001 Ferrari Maranello.

1. Mike Iandoli | False | ==NONE== | It is a name, not

Also the explanation generated by the LLM is trimmed in every output I have seen.

Here is my config.cfg

[nlp]
lang = "en"
pipeline = ["llm"]

[components]

[components.llm]
factory = "llm"

[components.llm.task]
@llm_tasks = "spacy.NER.v3"
labels = ["ORGANIZATION", "PRODUCT"]
description = Entities are the names of organizations, companies, and their lines of product and service offerings. Adjectives, verbs, and adverbs are not entities. Pronouns are not entities.

[components.llm.task.label_definitions]
ORGANIZATION = "Known organizations, companies and industries, e.g. Apple Inc., Microsoft Corporation, Pfizer Inc."
PRODUCT = "Physical or digital items offered for sale or use, e.g. MacBook Pro, Skype, Celebrex."

[components.llm.task.examples]
@misc = "spacy.FewShotReader.v1"
path = "./examples.json"

[components.llm.model]
@llm_models = "spacy.Mistral.v1"
name = "Mistral-7B-Instruct-v0.1"

[rmitsch]

Hi @siddharthtumre, thanks for reporting this. I've converted this to a discussion as this is not a bug per se.

As you can see from the IO you included in your post, the underlying problem here is that Mistral just doesn't response with more than one entity (I could reproduce this). Now it's possible that a differently written prompt might yield better results with Mistral, but maintaining a different prompt version for each model isn't sustainable on our side.

Unfortunately the only thing I can recommend is to a) experiment with the prompt or b) use a different model. In the long term we'd like to be able to guarantee that all our tasks/prompts work properly with all models, but at the moment the differences in capabilities are too significant to do so.

[stevereiner]

Mistral works better with ner v2 than ner v3