Using Outlines to Extract a Contiguous Text Span?

[JosephGatto]

Is it possible to restrict the output to be any contiguous span of text from the input?

In other words, I am wondering if Outlines can be used to do what would effectively be extractive inference?

[lapp0]

It currently isn't possible unless you pass an exceedingly complex regular expression. However while the regex for a given strings substring is of size O(N^2), a FSM for a substring is of linear size. In other words, this functionality could be integrated.

I'm also interested in extractive inference. Could you tell me more about what you're looking for, perhaps an example?

[alexmadey-oc]

Guidance implements this functionality in the substring method. It's extremely useful in cases where you want to prevent the model from hallucinating. For example, I may prompt the LLM with a product description from an ecommerce site and ask the model to generate a json with the product name, description, seller, etc. Forcing the model to generate a substring ensures that the model cannot hallucinate a completely nonexistent product name or description. I have noticed hallucinations can be especially common when, for example, the product name is a misspelling of a common word. The model will be naturally biased towards correct spelling, and will not faithfully generate the correct product name unless it is forced to only generate a substring of the prompt.

[rlouf]

Can't that be solved by an Enum?

[alexmadey-oc]

Isn't Enum just generating a selection from a list of options? Trying to specify a substring as an enum would be O(n^2), where n is the number of characters in the string. For long strings this would be a massive list.

Suppose I have a text input like this

Sea Stryker P6 Needlenose Plier, Multicolor

6.22
List Price: $8.99

When you’re looking to land your next big catch, the Sea Stryker will be by your side. Our saltwater lures include Got-Cha plugs, soft plastics, dusters and Buck tail teasers that come in bright colors with great action to grab a fish’s attention. Sea Striker ball bearing snap swivels are small but strong to handle almost any species and depth, while sinker slides are an inexpensive way to weigh down braided line without fraying it. Get rugged surf rods and standing roads for inshore coastal fishing along with rigging kits, beads and snelled hooks.

And I want to generate a JSON like

{
  "name": "P6 Needlenose Plier, Multicolor",
  "brand": "Sea Stryker",
  "price": "6.22",
  "description": "When you’re looking to land your next big catch, the Sea Stryker will be by your side. Our saltwater lures include Got-Cha plugs, soft plastics, dusters and Buck tail teasers that come in bright colors with great action to grab a fish’s attention. Sea Striker ball bearing snap swivels are small but strong to handle almost any species and depth, while sinker slides are an inexpensive way to weigh down braided line without fraying it. Get rugged surf rods and standing roads for inshore coastal fishing along with rigging kits, beads and snelled hooks."
}

I want all text in the JSON values to be from the input text. How would I implement this constraint using Enums?

[rlouf]

Ok I see what you're trying to do now. This should be doable with a suffix automaton built from the prompt, since for a given string such an automaton contains information about all its substrings. I'd be happy to have this implemented in Outlines.

[endolith]

Is this still a benefit of Guidance over Outlines?