Skip to content

Commit

Permalink
docs: Update docstring for adapt_tokenizer
Browse files Browse the repository at this point in the history
  • Loading branch information
saattrupdan committed Mar 11, 2024
1 parent babf479 commit e92a6f4
Showing 1 changed file with 4 additions and 4 deletions.
8 changes: 4 additions & 4 deletions outlines/integrations/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,11 +33,11 @@


def adapt_tokenizer(tokenizer: PreTrainedTokenizerBase) -> PreTrainedTokenizerBase:
"""Adapt vLLM's tokenizer to use to compile the FSM.
"""Adapt a tokenizer to use to compile the FSM.
The API of Outlines tokenizers is slightly different to that of
`transformers`. In addition we need to handle the missing spaces to
Llama's tokenizer to be able to compile FSMs for this model.
The API of Outlines tokenizers is slightly different to that of `transformers`. In
addition we need to handle the missing spaces to Llama's tokenizer to be able to
compile FSMs for this model.
Parameters
----------
Expand Down

0 comments on commit e92a6f4

Please sign in to comment.