Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make Microsoft.ML.OnnxRuntimeGenAI.Tokenizer a Microsoft.ML.Tokenizers.Tokenizer #970

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Make Microsoft.ML.OnnxRuntimeGenAI.Tokenizer a Microsoft.ML.Tokenizers.Tokenizer #970

wants to merge 1 commit into from

Commits on Oct 16, 2024

  1. Make Microsoft.ML.OnnxRuntimeGenAI.Tokenizer a Microsoft.ML.Tokenizer… 

    …s.Tokenizer

This enables an ONNX Runtime GenAI tokenizer instance to be used anywhere a Microsoft.ML.Tokenizers tokenizer is accepted. If we'd prefer, rather than having Tokenizer be a base class for the ONNX Runtime one, we could instead expose some sort of `public Microsoft.ML.Tokenizer.Tokenizer AsTokenizer()` conversion method that returns a wrapper object (though that's a bit confusing given the names of the type are the same, just different namespaces).
    @stephentoub
    stephentoub committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    0b759a7 View commit details
    Browse the repository at this point in the history