Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support for jinaai/jina-reranker-v2-base-multilingual model #2004

Open
2 of 4 tasks
bash99 opened this issue Aug 30, 2024 · 0 comments
Open
2 of 4 tasks

support for jinaai/jina-reranker-v2-base-multilingual model #2004

bash99 opened this issue Aug 30, 2024 · 0 comments
Labels
bug Something isn't working onnxruntime Related to ONNX Runtime

Comments

@bash99
Copy link

bash99 commented Aug 30, 2024

System Info

optimum                           1.21.4
Python 3.11.9
Ubuntu 22.04.4 LTS
NVIDIA-SMI 550.90.07              Driver Version: 550.90.07      CUDA Version: 12.4
GPU 2080 ti

Who can help?

@JingyaHuang @michaelbenayoun

Information

  • The official example scripts
  • My own modified scripts

Tasks

  • An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
  • My own task or dataset (give details below)

Reproduction (minimal, reproducible, runnable)

When I run jinaai/jina-reranker-v2-base-multilingual with infinity-emb server

pip install infinity-emb[all]
infinity_emb v2 --model-id jinaai/jina-reranker-v2-base-multilingual
 --engine optimum

the start-up logs

2024-08-30 18:21:25.407568117 [W:onnxruntime:, session_state.cc:1166 VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may or may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf.
2024-08-30 18:21:25.407597785 [W:onnxruntime:, session_state.cc:1168 VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignments.
INFO     2024-08-30 18:21:26,115 infinity_emb INFO: Optimizing model                              utils_optimum.py:139
WARNING  2024-08-30 18:21:26,118 infinity_emb WARNING: Optimization failed with Tried to use      utils_optimum.py:168
         ORTOptimizer for the model type , but it is not available yet. Please open an issue or submit a PR at https://github.com/huggingface/optimum.. Going to use the unoptimized         model.

It's even slow than use default torch engine of infinity-emb start as

infinity_emb v2 --model-id jinaai/jina-reranker-v2-base-multilingual

Expected behavior

optimum engine is faster than torch engine when model is support, for example "netease-youdao/Rerank"

@bash99 bash99 added the bug Something isn't working label Aug 30, 2024
@dacorvo dacorvo added the onnxruntime Related to ONNX Runtime label Oct 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working onnxruntime Related to ONNX Runtime
Projects
None yet
Development

No branches or pull requests

2 participants