Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature Request] Support DLPack protocol in non-training builds #15963

Open
SimonRelu opened this issue May 16, 2023 · 1 comment · May be fixed by #23110
Open

[Feature Request] Support DLPack protocol in non-training builds #15963

SimonRelu opened this issue May 16, 2023 · 1 comment · May be fixed by #23110
Labels
feature request request for unsupported feature or enhancement

Comments

@SimonRelu
Copy link

SimonRelu commented May 16, 2023 •
edited
Loading

Describe the feature request

Currently, the DLPack protocol can only be used in non-training build. See:

I believe it would make sense to enable this in the main build and not only the training one. Many AI modules already support this:

Having DLPack support in onnxruntime allows us to have "zero" cost copys between these modules. This is not only interesting during training. Often, multiple models are used in which case the ouput of one model will be used as the input of the next one. When we want to do postprocessing/preprocessing on these models we currently can't do this without moving them to CPU using .numpy(). This comes with a significant performce cost.

Describe scenario use case

We want to use cupy for processing our model inbetween different inference runs. Cupy supports the DLPack protocol which would allow us to do so. One option would be to build with training support but this makes our package size quite a bit bigger which I'd like to avoid.
@SimonRelu SimonRelu added the feature request request for unsupported feature or enhancement label May 16, 2023
@SimonRelu
Copy link
Author

Update:

It is possible to use an OrtValue in cupy using the data pointer like this:

import cupy as cp
x = cp.random.rand(1, 2, 3)
mem = cp.cuda.UnownedMemory(x.data_ptr(), x.__sizeof__(), owner=x)
memptr = cp.cuda.MemoryPointer(mem, offset=0)
arr = cp.ndarray(x.shape(), dtype=cp.float16, memptr=memptr)

However, it is not possible as far as I know to go from a cupy ndarray to an ortvalue without doing a copy of the data

@decahedron1 decahedron1 mentioned this issue Jun 29, 2023
Closed
@monzelr monzelr mentioned this issue Feb 24, 2024
Open
@justinchuby justinchuby mentioned this issue Sep 11, 2024
Open
@take-cheeze take-cheeze linked a pull request Dec 15, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature request request for unsupported feature or enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Enable dlpack by default take-cheeze/onnxruntime
1 participant
@SimonRelu