-
Notifications
You must be signed in to change notification settings - Fork 931
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support for CUDA Streams #1751
Comments
Here's a discussion I've opened for it on cudarc: coreylowman/cudarc#209 |
I was also looking into this, looks like cudarc now supports create a device_with_stream have you tested this yet @michaeleisel ? |
I haven't, but it appears sufficient |
Indeed this seems to be sufficient as all |
I'm looking to leverage more of my GPU when running multiple models in parallel. It'd be great if candle had some sort of support for running multiple concurrent streams at once, whether through changing the stream used internally to CUDA's per-thread default stream, or allowing the user to run closures in different streams (
with_stream(|| { })
), or something else.The text was updated successfully, but these errors were encountered: