Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Document DataFusion Threading / tokio runtimes (how to separate IO and CPU bound work) #12393

Open
Tracked by #13456
tustvold opened this issue Sep 9, 2024 · 4 comments · May be fixed by #13424
Open
Tracked by #13456

Document DataFusion Threading / tokio runtimes (how to separate IO and CPU bound work) #12393

tustvold opened this issue Sep 9, 2024 · 4 comments · May be fixed by #13424
Assignees
@alamb
Labels
enhancement New feature or request

Comments

@tustvold
Copy link
Contributor

tustvold commented Sep 9, 2024 •
edited
Loading

Is your feature request related to a problem or challenge?

DataFusion performs CPU bound work within async closures. This causes issues if running IO on the same async runtime, as the cooperative nature of such schedulers allows the CPU bound work to starve servicing of IO. This leads to errors such as apache/arrow-rs#5882.

Describe the solution you'd like

I think at the very least this needs to be better documented, I couldn't find any mention of this in the DataFusion documentation following a cursory search.

I also think more holistic approach would be valuable to this, as it stands the use of async within DataFusion acts as a massive footgun that encourages users to intermix IO and CPU work in a way that is at best inefficient, but this can be tracked as a separate follow on task.

Describe alternatives you've considered

No response

Additional context

No response
@tustvold tustvold added the enhancement New feature or request label Sep 9, 2024
@tustvold tustvold mentioned this issue Sep 9, 2024
Open
@alamb
Copy link
Contributor

alamb commented Sep 9, 2024

I recommend two things:

  1. Write a blog with background and explanation of why using two threadpools is important with DataFusion and examples of how to do it
  2. Add additional documentation (ideally linking to the blog) with a summary and linking to the blog with content.

@alamb alamb changed the title Document DataFusion Threading Document DataFusion Threading (and how to separate IO and CPU bound work) Sep 9, 2024
@tustvold tustvold mentioned this issue Oct 7, 2024
Open
@ozankabak
Copy link
Contributor

I think it'd be great to have a good documentation on this.

@alamb
Copy link
Contributor

alamb commented Oct 25, 2024

I think it'd be great to have a good documentation on this.

100% agree -- @itsjunetime and @tustvold are working on a bit of it in apache/arrow-rs#6612. I'll try and help with the documentation as well

@alamb alamb mentioned this issue Nov 6, 2024
Open
@alamb alamb changed the title Document DataFusion Threading (and how to separate IO and CPU bound work) Document DataFusion Threading / tokio runtimes (how to separate IO and CPU bound work) Nov 11, 2024
@alamb alamb mentioned this issue Nov 13, 2024
Open
@alamb alamb self-assigned this Nov 14, 2024
This was referenced Nov 14, 2024
Merged
Draft
@alamb
Copy link
Contributor

alamb commented Nov 16, 2024

Documentation

I hope to work on the example a bit more shortly

This was referenced Nov 17, 2024
Open
Merged
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@alamb alamb

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add example for using a separate threadpool for CPU bound work alamb/datafusion
3 participants
@alamb @tustvold @ozankabak