Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

modernizing support for parallelism #866

Open
simonpcouch opened this issue Mar 8, 2024 · 3 comments
Open

modernizing support for parallelism #866

simonpcouch opened this issue Mar 8, 2024 · 3 comments
Labels
breaking change ☠️ API change likely to affect existing code feature a feature request or enhancement

Comments

@simonpcouch
Copy link
Contributor

Currently, tune uses foreach to facilitate parallel computing. As a group, we've discussed gradually deprecating our support for foreach in favor of the futureverse. The big ideas:

  • future does a better job identifying globals (i.e. no need for namespacing where we usually wouldn't, reassigning in strange situations), generating random numbers soundly, and handling errors, and comes with better progress support via progressr.
  • Newer R users are more likely to be familiar with the future user API (plan(multisession), etc) than the foreach one.
  • The futureverse is more actively developed/maintained than foreach.

Spent some time thinking through how this might come together this morning. The process could look something like this:

Timeline User API Developer API Backend
Now foreach foreach foreach
Proposed future (foreach supported) foreach (via doFuture) future
2/3 Years Out future future (maybe via furrr) future

By using doFuture as a backend, our codebase looks similar to how it does with foreach, which can help us with maintainability during the transition period where we support users using either the foreach user API (e.g. registerDoParallel()) or the future user API (i.e. plan(multisession)). Depending on the user's setup, we'd use foreach::`%do%`, foreach::`%doPar%`, or doFuture::`%doFuture%` where we currently just decided between the first two.

Currently, for error logging, tune uses ad-hoc machinery for logging when tuning in parallel and cli for logging sequentially. I'm not sure that we want we want to maintain a third logging system for parallelism with progressr during the transition period. Once we fully deprecate support for foreach, though, we should consider rewriting all of our error logging with progressr.

🚀🚀🚀

@simonpcouch simonpcouch added breaking change ☠️ API change likely to affect existing code feature a feature request or enhancement labels Mar 8, 2024
@simonpcouch
Copy link
Contributor Author

Next step for post-v1.2.0: revert e79aac1 to introduce a deprecation warning when using foreach.

@jrosell

This comment was marked as off-topic.

@simonpcouch

This comment was marked as off-topic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking change ☠️ API change likely to affect existing code feature a feature request or enhancement
Projects
None yet
Development

No branches or pull requests

2 participants