Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallelize DE methods that support it #613

Open
grst opened this issue May 27, 2024 · 0 comments
Open

Parallelize DE methods that support it #613

grst opened this issue May 27, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@grst
Copy link
Collaborator

grst commented May 27, 2024

Description of feature

Some of the methods are embarrassingly parallel, e.g. statsmodels, wilcoxon test.

I suggest to use the following snippet from scirpy:

  • using joblib it is more robust than multiprocessing
  • joblib natively supports the dask backend, so you get out-of-machine support for free.

https://github.com/scverse/scirpy/blob/443e59e6245b917e87972f87df350ae4f429d011/src/scirpy/util/__init__.py#L567-L579

def _parallelize_with_joblib(delayed_objects, *, total=None, **kwargs):
    """Wrapper around joblib.Parallel that shows a progressbar if the backend supports it.

    Progressbar solution from https://stackoverflow.com/a/76726101/2340703
    """
    try:
        return tqdm(Parallel(return_as="generator", **kwargs)(delayed_objects), total=total)
    except ValueError:
        logging.info(
            "Backend doesn't support return_as='generator'. No progress bar will be shown. "
            "Consider setting verbosity in joblib.parallel_config"
        )
        return Parallel(return_as="list", **kwargs)(delayed_objects)

https://github.com/scverse/scirpy/blob/443e59e6245b917e87972f87df350ae4f429d011/src/scirpy/ir_dist/metrics.py#L231-L233

block_results = _parallelize_with_joblib(
         (joblib.delayed(self._compute_block)(*block) for block in blocks), total=len(blocks), n_jobs=self.n_jobs
)

Migrated from scverse/multi-condition-comparisions#16

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant