-
Notifications
You must be signed in to change notification settings - Fork 956
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Duplicated process and memory leakage for evaluation process in all_gather #3147
Comments
@SangbumChoi definitely open to trying out something more efficient! Best case scenario we have a flag to use |
5 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
System Info
Information
Tasks
no_trainer
script in theexamples
folder of thetransformers
repo (such asrun_no_trainer_glue.py
)Reproduction
Related issue
huggingface/transformers#15466
https://github.com/huggingface/transformers/pull/28769/files
Expected behavior
accelerate/src/accelerate/utils/operations.py
Line 353 in 55136b8
All the accelerate gather function is stricted to
all_gather
. However, there are also the way of usinggather
in main process to calculate the evaluation process. If we useall_gather
for the evaluation process and pass it to cpu it will cost n times (n is number of process). However we only require to gather the distributed variable to one place to calculate.What do you think about this?
https://github.com/facebookresearch/detectron2/blob/ebe8b45437f86395352ab13402ba45b75b4d1ddb/detectron2/utils/comm.py#L188
The text was updated successfully, but these errors were encountered: