Avoid race conditions when installing multiple dependencies from the same git repository. #9658
Conversation
|
Thank you so much |
There was a problem hiding this comment.
LGTM. Just a minor nitpick.
I thought a bit about how to write a unit test but even with mocking I could not come up with a good test. Maybe, we can just add kind of a smoke test like test_executor_should_append_subdirectory_for_git but with two packages from the same git repo and check that both are still installed. At least, the order should be fix now, since they are installed serially.
|
A workaround for now, that works for 1.8.3
|
|
@gustavgransbo @TheSven73 Just to be sure before I take another detailed look and potentially merge it: If we outsource #9658 (comment) into a separate issue/PR, this PR is ready from your end, isn't it? |
Yes, from my perspective it's ready to be merged! |
Multiple installs from the same git repository causes arace condition when the repository is cloned. This commit only allows one operation per repository to be executed by the parallel workers. Any extra operation will be performed serially. Since the git repository is cached after the first operation, this is blazingly fast. Resolves: python-poetry#6958
Operations for a single repository are grouped and executed serially by a single worker.
This simplifies the subsequent if/else block.
This reverts commit 58e5338.
radoering
force-pushed
the
feature/avoid-git-race-condition
branch
from
423f21d
to
259100f
Compare
There was a problem hiding this comment.
Thanks for the profound review @TheSven73 and thanks for keeping up with it @gustavgransbo .
|
Apologies for the time-fragmented review. And thanks so much @gustavgransbo for tackling this challenging issue !! |
|
Thanks a lot for the thorough review @TheSven73! It’s a privilege to receive such well thought out suggestions, and the resulting solution was much better for it. |
When installing packages from different directories of a forked monorepo, a race condition may occur, where multiple git clients are interacting in parallel with the same git repository. Fix by serializing git operations that interact with the same git repository. This makes the test succeed. Remove xfail. Links: python-poetry#9658 (comment) https://github.com/orgs/python-poetry/discussions/9718#discussioncomment-10785589
When installing packages from different directories of a forked monorepo, a race condition may occur, where multiple git clients are interacting in parallel with the same git repository. Fix by serializing git operations that interact with the same git repository. This makes the test succeed. Remove xfail. Links: python-poetry#9658 (comment) https://github.com/orgs/python-poetry/discussions/9718#discussioncomment-10785589
When installing packages from different directories of a forked monorepo, a race condition may occur, where multiple git clients are interacting in parallel with the same git repository. Fix by serializing git operations that interact with the same git repository. This makes the test succeed. Remove xfail. Links: python-poetry#9658 (comment) https://github.com/orgs/python-poetry/discussions/9718#discussioncomment-10785589
When installing packages from different directories of a forked monorepo, a race condition may occur, where multiple git clients are interacting in parallel with the same git repository. Fix by serializing git operations that interact with the same git repository. This makes the test succeed. Remove xfail. Links: python-poetry#9658 (comment) https://github.com/orgs/python-poetry/discussions/9718#discussioncomment-10785589
When installing packages from different directories of a forked monorepo, a race condition may occur, where multiple git clients are interacting in parallel with the same git repository. Fix by serializing git operations that interact with the same git repository. This makes the test succeed. Remove xfail. Links: #9658 (comment) https://github.com/orgs/python-poetry/discussions/9718#discussioncomment-10785589
…same git repository. (python-poetry#9658)
When installing packages from different directories of a forked monorepo, a race condition may occur, where multiple git clients are interacting in parallel with the same git repository. Fix by serializing git operations that interact with the same git repository. This makes the test succeed. Remove xfail. Links: python-poetry#9658 (comment) https://github.com/orgs/python-poetry/discussions/9718#discussioncomment-10785589
|
This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Multiple installs from the same git repository causes a race condition when the repository is cloned.
This change only allows one operation per repository to be executed by the parallel workers.
Any extra operation on the same git repository will be performed serially.
Since the git repository is cached after the first operation, this is fast.
Building on the setup of @JonathanRayner, I was able to consistently reproduce the error in this repository.
Thank you @JonathanRayner for the hard work!
Pull Request Check List
Resolves: #6958
The race condition occurs inside poetry.vcs.git.Git.clone, this method is mocked in all tests: conftest.py.
This makes the actual race condition hard to test.
Also, since the race condition required quite a bit of external setup, I think it could be challenging to add a good test for this.
This change should not require any new documentation.