Add graph-based pod stop #25169
Add graph-based pod stop #25169
Conversation
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: mheon The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
the
approved
|
Tagging no new tests as the existing pod stop/remove tests should exercise this. Would be nice to test the ordering aspect but I'm not sure if that's doable without being very racy. |
|
For tests one thing that we could do is check the |
libpod/container_api.go
Outdated
| exists, err := c.runtime.state.HasContainer(c.ID()) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| if !exists { | ||
| return fmt.Errorf("container %s does not exist in database: %w", c.ID(), define.ErrNoSuchCtr) | ||
| } |
There was a problem hiding this comment.
That adds a extra db queries overhead, I am not sure we need this at all.
Technically if the pod doesn't exists you could just ignore the error, i.e. only take locks in an if err == nil {} block.
Then the other code would return its normal error anyway.
There was a problem hiding this comment.
Oh, I like that. Will fix.
libpod/container_api.go
Outdated
| // Have to lock the pod the container is a part of. | ||
| // This prevents running `podman start` at the same time a | ||
| // `podman pod stop` is running, which could lead to wierd races. | ||
| // Pod locks come before container locks, so do this first. | ||
| if c.config.Pod != "" { |
There was a problem hiding this comment.
We should do the same thing around stop for consistency
The intention behind this is to stop races between `pod stop|start` and `container stop|start` being run at the same time. This could result in containers with no working network (they join the still-running infra container's netns, which is then torn down as the infra container is stopped, leaving the container in an otherwise unused, nonfunctional, orphan netns. Locking the pod (if present) in the public container start and stop APIs should be sufficient to stop this. Signed-off-by: Matt Heon <[email protected]>
mheon
force-pushed
the
graph_stop
branch
2 times, most recently
from
9c9d10f to
bee225e
Compare
|
Oops. Forgot to wait for the parallel executors to stop creating horrible races. Fixed now. |
mheon
force-pushed
the
graph_stop
branch
11 times, most recently
from
ed5adc4 to
3aa18df
Compare
|
Ephemeral COPR build failed. @containers/packit-build please check. |
|
Cockpit tests failed for commit 3aa18df. @martinpitt, @jelly, @mvollmer please check. |
mheon
force-pushed
the
graph_stop
branch
3 times, most recently
from
2fb56ab to
8f21503
Compare
First, refactor our existing graph traversal code to improve code sharing. There still isn't much sharing between inward traversal (stop, remove) and outward traversal (start) but stop and remove are sharing most of their code, which seems a positive. Second, add a new graph-traversal function to stop containers. We already had start and remove; stop uses the newly-refactored inward-traversal code which it shares with removal. Third, rework the shared stop/removal inward-traversal code to add locking. This allows parallel execution of stop and removal, which should improve the performance of `podman pod rm` and retain the performance of `podman pod stop` at about what it is right now. Fourth and finally, use the new graph-based stop when possible to solve unordered stop problems with pods - specifically, the infra container stopping before application containers, leaving those containers without a working network. Fixes https://issues.redhat.com/browse/RHEL-76827 Signed-off-by: Matt Heon <[email protected]>
|
The cockpit test failure from above is essentially this:
This feels related to this change? The latest push passed, but in commit 3aa18df it also passed in F41 and only failed once in Rawhide -- so this feels like a race condition, or possibly file system order etc., i.e. something not reliably reproducible? Does that ring a bell? |
|
@martinpitt Most of our tests fail as well so yes this patch is broken and cannot be merged like that. |
Implement a graph-based pod stop and use it by default, ensuring that containers stop in a dependency-based order. This prevents race conditions where application containers stopped after the infra container, meaning they did not have functional networking for the last seconds before they stopped, potentially causing unexpected application errors.
As a pleasant side-effect, make removing containers within a pod parallel, which should improve performance.
Full details in commit descriptions.
Does this PR introduce a user-facing change?