Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

libct: speedup process.Env handling #4325

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

kolyshkin
Copy link
Contributor

@kolyshkin kolyshkin commented Jun 24, 2024

This is a rework/carry of #1983.


The current implementation sets all the environment variables passed in
Process.Env in the current process, one by one, then uses os.Environ to
read those back.

As pointed out in [1], this is slow, as runc calls os.Setenv for every
variable, and there may be a few thousands of those. Looking into how
os.Setenv is implemented, it is indeed slow, especially when cgo is
enabled.

Looking into why it was implemented the way it is, I found commit
9744d72c and traced it to [2], which discusses the actual reasons.
It boils down to these two:

 - HOME is not passed into container as it is set in setupUser by
   os.Setenv and has no effect on config.Env;
 - there is a need to deduplication of environment variables.

Yet it was decided in [2] to not go ahead with this patch, but
later [3] was opened with the carry of this patch, and merged.

Now, from what I see:

1. Passing environment to exec is way faster than using os.Setenv and
   os.Environ (tests show ~20x speed improvement in a simple Go test,
   and ~3x improvement in real-world test, see below).
2. Setting environment variables in the runc context may result is some
   ugly side effects (think GODEBUG, LD_PRELOAD, or _LIBCONTAINER_*).
3. Nothing in runtime spec says that the environment needs to be
   deduplicated, or the order of preference (whether the first or the
   last value of a variable with the same name is to be used). We should
   stick to what we have in order to maintain backward compatibility.

So, this patch:
 - switches to passing env directly to exec;
 - adds deduplication mechanism to retain backward compatibility;
 - takes care to set PATH from process.Env in the current process
   (so that supplied PATH is used to find the binary to execute),
   also to retain backward compatibility;
 - adds HOME to process.Env if not set.

The benchmark added by the previous commit shows ~3x improvement:

                        │   before    │                after                 │
                        │   sec/op    │    sec/op     vs base                │
        ExecInBigEnv-20   61.53m ± 1%   21.87m ± 16%  -64.46% (p=0.000 n=10)

Remaining questions (and my answers to those):

Q: Are there any potential regressions (for example, from not setting values from process.Env to the current process?
A: Pprobably not; if yes, someone is exploiting some undocumented behavior.

Q: Should deduplication show warnings (maybe promoted to errors later)?
A: For best backward compatibility, let's not do that. Can always be added later (maybe with some addition to runtime-spec).

Q: Whether a default for PATH (e.g "/bin:/usr/bin" should be added, when PATH is not set.
A: This needs to be done in runtime-spec first (document the default for PATH, then add it to runtimes).

@@ -277,7 +278,7 @@ func (l *linuxStandardInit) Init() error {

if l.dmzExe != nil {
l.config.Args[0] = name
return system.Fexecve(l.dmzExe.Fd(), l.config.Args, os.Environ())
return system.Fexecve(l.dmzExe.Fd(), l.config.Args, l.config.Env)
Copy link
Member

@lifubang lifubang Jun 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before we merge #4323 , maybe we should also need to include l.config.Env when running StartContainer hook.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is definitely something to think about. Maybe it makes sense to do it selectively (for those hooks that are run inside the container -- AFAIR not all of them are)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better late than never -- thank you @lifubang, I finally got it now.

This is now fixed and a test case is added.

I still prefer something like #4323 as it's much more straightforward.

@kolyshkin kolyshkin force-pushed the opt-env branch 2 times, most recently from 9b67a06 to b0512bc Compare June 27, 2024 17:09
@thaJeztah
Copy link
Member

Nothing in runtime spec says that the environment needs to be deduplicated, or the order of preference (whether the first or the last value of a variable with the same name is to be used). In C (Linux/glibc), the first value is used. In Go, it's the last one.

Interesting; I didn't know about that discrepancy; sounds like something that wouldn't hurt to define in the OCI spec; given that all original implementations were in Go, and ISTR Docker also had its own code to remove duplicates, I'm inclined to describe that as the expected behavior (possibly recommend producers of the OCI config to handle duplicates themselves to prevent any ambiguity).

@kolyshkin
Copy link
Contributor Author

Nothing in runtime spec says that the environment needs to be deduplicated, or the order of preference (whether the first or the last value of a variable with the same name is to be used). In C (Linux/glibc), the first value is used. In Go, it's the last one.

I was not entirely correct here. Let me rephrase this:

  1. When the environment is set using glibc's setenv(3) or putenv(3), or Go's os.Putenv, the value from the latter call for a given key takes precedence. This is obvious.
  2. If the existing environment is not deduplicated (such as when you supply it directly to execve), both glibc's getenv(3) and Go's os.Getenv return the value of the first element.
  3. When using a higher level primitives (from os/exec) in Go, the environment passed is deduplicated in a way that the last value of a key is used. In glibc, there is no such deduplication (as far as I can see), and thus the first value of the key will be used. This is the difference between Go and C.

On an unrelated note, I also took a look at crun and it seems it is following runc logic, calling clearenv(3) followed by putenv(3) for every item in Process.Env, when using environ for execve -- which result in the latter value of the same env var being used. There is probably no room for a similar optimization as putenv in glibc is not thread safe and there are no mutexes to be held etc.

@kolyshkin
Copy link
Contributor Author

PTAL @opencontainers/runc-maintainers

@lifubang
Copy link
Member

lifubang commented Sep 3, 2024

PTAL @opencontainers/runc-maintainers

LGTM, it’s a big step.
But as I mentioned in #4325 (comment), I suggest to add these envs to hook’s process execution before we have a conclusion for #4323.

Copy link
Member

@rata rata left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kolyshkin left some comments, but seems great it makes a difference! This is marked for 1.3, though, but I guess you asking for reviews now means you want this for 1.2?

I'm fine with this in 1.2 or 1.3

libcontainer/env.go Show resolved Hide resolved
libcontainer/env.go Outdated Show resolved Hide resolved
libcontainer/env.go Show resolved Hide resolved
libcontainer/env.go Show resolved Hide resolved
Copy link
Member

@rata rata left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@kolyshkin
Copy link
Contributor Author

OTOH I think this is ready for more reviews and potential inclusion.

@rata
Copy link
Member

rata commented Oct 22, 2024

The PR to release 1.2 is out already, and it will be released today. I'd prefer to merge this after the 1.2.0 release, if the speedup is nice, we can add it in a patch release (or maybe to a 1.3 release in 3 months ;))

@kolyshkin
Copy link
Contributor Author

This is totally 1.3 material; let me rebase

@cyphar
Copy link
Member

cyphar commented Dec 4, 2024

FWIW, I sent a patch to improve the performance of os.Clearenv (with the patch it's ~30% faster for an environment with 1000 variables set), though this is fixing other issues in addition to the os.Clearenv one. golang/go#70672

@kolyshkin
Copy link
Contributor Author

I've updated the patch (with somewhat better commit message and prepareEnv documentation. I think we can go ahead and merge this for v1.3, and leave the question raised in #4325 (comment) for later.

@kolyshkin kolyshkin requested a review from lifubang December 5, 2024 00:03
@lifubang
Copy link
Member

lifubang commented Dec 5, 2024

and leave the question raised in #4325 (comment) for later

But it's a break change for StartContainer hook, though this hook may be not wide spread used in practice.
Maybe we should add this break to CHAGELOG because we may forget it when making a release.

@cyphar
Copy link
Member

cyphar commented Dec 7, 2024

Maybe we should add this break to CHANGELOG because we may forget it when making a release.

I think we need to start adding things to the changelog in the implementation PR in general because we have missed breaking changes before (the most recent example I can think of is #3468, but I've definitely made the same mistake before as well -- it's very easy to skim over a patch when reading through the log to write the release notes).

Here's what it shows on my laptop (with -count 10 -benchtime 10s,
summarized by benchstat):

	                │   sec/op    │
	ExecTrue-20       8.477m ± 2%
	ExecInBigEnv-20   61.53m ± 1%

Signed-off-by: Kir Kolyshkin <[email protected]>
This is a slight refactor of TestExecInEnvironment, making it more
strict wrt checking the exec output.

1. Explain why DEBUG is added twice to the env.
2. Reuse the execEnv for the check.
3. Make the check more strict -- instead of looking for substrings,
   check line by line.
4. Add a check for extra environment variables.

Signed-off-by: Kir Kolyshkin <[email protected]>
@kolyshkin kolyshkin force-pushed the opt-env branch 2 times, most recently from ae9f5e6 to 03faf10 Compare December 23, 2024 01:59
This is to ensure that changes in Process.Env handling won't affect
StartContainer hook.

Reported-by: lfbzhm <[email protected]>
Signed-off-by: Kir Kolyshkin <[email protected]>
1. Make CommandHook.Command a pointer, which reduces the amount of data
   being copied when using hooks, and allows to modify command hooks.

2. Add SetDefaultEnv, which is to be used by the next commit.

Signed-off-by: Kir Kolyshkin <[email protected]>
The current implementation sets all the environment variables passed in
Process.Env in the current process, one by one, then uses os.Environ to
read those back.

As pointed out in [1], this is slow, as runc calls os.Setenv for every
variable, and there may be a few thousands of those. Looking into how
os.Setenv is implemented, it is indeed slow, especially when cgo is
enabled.

Looking into why it was implemented the way it is, I found commit
9744d72 and traced it to [2], which discusses the actual reasons.
It boils down to these two:

 - HOME is not passed into container as it is set in setupUser by
   os.Setenv and has no effect on config.Env;
 - there is a need to deduplication of environment variables.

Yet it was decided in [2] to not go ahead with this patch, but
later [3] was opened with the carry of this patch, and merged.

Now, from what I see:

1. Passing environment to exec is way faster than using os.Setenv and
   os.Environ (tests show ~20x speed improvement in a simple Go test,
   and ~3x improvement in real-world test, see below).
2. Setting environment variables in the runc context may result is some
   ugly side effects (think GODEBUG, LD_PRELOAD, or _LIBCONTAINER_*).
3. Nothing in runtime spec says that the environment needs to be
   deduplicated, or the order of preference (whether the first or the
   last value of a variable with the same name is to be used). We should
   stick to what we have in order to maintain backward compatibility.

So, this patch:
 - switches to passing env directly to exec;
 - adds deduplication mechanism to retain backward compatibility;
 - takes care to set PATH from process.Env in the current process
   (so that supplied PATH is used to find the binary to execute),
   also to retain backward compatibility;
 - adds HOME to process.Env if not set;
 - ensures any StartContainer CommandHook entries with no environment
   set explicitly are run with the same environment as before. Thanks
   to @lifubang who noticed that peculiarity.

The benchmark added by the previous commit shows ~3x improvement:

	                │   before    │                after                 │
	                │   sec/op    │    sec/op     vs base                │
	ExecInBigEnv-20   61.53m ± 1%   21.87m ± 16%  -64.46% (p=0.000 n=10)

[1]: opencontainers#1983
[2]: docker-archive/libcontainer#418
[3]: docker-archive/libcontainer#432

Signed-off-by: Kir Kolyshkin <[email protected]>
@kolyshkin
Copy link
Contributor Author

Maybe we should add this break to CHANGELOG because we may forget it when making a release.

I think we need to start adding things to the changelog in the implementation PR in general because we have missed breaking changes before (the most recent example I can think of is #3468, but I've definitely made the same mistake before as well -- it's very easy to skim over a patch when reading through the log to write the release notes).

@cyphar the above incompatibility issue reported by @lifubang is now fixed, but in the process I had to add a small change to libcontainer API, which is documented in the CHANGELOG now.

Doesn't mean this PR won't break something else, but at least I am not aware of anything.

Comment on lines +201 to +208

// In case we have any StartContainer hooks to run, and they don't
// have environment configured explicitly, make sure they will be run
// with the same environment as container's init.
if h := l.config.Config.Hooks[configs.StartContainer]; len(h) > 0 {
h.SetDefaultEnv(l.config.Env)
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lifubang FYI this is the fix to the startContainer hook issue wrt environment you've reported back in June.


// In case we have any StartContainer hooks to run, and they don't
// have environment configured explicitly, make sure they will be run
// with the same environment as container's init.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add a note that this behavior is not part of runtime-spec, but a de-facto historical thing we afraid to change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants