Add global option process_extra_env for every Process call

[grihabor]

Precompiled binaries that were not created for NixOS usually have a so-called link-loader hardcoded into them. On Linux/x86_64 this is for example /lib64/ld-linux-x86-64.so.2. for glibc. NixOS, on the other hand, usually has its dynamic linker in the glibc package in the Nix store and therefore cannot run these binaries.

The solution is to use nix-ld, but it only works if you set NIX_LD env variable for every process call.

This pr adds a global option process_extra_env to set arbitrary env variables for every pants Process call.

[benjyw]

Thanks for tackling this!

So we actually already have an option for this: https://www.pantsbuild.org/dev/reference/subsystems/subprocess-environment

The problem is that we only actually apply those env vars in some Python-related processes. But the documentation of that option is very clear that it should apply to all processes. So this is simply broken today.

Therefore, instead of the new option, could you make this existing option apply to all processes, and remove the special handling in the python cases? This would also close #16565 .

[grihabor]

@benjyw I'm getting a bunch of rule errors. It looks like SubprocessEnvironment uses EnvironmentAware class which afaik needs an EnvironmentName which means that I can't use it in the -> Process rule, is this correct? If so, shall we create a different option for Process env vars?

[benjyw]

Your rule needs to request a SubprocessEnvironment.EnvironmentAware I believe. That is the trick by which different environments can set different env vars (and more generally is the trick by which different environments can have different values for a subsystem).

[grihabor]

This doesn't work, because I have to Get(EnvironmentVariables) inside to get the actual values, and it still fails with a missing rule. 26f69bf

[benjyw]

Ah, sorry, yes, you were doing the right thing to begin with, I read SubprocessEnvironmentVars as SubprocessEnvironment.

[benjyw]

The get_subprocess_environment rule takes care of going from SubprocessEnvironment.EnvironmentAware to actual dict of env vars.

[benjyw]

I'll take a closer look in a bit

[benjyw]

Oh boy, yes, this is more complicated than I first appreciated.

However, the complication is justified, and adding a new option won't solve it, unless the new option is not EnvironmentAware, which seems simply incorrect.

I will noodle on this a bit more.

[benjyw]

I've asked @stuhood to take a look at this, as his engine expertise is second to none.

[grihabor]

Thank you!

[grihabor]

@benjyw @stuhood Any updates on this?