-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Nx 19 not properly loading environment variables in parallel jobs if daemon disabled #27586
Closed
1 of 4 tasks
Labels
Comments
I would be willing to contribute to the fix, maybe with a bit of guidance/explanations about the task executor and environment variable loading. My assumption is that reading the |
FrozenPandaz
pushed a commit
that referenced
this issue
Sep 17, 2024
<!-- Please make sure you have read the submission guidelines before posting an PR --> <!-- https://github.com/nrwl/nx/blob/master/CONTRIBUTING.md#-submitting-a-pr --> <!-- Please make sure that your commit message follows our format --> <!-- Example: `fix(nx): must begin with lowercase` --> <!-- If this is a particularly complex change or feature addition, you can request a dedicated Nx release for this pull request branch. Mention someone from the Nx team or the `@nrwl/nx-pipelines-reviewers` and they will confirm if the PR warrants its own release for testing purposes, and generate it for you if appropriate. --> ## Current Behavior <!-- This is the behavior we have today --> - between this.tasksSchedule.scheduleNextTasks() and this.tasksSchedule.nextTask(), scheduledTasks are different when in parallel, so when calling `processAllScheduledTasks`, it does not process the all scheduled tasks, so it did not populate env. ## Expected Behavior <!-- This is the behavior we should expect with the changes in this PR --> ## Related Issue(s) <!-- Please link the issue being fixed so it gets closed when this is merged. --> Fixes #27586
FrozenPandaz
pushed a commit
that referenced
this issue
Sep 18, 2024
<!-- Please make sure you have read the submission guidelines before posting an PR --> <!-- https://github.com/nrwl/nx/blob/master/CONTRIBUTING.md#-submitting-a-pr --> <!-- Please make sure that your commit message follows our format --> <!-- Example: `fix(nx): must begin with lowercase` --> <!-- If this is a particularly complex change or feature addition, you can request a dedicated Nx release for this pull request branch. Mention someone from the Nx team or the `@nrwl/nx-pipelines-reviewers` and they will confirm if the PR warrants its own release for testing purposes, and generate it for you if appropriate. --> ## Current Behavior <!-- This is the behavior we have today --> - between this.tasksSchedule.scheduleNextTasks() and this.tasksSchedule.nextTask(), scheduledTasks are different when in parallel, so when calling `processAllScheduledTasks`, it does not process the all scheduled tasks, so it did not populate env. ## Expected Behavior <!-- This is the behavior we should expect with the changes in this PR --> ## Related Issue(s) <!-- Please link the issue being fixed so it gets closed when this is merged. --> Fixes #27586 (cherry picked from commit adc1abd)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Current Behavior
Given an Nx repo, if:
NX_DAEMON
to be false)t
is present in enough projectsp
innpx nx run-many --target=t --parallel=p
Then some targets do not start with the environment variables from
.env
files loaded in their memory.I am aware of how imprecise this description is. Please see example repo below and instructions on how to reproduce.
I attach a task tree that triggers this behaviour in my computer. The
echo-vars
targets do not find the environment variables.Example logs when it fails:
Expected Behavior
Changing the number of tasks being run, their order, the
--parallel
flag, or setting on/off the daemon should not alter the behaviour of Nx when running targets.Environment variables from
.env
files should always be provided to targets. Here is an example the logs after running the same command but less parallel jobs:GitHub Repo
https://github.com/mpsanchis/nx-19-example
Steps to Reproduce
yarn
andnode
yarn install --frozen-lockfile
.env
variables in thefake/
dir by runningnpx nx run var-creator:create-vars
export NX_DAEMON=false
(equivalently for this exercise,export CI=true
)npx nx run-many --target=release-me --parallel=1 --skip-nx-cache --verbose
npx nx run-many --target=release-me --parallel=8 --skip-nx-cache --verbose
Note the difference when running the tasks with different level of parallelism. With
parallel=8
, the targets aren't provided with all the environment variables: the ones coming from the.env
files are not inprocess.env
.There are also differences if you pass
--projects=<comma,separated,projectNames>
: one project will always work, but if you add many, it starts to fail.Nx Report
Failure Logs
No response
Package Manager Version
No response
Operating System
Additional Information
Migrating to Nx 19 made some of our builds in CI break. It has an erratic behaviour since it depends of the number of tasks being dispatched to Nx. I had to match the task graph we had in the failing repo.
In the MacOS that ran this (M2 with 8+4 cores), up to
--parallel=4
works OK. Then:5
, one job fails6
and7
, two jobs fail>=8
, all three jobs failHowever it's not 100% the same in another repo that also contains the
fake/
folder. The trend is the same, though: the more tasks Nx executes, and the more in parallel it attempts to do it, the less chances that the tasks get the env vars.The text was updated successfully, but these errors were encountered: