Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nx 19 not properly loading environment variables in parallel jobs if daemon disabled #27586

Closed
1 of 4 tasks
mpsanchis opened this issue Aug 22, 2024 · 1 comment · Fixed by #27889
Closed
1 of 4 tasks
Assignees
Labels
scope: core core nx functionality type: bug

Comments

@mpsanchis
Copy link
Contributor

Current Behavior

Given an Nx repo, if:

  • Nx daemon is disabled (either because Nx is running in CI, or simply because we set NX_DAEMON to be false)
  • Target t is present in enough projects
  • Tasks are run in parallel, with a high enough p in npx nx run-many --target=t --parallel=p

Then some targets do not start with the environment variables from .env files loaded in their memory.

I am aware of how imprecise this description is. Please see example repo below and instructions on how to reproduce.

I attach a task tree that triggers this behaviour in my computer. The echo-vars targets do not find the environment variables.

Screenshot 2024-08-22 at 11 37 18

Example logs when it fails:

Screenshot 2024-08-22 at 11 40 49

Expected Behavior

Changing the number of tasks being run, their order, the --parallel flag, or setting on/off the daemon should not alter the behaviour of Nx when running targets.

Environment variables from .env files should always be provided to targets. Here is an example the logs after running the same command but less parallel jobs:

Screenshot 2024-08-22 at 11 42 53

GitHub Repo

https://github.com/mpsanchis/nx-19-example

Steps to Reproduce

  1. Clone the repo provided above, and make sure you have yarn and node
  2. Run yarn install --frozen-lockfile
  3. Create a bunch of .env variables in the fake/ dir by running npx nx run var-creator:create-vars
  4. Disable the Nx daemon: export NX_DAEMON=false (equivalently for this exercise, export CI=true)
  5. Run npx nx run-many --target=release-me --parallel=1 --skip-nx-cache --verbose
  6. Run npx nx run-many --target=release-me --parallel=8 --skip-nx-cache --verbose

Note the difference when running the tasks with different level of parallelism. With parallel=8, the targets aren't provided with all the environment variables: the ones coming from the .env files are not in process.env.

There are also differences if you pass --projects=<comma,separated,projectNames>: one project will always work, but if you add many, it starts to fail.

Nx Report

Node           : 22.2.0
OS             : darwin-arm64
Native Target  : aarch64-macos
yarn           : 1.22.19

nx                 : 19.6.0-beta.6
@nx/js             : 19.6.0-beta.6
@nx/jest           : 19.6.0-beta.6
@nx/linter         : 19.6.0-beta.6
@nx/eslint         : 19.6.0-beta.6
@nx/workspace      : 19.6.0-beta.6
@nx/angular        : 19.6.0-beta.6
@nx/cypress        : 19.6.0-beta.6
@nx/devkit         : 19.6.0-beta.6
@nx/eslint-plugin  : 19.6.0-beta.6
@nx/plugin         : 19.6.0-beta.6
@nx/react          : 19.6.0-beta.6
@nrwl/tao          : 19.6.0-beta.6
@nx/web            : 19.6.0-beta.6
@nx/webpack        : 19.6.0-beta.6
typescript         : 5.5.3
---------------------------------------
Registered Plugins:
@nx/eslint/plugin
@nx/cypress/plugin
@nx/jest/plugin
---------------------------------------
Community plugins:
@ngrx/component-store : 18.0.2
@ngrx/effects         : 18.0.2
@ngrx/entity          : 18.0.2
@ngrx/operators       : 18.0.2
@ngrx/router-store    : 18.0.2
@ngrx/store           : 18.0.2
@ngrx/store-devtools  : 18.0.2
---------------------------------------
Local workspace plugins:
         my-plugin

Failure Logs

No response

Package Manager Version

No response

Operating System

  • macOS
  • Linux
  • Windows
  • Other (Please specify)

Additional Information

Migrating to Nx 19 made some of our builds in CI break. It has an erratic behaviour since it depends of the number of tasks being dispatched to Nx. I had to match the task graph we had in the failing repo.

In the MacOS that ran this (M2 with 8+4 cores), up to --parallel=4 works OK. Then:

  • With 5, one job fails
  • With 6 and 7, two jobs fail
  • With >=8, all three jobs fail

However it's not 100% the same in another repo that also contains the fake/ folder. The trend is the same, though: the more tasks Nx executes, and the more in parallel it attempts to do it, the less chances that the tasks get the env vars.

@mpsanchis
Copy link
Contributor Author

I would be willing to contribute to the fix, maybe with a bit of guidance/explanations about the task executor and environment variable loading. My assumption is that reading the .env files is not completed before all tasks are dispatched.

@FrozenPandaz FrozenPandaz added the scope: core core nx functionality label Aug 23, 2024
FrozenPandaz pushed a commit that referenced this issue Sep 17, 2024
<!-- Please make sure you have read the submission guidelines before
posting an PR -->
<!--
https://github.com/nrwl/nx/blob/master/CONTRIBUTING.md#-submitting-a-pr
-->

<!-- Please make sure that your commit message follows our format -->
<!-- Example: `fix(nx): must begin with lowercase` -->

<!-- If this is a particularly complex change or feature addition, you
can request a dedicated Nx release for this pull request branch. Mention
someone from the Nx team or the `@nrwl/nx-pipelines-reviewers` and they
will confirm if the PR warrants its own release for testing purposes,
and generate it for you if appropriate. -->

## Current Behavior
<!-- This is the behavior we have today -->
- between this.tasksSchedule.scheduleNextTasks() and
this.tasksSchedule.nextTask(), scheduledTasks are different when in
parallel, so when calling `processAllScheduledTasks`, it does not
process the all scheduled tasks, so it did not populate env.

## Expected Behavior
<!-- This is the behavior we should expect with the changes in this PR
-->

## Related Issue(s)
<!-- Please link the issue being fixed so it gets closed when this is
merged. -->

Fixes #27586
FrozenPandaz pushed a commit that referenced this issue Sep 18, 2024
<!-- Please make sure you have read the submission guidelines before
posting an PR -->
<!--
https://github.com/nrwl/nx/blob/master/CONTRIBUTING.md#-submitting-a-pr
-->

<!-- Please make sure that your commit message follows our format -->
<!-- Example: `fix(nx): must begin with lowercase` -->

<!-- If this is a particularly complex change or feature addition, you
can request a dedicated Nx release for this pull request branch. Mention
someone from the Nx team or the `@nrwl/nx-pipelines-reviewers` and they
will confirm if the PR warrants its own release for testing purposes,
and generate it for you if appropriate. -->

## Current Behavior
<!-- This is the behavior we have today -->
- between this.tasksSchedule.scheduleNextTasks() and
this.tasksSchedule.nextTask(), scheduledTasks are different when in
parallel, so when calling `processAllScheduledTasks`, it does not
process the all scheduled tasks, so it did not populate env.

## Expected Behavior
<!-- This is the behavior we should expect with the changes in this PR
-->

## Related Issue(s)
<!-- Please link the issue being fixed so it gets closed when this is
merged. -->

Fixes #27586

(cherry picked from commit adc1abd)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
scope: core core nx functionality type: bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants