Very Strange Cache Behaviour

[ChrisSargent]

Current Behavior

We have been having a lot of unusual issues with unexpected cache misses and after spending an inordinate amount of time trying to debug, we have found some very odd behaviour.

We have found that deleting and re-creating the same file in VSCode can result in a cache miss. This was a massive surprise to us since everything we had read points to Nx only using the file contents to create its hashes - and worth pointing out at this point that we were simply deleting the file and then discarding the change via git. Of course, this results in a 'new' file with a new created / modified date.....

But then, we tried simply deleting the file through Finder and then hitting cmd+z to undo the change, whereby created & modified dates do not get changed, we saw the exact same behaviour. Simply deleting the file brought a whole bunch of changes in the .nx/workspace-data/file-map.json file. Running the same task results in a cache miss.

Note that re-performing these steps does NOT result in a cache miss on the subsequent runs.... i.e. performing the delete / restore then doing a run, then doing the delete / restore / run again is cached as expected.

We have a pretty large repo with >100 projects so we wondered if it was something in our setup but we were able to recreate this in a brand new, tiny repo which is linked below.

Expected Behavior

Deleting / restoring files with the exact same content should not result in a cache mis.s

GitHub Repo

https://github.com/noah-labs/nx-cache-issue

Steps to Reproduce

Steps are provide in the github repro too:

Create an Nx Project

1. npx create-nx-workspace --pm yarn
2. Name the workspace
3. Select none for 'stack'
4. Select "Package-based monorepo"
5. Select "Do It Later"
6. Select "No" to remote caching

(In my case here, I also removed the .nx/workspace-data folder from the git ignore so I could trace changes)

Create a JS Lib

1. npx nx g @nx/js:lib mylib
2. Select "None"
3. Select "None"
4. Select "As provided"

Run the tasks

1. Run yarn nx run-many -t lint
2. This populates the local cache
3. Run yarn nx run-many -t lint again, result is cached
4. In Finder, delete mylib/src/index.ts and then hit cmd+z to undo
5. You will see that the .nx/workspace-data/file-map.json file changes significantly
6. Run yarn nx run-many -t lint again, result is NOT cached
7. Repeat the delete, undo and lint run, result is cached again
8. Now note that running yarn nx reset followed by the lint command will revert the changes that happened to the .nx/workspace-data/file-map.json in step 5
9. Subsequent deletes to not alter the file-map.json file now

Nx Report

Node           : 20.17.0
OS             : darwin-arm64
Native Target  : aarch64-macos
yarn           : 4.4.1

nx                 : 19.7.2
@nx/js             : 19.7.2
@nx/linter         : 19.7.2
@nx/eslint         : 19.7.2
@nx/workspace      : 19.7.2
@nx/devkit         : 19.7.2
@nx/eslint-plugin  : 19.7.2
@nrwl/tao          : 19.7.2
typescript         : 5.5.4
---------------------------------------
Registered Plugins:
@nx/eslint/plugin

Failure Logs

No response

Package Manager Version

No response

Operating System

• macOS
• Linux
• Windows
• Other (Please specify)

Additional Information

No response

[ChrisSargent]

A little bit more information here:

1. Deleting the repo, recloning and performing the steps again results in the exact same behaviour
2. Opening, then editing a file and then saving then undoing changes does HIT the cache.

Tried running every stat I (chatgpt) could think of on a file change that result in a cache MISS.

And the same stats / diff on a file which resulted in a cache MISS

[ChrisSargent]

Also worth noting that the hash reported by .nx/workspace-data/file-map.json did not change when we saw cache misses.

Final comment for now - using affected does not suffer from the same problem.

[ChrisSargent]

We're also seeing this behaviour in more complex use cases too. e.g. When we're building our docs, we run rm -rf to clean some of the generated files which get re-generated on a build. These files need to be included in the cache calculation. However, currently, despite the contents of the file not changing at all, we see that because they're created 'new', it always results in cache misses.

Of course, we can safely exclude those files from the "inputs" because they only change if other files change. Still, it's very unexpected and not documented anywhere - and in fact the docs say "the contents of files", here: https://nx.dev/reference/inputs#source-files and here: https://nx.dev/reference/inputs#outputs-of-dependent-tasks

[ChrisSargent]

I have now discovered the following which is also very unexpected.

Even if I have these lines in my .nxignore:

apps/fe/docs/api-reference/
apps/fe/docs/static/files/oas-schema.generated.json

"Changes" to those files are causing cache misses (even though the contents of the file are actually not changing at all as above)

If I exclude the very same files from the "inputs" array, the changes are not causing a cache miss, e.g like this

"inputs": [
  "production",
  "^production",
  "!{projectRoot}/docs/api-reference/**/*",
  "!{projectRoot}/static/files/oas-schema.generated.json"
],

I worry quite a lot that this behaviour will be the same with .gitignore files which would cause an awful lot of cache misses.

Scratch that, had a typo in my path in the nxigniore

[meeroslav]

Hi Chris, you should never remove .nx/workspace-data from .gitignore since it includes details about daemon process ID. Using something like git reset while daemon is running could lead to process exploding with

 NX   disk I/O error

Regarding the issue, I have identified the problem and will check how to solve it. Than you for reporting it.

[ChrisSargent]

Hi Chris, you should never remove .nx/workspace-data from .gitignore since it includes details about daemon process ID. Using something like git reset while daemon is running could lead to process exploding with

 NX   disk I/O error

Regarding the issue, I have identified the problem and will check how to solve it. Than you for reporting it.

we would never commit it normally no, but we were interested to see what was changing between runs to see if it gave some clues.

[madelk]

I've been seeing those disk I/O error appearing out of nowhere over the last few days and I've been telling teams to just keep retrying their builds

[ChrisSargent]

possibly related: #23106

[kylecannon]

add NX_DISABLE_DB: true as an env var to hopefully mitigate until a fix is live.

[meeroslav]

Hi, this should be fixed in the v20. @ChrisSargent can you please verify?

[meeroslav]

Fixed by #28024