Bug: @snowpack/web-test-runner coverage

[jakesgordon]

If my source code is in a project subdirectory (e.g. /src), and I use a snowpack mount option in order to serve it (e.g. I mount /src to /) then any code coverage report generated by web test runner is broken because it fails to locate the physical source files. It works just fine if I'm serving my source directly from the root without the snowpack mount option.

I can reproduce with a stock vanilla snowpack/web-test-runner example project if I simply mount my code from a /src directory then the coverage report that is generated is broken - you can find the full example here

• https://github.com/jakesgordon/snowpack-coverage-bug

And for reference, the istanbul coverage report generates files that contain the following error.... it looks like its trying to locate source files by comping the CWD with the web import path, but without the (mounted) /src prefix it's not going to find them that way.

nable to lookup source: /home/jake/tmp/foo/example.js (ENOENT: no such file or directory, open '/home/jake/tmp/foo/example.js')
Error: Unable to lookup source: /home/jake/tmp/foo/example.js (ENOENT: no such file or directory, open '/home/jake/tmp/foo/example.js')
    at Context.defaultSourceLookup [as sourceFinder] (/home/jake/tmp/foo/node_modules/istanbul-lib-report/lib/context.js:17:15)
    at Context.getSource (/home/jake/tmp/foo/node_modules/istanbul-lib-report/lib/context.js:71:21)
    at annotateSourceCode (/home/jake/tmp/foo/node_modules/istanbul-reports/lib/html/annotator.js:213:40)
    at HtmlReport.onDetail (/home/jake/tmp/foo/node_modules/istanbul-reports/lib/html/index.js:409:33)
    at LcovReport. [as onDetail] (/home/jake/tmp/foo/node_modules/istanbul-reports/lib/lcov/index.js:28:23)
    at Visitor.value (/home/jake/tmp/foo/node_modules/istanbul-lib-report/lib/tree.js:38:38)
    at ReportNode.visit (/home/jake/tmp/foo/node_modules/istanbul-lib-report/lib/tree.js:88:21)
    at /home/jake/tmp/foo/node_modules/istanbul-lib-report/lib/tree.js:92:19
    at Array.forEach ()
    at ReportNode.visit (/home/jake/tmp/foo/node_modules/istanbul-lib-report/lib/tree.js:91:28)

[LarsDenBakker]

We run the istanbul coverage reporter as a separate process after testing, this looks up the files on the file system by default. We need to see if we can make it query the files back from the dev server again. Tracking in modernweb-dev/web#1212

[LarsDenBakker]

This is now fixed with @web/test-runner v0.12.6 or later. If you use other test runner packages, you might need to update those as well.

Note that you probably want to ignore the generated snowpack files from coverage:

// NODE_ENV=test - Needed by "@snowpack/web-test-runner-plugin"
process.env.NODE_ENV = 'test';

module.exports = {
  coverageConfig: {
    exclude: ['**/*/_snowpack/**/*'],
  },
  plugins: [require('@snowpack/web-test-runner-plugin')()],
};

[jakesgordon]

That's awesome. Thank you!

[ambrt]

I've upgraded @web/test-runner to 0.12.7 and it works for .svelte files. Thanks!

[jakesgordon]

Oops, I spoke too soon. Unfortunately this is still broken. I've upgraded the repro project to use the latest @web/test-runner and latest snowpack, but the code coverage report is still looking for the source relative to the root instead of relative to the snowpack mounted directory.

• https://github.com/jakesgordon/snowpack-coverage-bug

Note, paths are missing the /src in between the root snowpack-coverage-bug folder and the source files.

[ambrt]

Oops, I spoke too soon. Unfortunately this is still broken. I've upgraded the repro project to use the latest @web/test-runner and latest snowpack, but the code coverage report is still looking for the source relative to the root instead of relative to the snowpack mounted directory.

• https://github.com/jakesgordon/snowpack-coverage-bug

Note, paths are missing the /src in between the root snowpack-coverage-bug folder and the source files.

In your repo there is:

module.exports = {
  mount: {
    src: '/'
  },
};

and i won't work in my repo too.
When one will change it to:

module.exports = {
  mount: {
    public: {url: '/', static:true},
    src: {url: '/src'},
  },
};

it will pick up src folder.

[LarsDenBakker]

@jakesgordon in your example you don't have source maps enabled. Snowpack changes the all the paths of your code, serving them from a dist folder... so the test runner and the test coverage reporter is completely lost without source maps.

You can see some example snowpack projects here: https://github.com/modernweb-dev/example-projects

I do see issues here and there still, often because of missing source maps. Those needs to be fixed inside snowpack... I filed an issue for svelte: #2543

[jakesgordon]

Thanks @LarsDenBakker - I've tried it with and without enabling sourcemaps and see the same results.

In the simplified example I'm actually just using a pure javascript .js file so there is no transformation occurring. This is pure speculation - but I wonder if the code coverage is relying on the sourcemap to find the original source, but for pure .js files where no transformation occurs it looks like snowpack is not generating a sourcemap?

[LarsDenBakker]

Because snowpack rewrites the files to serve from dist always, we do indeed need the source maps to find where the files actually came from... but if you're not relying on any transformations you don't need to use the snowpack plugin so that should be fine.

[airimiea]

I have the same problem with snowpack, svelte and @web/test-runner 0.12.15.

[jbuckner]

I'm seeing this issue as well, using [email protected] and @web/[email protected]. We're using Typescript in our snowpack project.

Our snowpack mount config looks like this:

  mount: {
    public: { url: '/', static: true },
    src: { url: '/dist' },
    test: { url: '/test' },
  },

And I'm seeing an error like this:

Unable to lookup source: /path/to/project/dir/dist/app-root.js <-- dist doesn't exist

The dist directory that is getting an error doesn't actually exist on disk.. it looks like snowpack doesn't write the build directory out to disk during tests. Is there any way to configure istanbul (or @web/test-runner?) to pull from the urls that are being served by snowpack instead of trying to read it off disk?

[LarsDenBakker]

That's not possible because the test report is not served by the test runner. The issue here is twofold:

• Snowpack rewrites URLs to URLs that don't exist on disk
• Snowpack doesn't always include (correct) source maps that point to the original file on disk

We remap the urls for test coverage using these source maps, but if they're not correct there isn't anything we can do about it. Personally I'd prefer snowpack didn't rewrite the urls, it would make everything a lot simpler.

[jbuckner]

Interesting, thanks for the details. Can you think of any path forward to solve this? This must be a common problem.

Would it be possible to write the files out to disk either with snowpack build or just use the typescript compiler directly and not use the @snowpack/web-test-runner-plugin?