Implement externalized WASM build

[mochaaP]

An alternative to #97.
See #96 for more info.

[guybedford]

I'm still curious how the fs.readFile is supposed to work at runtime in a distribution? Shouldn't all of this be happening at a binary level somehow?

[khardix]

@mochaaP I tried to grok the code and perhaps come with an explanation myself, but to no avail. Adding some docs (README? Comment block in code?) would probably help any future maintainer to understand what we are trying to archive here and why it's done in this specific way.

@guybedford Aside – thanks for the patience with this change, I really appreciate it!

[mochaaP]

Added comments to explain the load process. If something is still unclear, please point it out :)

[guybedford]

@mochaaP thanks for adding comments, my question though specifically is how you are hooking the readFile call in the Node implementation to support inlining the binary in the runtime?

[mochaaP]

To clear some confusion: fs.readFile is not hooked -- instead there are two flavors: CJS and ESM.

The CJS flavor inlines the WASM binary as a base64 string. This is what libnode bundles, and it won't be published to npm (or not exported by package.json).

The ESM flavor assumes this module is properly installed as a package, and reads the WASM binary from a relative path.

(Sorry for the late reply. I was working on other projects previously.)

[guybedford]

The ESM flavor assumes this module is properly installed as a package, and reads the WASM binary from a relative path.

This is the part I still do not understand properly - you mean it is loaded relative to the Node.js binary itself? Why would we want to ship a Node.js binary that loads another separate Wasm file at runtime?

[mochaaP]

Normal Node.js distribution will only inline the CJS flavor, since it's not installed as a module. (i.e. have full filesystem structure, package.json, etc.) Builtins also can only load as CJS iirc, since the initialization process isn't asynchronous.

If one opts in to use the ESM flavor, it means the module has a place on the filesystem, thus we are able to resolve a WASM file relative to lexer.mjs.