emcc --clear-cache: "Emscripten setup is not complete"

[juj]

I wonder if it might make sense to skip the bootstrap check on some commands, e.g.

C:\emsdk\emscripten\main>emcc --clear-cache
emcc: error: emscripten setup is not complete ("npm packages" is out-of-date). Run `bootstrap` to update

?

[sbc100]

Agreed, if its a easy change that would make sense.

On the other hand I expect most of our users are not running bootstrap themselves since they are using a pre-packaged version of emscripten (bootstrap is really only something that emscripten dev / git users should need to run).

[juj]

Actually, thinking even further, given that we are able to detect when it is needed to run bootstrap, I wonder if we should run bootstrap automatically when we are about to touch any npm action, if frozen cache is not in effect? Like, why do we ask the user to run bootstrap, and don't just run bootstrap automatically on the fly?

[sbc100]

I'm not a fan of the compiler magically running stuff like this on any invocation. For example, I think its a mistake that we allow the automatic build of library at link time. I think the compiler commands should be a simple as possible, and running npm and git commands behind the scenes seems like a bad idea. I can elaborate more on the reasons why I think this.. but I'm actually heading out for the day right now.

[brendandahl]

I also would prefer it to autorun. When I jump around branches, I run into bootstrap complaining. Maybe we could make the auto bootstrap a .emscripten setting?

[sbc100]

That sounds like a great reason to add a git checkout hook, but not something we should build into emcc itself.

I don't think its reasonable to expect a complex tool like emscripten to just work on checkout without an kind of post-checkout/build step at all.

If we go down this path why not also have emcc download binaryen and llvm on demand. There are several reasons I don't like this kind of idea:

1. Too much magic. A compiler should be a compiler, it should not need to go online and download stuff when you run it.
2. We have users that to run stricly offline. They have been asking for a way force the compiler never to go out to the network
3. It can be very confusing for tools that like to run, for example, emcc --version if the output also contains all kinds of other stuff, but only in some cases.

Anyway, a git checkout hooks it probably something we could have. I personally don't mind running in manually once in while. @brendandahl, how do node projects generally deal with this problem? Are they running npm install on every checkout, or automatically in some way, or are developers normally expected to run npm install themselves?

[juj]

I totally agree with the too much magic argument.

One of the pain points with the bootstrap.py error is that it complains about false positives, e.g. when I check out to another branch and then back. I wonder if we could demote it to a yellow warning maybe. So many times when I git checkout between branches my work gets stopped by needing to rerun bootstrap yet again, even though nothing relevant (npm, entry points, posixtestsuite) has really changed. I just keep being stopped to bootstrap.

Maybe missing bootstrap could be demoted to a yellow warning?

Sidenote, I wonder maybe if missing the posixtestsuite could be an error print when attempting to run test/runner posixtest and the submodule is not present. 99% of Emscripten users will never run the posixtest suite, so that submodule will not ever be relevant to be checked out for them.

[sbc100]

Regarding the need to run bootstrap between when switch branches. The issue is that the checks are based on file timestamps vs file contents, so if you check between branches A and B and then a have different version of package.json then it will ask you to re-run the bootstrap script. As the person who probably has the most emscripten branch around (I normally have around 50 active branches in my current checkout) I can say that in my experience is that since these files tend to change rarely its not very often I'm asked to re-run the boostrap script. However, I also like to rebase all of my branches all of the time so I normally only have to run the script once then I rebase my branch on top of change to package.json (for example). If you have branches the often stradle or include include changes to package.json then I think the git checkout hook is probably the way to go. The alternative would be to switch from a timesptamp trigger to content-hash trigger, and based on my experience with scons (content-hash-based) vs make (timespamp based) over the years I think that wouldn't be great idea.

Regarding the posixtestsuite question, this submodule basically never changes so it should (almost) never be a reason to run the bootstrap script. So its a purely one-off cost, that is only payed by emscripten developers (or users how prefer to use the git checkout of emscripten). Users of emsdk or other bundled version of emscripten should not be effected by this.

I think a reasonable path forward is to recommend a simple checkout hook for folks that don't like their flow interrupted when they checkout a new version of package.json.

[juj]

As the person who probably has the most emscripten branch around (I normally have around 50 active branches in my current checkout) I can say that in my experience is that since these files tend to change rarely its not very often I'm asked to re-run the boostrap script.

This is not my experience. I get the emscripten setup is not complete every time whenever I change branches, and in my day-to-day use when contributing to Emscripten I need to keep running bootstrap.py several times a day. Surely something must be amiss here then?

[juj]

e.g. today with

this is due to the package json files changing.

[sbc100]

I think its pretty normal for projects (e.g. node projects) to require a step such an npm install to run when you switch between branches and (for example) the package.json has changes. i.e. i don't think emscripten is unusual in this regard. And of course projects with compiles source code have even heavier weight build steps.

So I don't think is reasonable to require that emscripten just work out of the box with zero build or configuration steps when switching branched. However, I think a post checkout git hook that runs these steps is a great way to make it feel like that.

I'm going to look into that now.