Support SourceMaps

[CraigCav]

Need to figure out how to get traceurs generated sourcemaps fed back into mimosa.

mimosa-traceur may have to become a fully fledged compiler, rather than an afterCompile step, in order to achieve this, although this may have other implications; The default extension js will likely conflict with the copy step and i'd rather not have to use a different extension if I can avoid it.

[dbashford]

I hope to deconflict the copy extensions from the other compilers, letting the others win. In the short term you could detect js in the copy.extensions and either a) remove it or b) logger.warn it. You'd do this in the validate method.

[CraigCav]

Sounds workable.

I was also having a few other issues when trying to switch mimosa-traceur to being a compiler. I used this file from mimosa-coffee as the basis for structuring the module, but the compiler wasn't being picked up at all. Apart from adding traceur to my mimosa modules, do I have to do anything else in my mimosa.config to let mimosa know that mimosa-traceur should be used to compile? I'm currently assuming that simply adding the name, compilerType, compile and extensions properties to the exported module is enough...

I'll push up a branch with my attempt so far, I would love it if you wouldn't mind taking a look to see if i'm missing something obvious.

[CraigCav]

I've just push up a branch with my failing attempt to convert to a compiler. Here's the branch.

[CraigCav]

I've also pushed a semi-working prototype of sourcemaps (without converting to a compiler) here. I've resorted to copying a bunch of mimosa's built-in javascript compiler though, so I'd definitely prefer a compiler approach. Currently, this branch just generates sourcemaps with a data-uri.

I haven't figured out a way to make this play nicely with the minify and optimize steps yet either, so I need to give that some attention too.

[dbashford]

What you have is working for me. I tossed a console.log('COMPILING!!') inside the compile function, mimosa mod:installed your branch, created a mimosa new project with plain js, added traceur to the modules list (maybe you missed this step?) and when I ran, this happened:

COMPILING!!
08:13:19 - Success - Wrote file public/javascripts/main.js
COMPILING!!
08:13:19 - Success - Wrote file public/javascripts/app/example-view.js
COMPILING!!
COMPILING!!
08:13:19 - Success - Wrote file public/javascripts/vendor/jquery/jquery.js
08:13:19 - Success - Wrote file public/javascripts/vendor/requirejs/require.js

I just wanted to make sure it was being called, I didn't fiddle with the copy extensions.

One thing you may want to do in your README is mention that for this to work properly it is best that js be removed from copy.extensions. I think you are pretty much done. Though I admit to not actually flexing traceur.

One thing I'd mention, the modules don't prefix the config block with comments anymore. So this can have the leading # removed. Probably also want to indent the settings in from traceur a bit.

[CraigCav]

You know what, you're absolutely right, the compiler is working. I had an old version of mimosa installed on my machine - when I updated to the latest, the compiler is working just fine.

I'll update master so it has the latest version from the compiler branch.

The traceur parts seem to be working just fine too, albeit with traceur's default configuration. In time (and tracked under a separate issue), i'll probably want to surface other configuration options for experimental features and toggling language features.

The piece to the puzzle that is missing is sourcemaps. At the moment I see two issues:

Currently the plugin will generate sourcemaps, however:

1. When running mimosa watch -s the "map" is surfacing in devtools under the same name as the source file (as they have the same extension) and I think this means that the source is taking precedence over the map.
2. When running mimosa watch -som I think the source map is effectively ignored and only the source map for the optimized build is considered (debugging shows the traceur compiled result, rather than the ES6 based source). i.e. multi-level source maps aren't handled.

Are sourcemaps supported when running coffescript with minified/combined flags? If so, did you have to specify anything special to enable multi-level sourcemaps?

[dbashford]

Finally got back to working on trying to have multiple compilers co-exist.

I'm not sure I'm going to be able to get what we need working -- to allow a traceur compiler and the copy compiler both doing the right things at the right time -- without some small hacks.

The traceur compiler would need a way to opt out of "claiming" certain files. Non-copy compilers trump the copy compiler. So if a JavaScript compiler thinks it should be processing a file, it'll process the file and the copy compiler will leave it alone.

An options object travels to each workflow step and contains details about the file being processed. The copy compiler recognizes that another compiler has laid claim to a file by checking the options object for a destinationFile. If it sees that that property exists it will not claim the file for itself.

The copy compiler also defers to any other compiler by not doing anything if the file already has outputFileText, which means it has been compiled by something else. The copy compiler will NOT overwrite another compiler's work.

So the copy compiler is essentially the bitch. All of this is as of 2.2, currently published as mimosa-canary.

So what would happen by default, given the changes I've made for 2.2, is the copy compiler will see that the traceur compiler "claimed" all JavaScript files and then ignore them. I assume the traceur compiler won't monkey with a JavaScript file that doesn't actually have anything to compile, or will it still rewrite it to something else? If it recognizes that a file doesn't need to be traceur-ed and leaves it alone, well, that's almost as good as copying (assuming you populate outputFileText regardless) and we'd be all good.

But what if traceur jacks up plain .js files, like, for instance, backbone.js? Altering that code isn't terribly acceptable, so we'd need some way to have traceur ignore it and let copy do its thing.

Mimosa would have no way, given they have the same extension, to determine the difference between es6 JS and non-es6 JS. So your compiler would need to take some steps. It would need an exclude property... some way for a user to say "please don't traceur-ize these files". If one of those files comes into your compiler, you'd need to "unclaim" it. Essentially that means registering for the init workflow step and setting options.destinationFile to undefined if a file is excluded.

You'd also need to just return next() if the file has been excluded in your compile function too.

If you unclaim (options.destinationFile = undefined) and don't set outputFileText, the copy compiler will think that it is ok to process the file itself.

I hope that all makes sense. Mimosa is heavily extension based, so its been some work to allow the same extension to do multiple things at different times.

[CraigCav]

@dbashford Thanks again for all your help!

But what if traceur jacks up plain .js files, like, for instance, backbone.js

Traceur doesn't have any smarts for detecting if a file is ES6 (as far as I can tell), so as mimosa-traceur stands right now, vendor libs like backbone would also be processed by the compiler. Although this shouldn't have side effects, I can definitely appreciate that this is undesirable.

I'll add a new issue for supporting an exclude property and I'll add the work-around you've mentioned for unclaiming any excluded files.

I'll also give mimosa-traceur a spin through mimosa-canary and see what sticks - watch this space!

[CraigCav]

I guess we must be getting closer...

When running mimosa watch -s I can run the app and debug with the compiled source.

When looking at the sources tab in chrome I see:

app/
  viewmodels/
    C:\projects\mimosa-traceur-demo\assets\app\viewmodels\shell.js
    shell.js
  main.js
content/
  styles.css
socket.io
  socket.io.js

shell.js is the compiled source (served from C:\projects\mimosa-traceur-demo\public\app\viewmodels\shell.js).

The second file C:\projects\mimosa-traceur-demo\assets\app\viewmodels\shell.js is the actual source ES6 served by the map (served from C:\projects\mimosa-traceur-demo\public\app\viewmodels\shell.js.map).

I can put a breakpoint in the ES6 source and the debugger hits at the appropriate point!

Apart from the full file path in the map, this is pretty much as close as I'd expect to get. I'm fairly sure the full file path is just read from the sources array inside the map file and I believe I could have this output something like shell.js.map rather than the full file path. Unfortunately, I don't think I can change this to just shell.js (thereby only showing the ES6 source) as I think complied source would take precedence instead of the ES6 source.

The story isn't quite as smooth when throwing the --optimize and --minify flags into the mix.

The first thing I notice when running mimosa watch -som is I get the following warning from the mimosa console output:

WARN: Couldn't figure out mapping for C:\projects\mimosa-traceur-demo\public\app\viewmodels\shell.js:12,4 → 1,862 []

When looking at the sources tab in chrome I see:

app/
  main-built.js
  reload-client.js
content/
  styles.css
socket.io
  socket.io.js

if I open up main-built.js, I can see there is a sourcemap url present:

define('viewmodels/shell',['require','knockout' ...etc
/*
//@ sourceMappingURL=shell.js.map
*/

but since the path is wrong (presumably it should be viewmodels/shell.js.map), the file isn't served and therefore I can't debug.

When running mimosa watch -so isn't any better. The build time was significantly slower too. I'm guessing that this is because the optimizer is performing the minification rather than mimosa.

When looking at the sources tab in chrome I see:

app/
  viewmodels/
    shell.js
  main-built.js
  reload-client.js
content/
  styles.css
socket.io
  socket.io.js
text!views
  shell.html

In this case, app/viewmodels/shell.js is the compiled source, rather than the original ES6 source.
The map file is in the public/viewmodels directory and the sourcesContent seems correct. The main-built.js.map doesn't quite seem right though; the sourcesContent contains the compiled source rather than the ES6 source.

Lastly, running mimosa watch -sm. This doesn't pan out either. I again get the "Couldn't figure out mapping for" warning that I had when running mimosa watch -som.

When looking at the sources tab in chrome I see:

app/
  viewmodels/
    \app\viewmodels\
      C:\projects\mimosa-traceur-demo\assets\app\viewmodels\shell.js
    shell.js
  main.js
  reload-client.js
content/
  styles.css
socket.io
  socket.io.js

shell.js contains minified source and a sourcemappingurl:

/*
//@ sourceMappingURL=shell.js.map
*/

The other file in the nested \app\viewmodels\ dir is does not match any static file (404).