Add a JUnit output formatter compatible with Jenkins

[calj]

• Added the parameter --junit to bats which output into JUnit XML format, the purpose was to integrate the report into Jenkins
• I had to add the suite name into the extended report format so I can classify tests by file
• I added --extended parameter to bats command so you can run the test suite once and format the output into several format (ie: pretty for the console and junit into a result.xml file)

Please advise if you see anything to change!

[schrepfler]

This is awesome!

[harschware]

As noted by the Travis build, the tests fail with this PR.

I was interested in merging this PR into my harschware/cactus branch (which is where I keep a merge of all the PRs I'm interested in, since PRs are not being merged into the sstephenson/bats repo for sometime now).

The main thing I'm interested in from this PR was the timing information, since I see you added it to the TAP test status lines, for extended syntax. But after looking at the code I don't think it work well as a starting point for my purpose. I am looking to test duration in millseconds to the TAP output via YAML extension.

Be aware that the TAP13 specification does not provide for a way to indicate time, but the issue of supporting that in the spec has been discussed. No TAP14 spec exists today, but it essentially comes down to two methods being discussed.
add # time=NNN to the test line
add YAML to the test. e.g.

1..1
ok YAML time with duration
  ---
    duration_ms: 45293
  ...

The YAML extension is supported by Jenkins TAP Plugin (https://wiki.jenkins-ci.org/display/JENKINS/TAP+Plugin), incidentally JUnit isn't strictly required to use BATS in Jenkins if you use the TAP plugin instead of JUnit.

Not all TAP consumers respect the YAML extension, but at minimum tap4j does.

I will probably want to at minimum, ammend the extended format in BATS to include time in milliseconds rather than seconds. When it comes time to merge our two PRs to this repo (if ever) there will be a slight divergence there.

[harschware]

See PR #177 for details about TAP with YAML extension.

[calj]

Here is the fix for tests.

Your work looks great, I didn't found any way to calculate the time in milliseconds in pure "bash", I used this framework because it's highly portable, for example the usage of the command date is different on AIX and on Solaris 10, so I simplified the problem by displaying the time in seconds.

Suggestions are welcome :)

[astorije]

What's the status of this?

FYI, Solano CI parser does not support TAP, sadly, so having a JUnit formatter is the only alternative.

@sstephenson or @mislav, is this project still being looked at? Thanks for your great work!

[astorije]

I think it would be more explicit and future-proof to name them -x, --extended-tap and -j, --junit .

[schrepfler]

Better to say Jenkins or Jenkins/Hudson here as most people use Jenkins. Additionally it's a personal name so caps are grammatically correct.

[astorije]

I'd remove what's after the comma entirely. Whether it's for Hudson, Jenkins or whatever, it's a JUnit format, and it's all that matters :)

[schrepfler]

True.

[calj]

Right, we can do so.
I have specified Hudson here because I noticed difference with the JUnit specifications.

[nvgoldin]

Any news on this? Would love to see this gets in.

[matt-land]

Would also like this in. PR Contributor did the work and fixed the tests.

[jasonkarns]

I rather feel that supporting multiple output formats is outside the role of bats.

TAP output is (or is becoming?) a near lingua-franca of test report formats. TAP can be consumed and converted to virtually any other format (dot, spec, xunit, etc)

If all that is desired is TAP consumption by jenkins/hudson, I would advocate for the TAP plugin: https://wiki.jenkins.io/display/JENKINS/TAP+Plugin

Separately, though, I would advocate for any other test report formats to consume the TAP output on STDIN, and transform to the desired output by piping. For example: https://github.com/aghassemi/tap-xunit (which could be used as-is to convert Bats' TAP output to xunit without needing the option supported in bats.)

As it stands, the current formatter could be shipped as a standalone TAP converter that just receives Bats' TAP output. Creating a separate bats-xunit project would be a fine repo for future inclusion in a potential bats-core github org. I just don't think it should necessarily be bundled within bats itself.