Possibility to merge results from several test binaries?

[nlohmann]

I have a test suite of multiple test binaries. The cmake files work so far, but they only seem to determine the coverage of one (first or last) test binary. Is there a way to merge the coverage results from multiple tests?

[JoakimSoderberg]

Hello,

I am a bit unsure what you mean exactly, is this a public project I can look at maybe?

But I mean you can have any number of test binaries, that is not what matters. You add each in CMake using the add_test cmake command.

https://github.com/JoakimSoderberg/coveralls-cmake-example/blob/master/CMakeLists.txt#L37

Then when the coverage is calculated, cmake will run each executable in turn and calculate the coverage and all that should be gathered in the output.

I mean this is how my Cmake script runs the actual tests:

coveralls-cmake/cmake/Coveralls.cmake

Line 68 in 3717770

COMMAND ${CMAKE_CTEST_COMMAND} --output-on-failure

Which is just a fancy way to run ctest --output-on-failure.

https://cmake.org/Wiki/CMake/Testing_With_CTest

CTest is a test runner which is independent from CMake itself... it will simply take a bunch of tests and run them. It does not care how many executables there are, it just has a list of tests

1. Clear previous coverage results
2. Ctests runs any number of executables (all executables must of course have been compiled with coverage)
3. Cmake script finds all gcov files outputted as tests ran (it does not care about test executables) and translates them to JSON and uploads to coveralls

What step 3 above cares about is not the executables, but the source files you want the coverage from (you specify these as per the instructions for this project in your CMake file).

Hope all that made sense :)

---

P.S.

I'd also recommend looking into https://codecov.io/ it's also free. And from my experience more stable + has more features. And is simpler to use :)

[JoakimSoderberg]

https://github.com/JoakimSoderberg/coveralls-cmake/blob/3717770200eca844a19df8fc8001a087e17a0416/cmake/CoverallsGenerateGcov.cmake

Oh yea, this does the meat of the collection business. As you see there, all it does is look for gcov files, nothing related to specific test executables.

[nlohmann]

Hi there, thanks for the quick response!

The project is nlohmann/json, and I added a feature branch (see https://github.com/nlohmann/json/tree/feature/cmake_cleanup) to move the build system to Cmake. See https://travis-ci.org/nlohmann/json/jobs/186321152 for the Travis job.

It uses the same C++ sources than the 100% covered develop-branch (see https://coveralls.io/builds/9376166), but instead of having one large binary, the test suite is split into several binaries. All are added via add_test. The CMakeLists are here and here.

If you need any further information, please let me know. I shall try not to run any tests before executing make coveralls.

[JoakimSoderberg]

You forgot the important part. The coveralls that does not have 100% coverage :)
https://coveralls.io/builds/9409833

Because what is interesting is the build output:
https://travis-ci.org/nlohmann/json/jobs/186347999#L630

Anyway if you look in that output you see that GCOV is rewriting teh .gcov file for each test:

File '/home/travis/build/nlohmann/json/src/json.hpp'
Lines executed:63.18% of 402
Creating '#home#travis#build#nlohmann#json#src#json.hpp.gcov'

File '/home/travis/build/nlohmann/json/src/json.hpp'
Lines executed:56.79% of 1539
Creating '#home#travis#build#nlohmann#json#src#json.hpp.gcov'

File '/home/travis/build/nlohmann/json/src/json.hpp'
Lines executed:82.56% of 1055
Creating '#home#travis#build#nlohmann#json#src#json.hpp.gcov'

File '/home/travis/build/nlohmann/json/src/json.hpp'
Lines executed:80.10% of 603
Creating '#home#travis#build#nlohmann#json#src#json.hpp.gcov'

File '/home/travis/build/nlohmann/json/src/json.hpp'
Lines executed:64.51% of 417
Creating '#home#travis#build#nlohmann#json#src#json.hpp.gcov'

...

File '/home/travis/build/nlohmann/json/src/json.hpp'
Lines executed:83.45% of 1432
Creating '#home#travis#build#nlohmann#json#src#json.hpp.gcov'

Your last test happens to cover 83.45% of the file, and that's what ends up on coveralls.

I cannot say I have noticed this behavior myself, but maybe I never ran this with multiple test EXEs come to think of it. Or... your gcov version is old and does

So just quickly googling I found this thread from 2007 about fixing this behavior.
https://gcc.gnu.org/ml/gcc-patches/2007-06/msg00423.html

this patch adds a new option to gcov: --merge-results (-m).
When this option is set, gcov reads all input files before writing results instead of writing results after each input file.
Therefore, if two .gcda reference a common source file com.c, only one com.c.gcov file is generated with counts added rather than two
com.c.gcov, the second one overwriting the first one.

The -m option is removed in a later version of the patch:

this is the updated version of the patch. No new option, results are merged as soon as there are multiple input files.

So this might depend on the gcov version. Seems this is not in versione < 4.3
...

I can't say I know the reason for this. If it's at the .gcda stage or the .gcov stage the files are not merged.

Codecov also supports pushing multiple reports for a single commit. Even from different OS versions and so on, as I said it has more features and is simpler to use. So look at that as an alternative.

Here is how easy it is to include. Since my cmake script has compiled + generated all coverage files it simply finds those and does a bunch of magic. So try that and see if it solves your issues:
https://github.com/JoakimSoderberg/cargo/blob/master/.travis.yml#L41

I am not really enough invested in this script to hunt down this bug. But if you can fix this a Pull Request is always welcome :)

[nlohmann]

Thanks for the response!