Consider codecov.io for coverage evaluation

[fedorov]

There are examples that show how to use it with C++ and with cmake:

https://docs.codecov.io/docs/supported-languages

[jcfr]

We have been using it in most of our python projects.

It is also now used to report ITK coverage (cc: @thewtex). Initially it was not working, too much date for their infrastructure. They were very responsive in improving their infrastructure so that it can process all ITK coverage files.

[thewtex]

FYI, here are the scripts we use for ITK:

• https://gist.github.com/e3f59fdfba4f30ad2580dbb4db7a22e9
• https://gist.github.com/1f410c90e8b9e9826989c0e04e97738b
• https://gist.github.com/cc7ade395c7aa309f38b137b8934e075

[fedorov]

@jcfr @thewtex thanks a lot for following up on this topic! Your suggestions should be very helpful when we get to this.

[fedorov]

@jcfr @thewtex

I was able to do some coverage testing by following the pointers you gave me. Here's what I did:

1. sudo apt-get install lcov on my local Ubuntu
2. set manually CMAKE_C_FLAGS and CMAKE_CXX_FLAGS to -g -O0 -fprofile-arcs -ftest-coverage, set CTEST_COVERAGE_COMMAND="/usr/bin/lcov".
3. configured and super-built dcmqi
4. discovered that flags were not propagated to the inner build (@jcfr - is this expected or a bug?), re-ran cmake with the flags above in the inner build, rebuilt
5. ran ctest
6. captured coverage info: lcov --directory . --capture --output-file coverage.info
7. manually uploaded coverage report to codecov

The result is here: https://codecov.io/gh/QIICR/dcmqi

Do you have any idea why the only files shown in the report are the CLP headers and itkImageIOFactoryRegisterManager.h?

I also realized I did not set LDFLAGS env - not sure if it changes anything, but will test that later.

[fedorov]

Coveralls is another option: https://coveralls.zendesk.com/hc/en-us/articles/201342799-C-C-

[fedorov]

I made some progress.

Relevant cmake variables (make sure these are in the inner dcmqi-build folder):

CMAKE_CXX_FLAGS:STRING=-g -O0 -fprofile-arcs -ftest-coverage
CMAKE_C_FLAGS:STRING=-g -O0 -fprofile-arcs -ftest-coverage
CMAKE_EXE_LINKER_FLAGS:STRING=-fprofile-arcs -ftest-coverage
CMAKE_MODULE_LINKER_FLAGS:STRING=-fprofile-arcs -ftest-coverage
CMAKE_SHARED_LINKER_FLAGS:STRING=-fprofile-arcs -ftest-coverage

After running ctest ., run lcov in the inner build folder:

$ lcov --directory . --capture --output-file o.lcov

Clean up the output:

$ lcov --remove o.lcov '/usr/*' 'dcmqi-build/*' --output-file o_clean.lcov

Generate html view:

$ genhtml -o o_clean.html o_clean.lcov 

Result in o_clean.html/index.html:

The bad news that I still cannot make sense out of the Codecov reported coverage. Without changing anything (presumably, same gcov reports should be used), and running

$ bash <(curl -s https://codecov.io/bash) -X gcov -B master -C <token>

I am getting the same result in Codecov, which makes no sense to me, and is not consistent with what I get with genhtml:

[fedorov]

Coveralls is another option: https://coveralls.zendesk.com/hc/en-us/articles/201342799-C-C-

cpp-coveralls appears to be unable to handle gcov output generated for our code. Issue reported: eddyxu/cpp-coveralls#116

[fedorov]

Making some progress understanding how this is supposed to work ...