Enhance unified test coverage reporting

[Arthur-Prince]

Is your feature request related to a problem? Please describe.
I’m studying the Parsl code and noticed that there is no real make coverage target—the existing rule is currently:

.PHONY: coverage
coverage: ## show the coverage report
	# coverage report
	echo no-op coverage report

Describe the solution you'd like
A proper make coverage target that:

Discovers all config files in parsl/tests/configs/

Runs pytest --cov-append for each config

Describe alternatives you've considered
Writing external shell scripts to collect coverage and test lists

Additional context
I’d like to know if there is an existing alternative the team uses for this. If not, do you have any ideas on how you would implement such a feature? I would implement it if possible.

[benclifford]

That target is to generate a coverage report, not to orchestrate the whole test suite.

Prior to PR #2205 or so, the various pytest targets in Makefile ran with coverage enabled. However, that PR was a one shot removal of everything that was causing me trouble in CI at the time, and that included coverage. No one has cared about coverage since then to maintain it.

The right thing to do would probably be to remove the Makefile coverage target entirely - because as you point out, it isn't really useful without the other bits removed in #2205 - and we haven't got anywhere near enough test coverage to make any automatic coverage checking useful in CI.

So you could make a PR to remove it.

[Arthur-Prince]

At the suggestion of my advisor Alfredo and Daniel S. Katz, I’m exploring the possibility of integrating SonarQube into the Parsl CI pipeline. After running SonarQube locally, I identified several issues that I believe could have been avoided with more comprehensive test coverage. While investigating, I noticed there isn’t a dedicated command for collecting coverage across all tests so I decided to implement one myself.

Initially, I intended to implement coverage only in the make test target:

.PHONY: test ## run all tests with all config types
test: clean_coverage isort lint flake8 mypy local_thread_test htex_local_test htex_local_alternate_test config_local_test perf_test ## run all tests

I noticed that several test commands weren’t included in the make test target (Flux, WorkQueue, TaskVine, Radical), so I studied your GitHub Actions workflows and created Docker containers to simulate those environments and generate coverage reports.

I hit a blocker when I saw the Parsl+Slurm GitHub Actions workflow, where the test command wasn’t listed in the Makefile. Browsing parsl/tests/configs/ revealed many cloud and specialized configurations that aren’t referenced in the Makefile at all. I’d like to understand how you normally run those configurations in CI.

Actually, I already have a script that generates code coverage for all tests listed under make test.

.PHONY: test ## run all tests with all config types
test: clean_coverage isort lint flake8 mypy local_thread_test htex_local_test htex_local_alternate_test config_local_test perf_test ## run all tests

We would update the coverage target as follows:

.PHONY: coverage
coverage: ## show the coverage report
    ./local_test.sh

Or, if you prefer, I can provide a more elaborate approach that replaces each test command in the Makefile with something like:

.PHONY: wqex_local_test
wqex_local_test:
    ./test.sh workqueue

and uses environment variables to control coverage collection:

export PARSLCOV=1  # Enables --cov=parsl and --cov-report=xml:"cov/coverage.xml"
export PARSLCOV=2  # Enables --cov-append for merging unit + integration reports

I think the problem with issue #2205 was that the --cov-report= option was empty.

[benclifford]

Sorry I don't have much time to help you with this.

If you're adding something to CI, the main question I would ask is:

• what is the pass/fail question that is being asked?

This is so that the CI can fail when that question is violated.

One of the problems with having coverage checking as we had before was that there was no pass/fail question: a report was output in to the test logs that nobody read, without any pass/fail feedback.

If there is no path to feedback-via-failure on a particular pull request, adding more stuff into the CI makes it slower without adding value - a worse developer experience.