feature request: hybrid Foundry/Hardhat coverage

[wschwab]

solidity-coverage is currently a framework agnostic project. While Foundry is working on its own coverage tool, I think that there is value in extending solidity-coverage's support to Foundry too. Once specific use case follows.

Certain things are easier to test from JS/TS as an EOA. Some are easier to test as a smart contract (on-chain). Hardhat provides the former, Foundry provides the latter (and easy access from within tests to fuzz and invariant tests). Some repos have decided on a hybrid environment, bringing in features from both. Seaport is a prominent example.

Getting coverage results that include the coverage of both the JS/TS and Solidity tests would be where solidity-coverage could shine. Once the tool would be able to monitor coverage from Foundry, it would seem to me that it wouldn't be too hard for it to monitor overall coverage in a hybrid repo.

If there's anything I can do to add context and/or information, please let me know.

Am I looking at this right?

[cgewecke]

@wschwab

Once the tool would be able to monitor coverage from Foundry

I don't think this is likely to happen (or desirable). solidity-coverage is tightly coupled to the EthereumJS VM and works best with JS toolchains.

Maybe hybrid repo users could just merge saved reports from different test runs using a tool like istanbul-combine-updated. You can select what kind of report to produce in solidity-coverage using the istanbulReporter option (IIRC, Foundry generates a standard LCOV artifact).

If you're using Codecov as a CI service, multiple reports uploaded at once will be auto-merged on their side.

Does this approach seem reasonable for your use case?

[wschwab]

Thanks for the detailed answer @cgewecke - we're looking into implementing a solution using your answer. If you don't mind, I'll keep this open in the meantime just in case we run into trouble.

[yorhodes]

https://github.com/hyperlane-xyz/hyperlane-monorepo/blob/main/solidity/coverage.sh we merge the two reports here

[yorhodes]

I spent ages fiddling with this but hands down codecov is the best solution by far
hyperlane-xyz/hyperlane-monorepo#2449
free for open source!