v0.8.4

What's Changed

• Prepare for next hardhat version by @fvictorio in #796
• Update solidity-parser to 0.16.0 by @cgewecke in #802

New Contributors

• @fvictorio made their first contribution in #796

Full Changelog: v0.8.2...v0.8.4

0.8.2

What's Changed

• Set web3-utils dep to ^1.3.6 by @markuschick in #744

New Contributors

• @markuschick made their first contribution in #744

Full Changelog: v0.8.1...v0.8.2

0.8.1

What's Changed

• Bug fix: restore missing web3-utils dependency by @cgewecke in #743

Full Changelog: v0.8.0...v0.8.1

0.8.0

Hi!

⚠️ This version requires Hardhat >= 2.11.0 (Ethereum Merge)

New Features

A central focus of the 0.8.0 release is improving the coverage tool's branch detection.

Beginning with this version the following syntax is measured as a branch:

OR conditions

When a logical expression is composed with the || operator, both sides can be considered branches. To test the entire expression

if (a == 1 || a == 2)

... a must equal 1, 2 and neither of those values. (Thanks to Gnosis engineer @rmeissner for proposing this in #175)

Ternary Conditionals

Long ago, when Solidity was 0.4, solidity-coverage treated ternary conditionals like regular if/else statements. Some language improvements v0.5 subsequently made this impossible. Now it's back...

Modifier Invocations

Solidity-coverage already covers the code within modifier definitions. However, each modifier invocation at the function level should really be considered its own branch. Some of the most critical logic in Solidity contracts is handled this way (ex: onlyOwner). Testing the pass/fail cases for each occurrence of these gates protects you from accidentally removing them during a refactor.

Because it's possible to write a modifier which performs a preparatory task and never reverts, there's a new option (modifierWhitelist) which allows you to exclude specific modifiers from branch measurement.

And if you don't like modifier invocation coverage you can turn it off by setting the option measureModifierCoverage to false.

(Many thanks to OpenZeppelin engineer @nventuro for proposing this improvement in #286 and helping to design it.)

Test Matrix

The hardhat and truffle plugins support a new cli flag: --matrix. (Short for "test matrix".)

This flag generates a JSON object that maps which tests in your suite hit which lines of code. (An example can be seen at docs/matrix.md. More info can be found in the advanced docs, here.)

This data is useful for many advanced testing applications - Security researcher @JoranHonig has written two that are worth checking out.

• vertigo: a mutation testing framework
• tarantula: a fault localization tool

Coverage is often a core component of fuzzing and generative test strategies because it helps narrow the range of inputs required to traverse every path in the code. If you're working on applications like this and have ideas for how solidity-coverage might serve your ends, please feel free to open an issue.

Note to Truffle users

solidity-coverage for Truffle is moving to its own repository and published under a different name (details soon...).

0.8.0 Beta

Hi!

This beta contains several new features. A central focus of the next major release is improving the coverage tool's branch detection.

Beginning with this version the following syntax is measured as a branch:

OR conditions

When a logical expression is composed with the || operator, both sides can be considered branches. To test the entire expression

if (a == 1 || a == 2)

... a must equal 1, 2 and neither of those values. (Thanks to Gnosis engineer @rmeissner for proposing this in #175)

Ternary Conditionals

Long ago, when Solidity was 0.4, solidity-coverage treated ternary conditionals like regular if/else statements. Some language improvements v0.5 subsequently made this impossible. Now it's back...

Modifier Invocations

Solidity-coverage already covers the code within modifier definitions. However, each modifier invocation at the function level should really be considered its own branch. Some of the most critical logic in Solidity contracts is handled this way (ex: onlyOwner and nonReentrant). Testing the pass/fail cases for each occurrence of these gates protects you from accidentally removing them during a refactor or emergency patch.

Because it's possible to write a modifier which performs a preparatory task and never reverts, there's a new option (modifierWhitelist) which allows you to exclude specific modifiers from branch measurement.

And if you don't like modifier invocation coverage you can turn it off by setting the option measureModifierCoverage to false.

(Many thanks to OpenZeppelin engineer @nventuro for proposing this improvement in #286 and helping to design it.)

Test Matrix

The hardhat and truffle plugins support a new cli flag: --matrix. (Short for "test matrix".)

This flag generates a JSON object that maps which tests in your suite hit which lines of code. (An example can be seen at docs/matrix.md. More info can be found in the advanced docs, here.)

This data is useful for many advanced testing applications - Consensys security researcher @JoranHonig has written two that are worth checking out.

• vertigo: a mutation testing framework
• tarantula: a fault localization tool

Coverage is often a core component of fuzzing and generative test strategies because it helps narrow the range of inputs required to traverse every path in the code. If you're working on applications like this and have ideas for how solidity-coverage might serve your ends, please feel free to open an issue.

What's Coming!

We're hoping to get a major release out by the beginning of February (2021). It will include two additional features you can run in CI to get better feedback about what's changing in your code from PR to PR.

• A unified diff of the public API changes in your contracts
• Natspec documentation coverage

0.7.0

Hello!

Solidity-Coverage has undergone a major re-design in 0.7.0.

Its core remains the elegant solidity instrumentation logic which JoinColony engineer @area first worked out in 2016. But the mechanics of collecting your tests' execution trace and the tool's integration with the Ethereum development stack have been re-written to fix a host of problems that have built up over the last two years. Beginning with this release, you'll

• be able to use any Ganache version you want
• slot coverage into your other tooling as a plugin with much less configuration.
• run into far fewer cases where normally working tests fail when being covered
• be able to build your own coverage application using the tool's API.

We've also focused on improving error reporting and making overall performance smoother. It's been beta tested for several months on many of Ethereum's largest projects and everything looks good :)

Documentation

• Installation for Buidler and Truffle
• Options and other instructions
• A guide to upgrading from 0.6.x to 0.7.x
• Some notes on advanced use cases

🐘 What's coming... 🐘

Another major phase of development is scheduled for the next couple of months. We'll be implementing a set of new, solidity-specific coverage measurements which engineers have proposed over the years to increase the precision of the reports you get:

• Per-function modifier coverage , which treats modifiers as branches in the code for each function, e.g for each method qualified by the modifier Ownable, measure whether it was called by owners and non-owners (Proposed by Zeppelin engineer Nicolas Venturo in solidity-coverage #286)
• Public variable coverage, which treats public variables as methods (since getters are implicitly generated for them) and provide a way for developers to track changes in their contracts' public API. (Proposed by Zeppelin engineer Nicolas Venturo in solidity-coverage #281)
• && / || conditional coverage: , which treats conjoined conditions as distinct execution branches. This precision in branch coverage reporting is available for Java with the cobratura coverage tool - Solidity could benefit from it too. (Proposed by Gnosis engineer Richard Meissner in solidity-coverage #175)
• Documentation coverage, which measures Natspec comment-to-parameter mappings and provides a way to track documentation completeness
• CI integrations for the above Where additional coverage measurements can't be adequately represented by Istanbul/Coveralls we'll add support for them with a githooks/CI integration that tracks pull-request changes relative to their target branch.

Thanks for trying this out!

Please open issues/ask for help if you run into trouble installing or see problems ;)

Buidler Plugin & Coverage API

Hi!

Phase II of solidity-coverage's redesign is complete with this release. A long-standing goal here has been to make the tool more 'library-like' and decouple from any single development platform. beta.3 moves in that direction by:

• Publishing an API
• Using the API to create a coverage plugin for the Buidler development platform.

Buidler Plugin 👷‍♀

Buidler is a beautifully written, ultra-fast Ethereum task/test runner. There's also some really interesting work going on over there to develop an alternative JS Ethereum client with Solidity stack traces which is well worth checking out.

To use solidity-coverage with Buidler, install as usual and list it as a plugin in your buidler.config.js

usePlugin('solidity-coverage')

module.exports = {
  networks: {...},
}

Run the 'coverage' task:

npx buidler coverage

Coverage API 🐘

If you're interested in writing your own coverage plugin or have a custom testing setup, you might find the new API useful. You can now access most of this repo's internals by writing:

const CoverageAPI = require('solidity-coverage/api');
const utils = require('solidity-coverage/utils');

Full API documentation is available here

Bug-fixes

• Fix missing coverage when truffle-config.js used a V4 style solc configuration (#434)
• Fix crash when test directory contains an empty folder (#432)
• Fix coverage hanging when used in combination with eth-gas-reporter (#423)

0.7.0-beta

Hello!

Solidity-Coverage is undergoing a major re-design in 0.7.0....

Its core remains the elegant solidity instrumentation logic which JoinColony engineer @area first worked out in 2016. But the mechanics of collecting your tests' execution trace and the tool's integration with the Ethereum development stack have been re-written to fix a host of problems that have built up over the last two years. Beginning with this release, you'll

• be able to use any Ganache version you want
• slot coverage into your other tooling as a plugin with much less configuration
• run into far fewer cases where normally working tests fail when being covered

We've also focused on improving error reporting and making overall performance smoother. It's been beta tested on a handful of large projects and everything looks good - worth checking out :)

What to do

Install

$ npm install --save-dev solidity-coverage@beta

Add this package to your plugins array in truffle-config.js

module.exports = {
  networks: {...},
  plugins: ["solidity-coverage"]
}

Run

truffle run coverage [command-options]

Documentation

• Options and other instructions
• A guide to upgrading from 0.6.x to 0.7.x
• Some notes on advanced use cases

🐘 What's coming... 🐘

There are two more phases of development planned for solidity-coverage in the next couple of months:

Phase 2: We'll publish a Buidler plugin and finalize a general API which lets you use this tool with any development setup.

Phase 3: We'll implement a set of new, solidity-specific coverage measurements which engineers have proposed over the years to increase the precision of the reports you get:

• Per-function modifier coverage , which treats modifiers as branches in the code for each function, e.g for each method qualified by the modifier Ownable, measure whether it was called by owners and non-owners (Proposed by Zeppelin engineer Nicolas Venturo in solidity-coverage #286)
• Public variable coverage, which treats public variables as methods (since getters are implicitly generated for them) and provide a way for developers to track changes in their contracts' public API. (Proposed by Zeppelin engineer Nicolas Venturo in solidity-coverage #281)
• && / || conditional coverage: , which treats conjoined conditions as distinct execution branches. This precision in branch coverage reporting is available for Java with the cobratura coverage tool - Solidity could benefit from it too. (Proposed by Gnosis engineer Richard Meissner in solidity-coverage #175)
• Documentation coverage, which measures Natspec comment-to-parameter mappings and provides a way to track documentation completeness
• CI integrations for the above Where additional coverage measurements can't be adequately represented by Istanbul/Coveralls we'll add support for them with a githooks/CI integration that tracks pull-request changes relative to their target branch.

Thanks for trying this out!

Please open issues/ask for help if you run into trouble installing or see problems ;)

v0.6.4

🐘 🐘

This release upgrades the coverage client testrpc-sc to 6.5.1, tracking ganache-cli. There are some long-awaited bug fixes arriving from upstream...

• Block timestamps no longer float within a given block. The non-static (and erroneous) behavior was especially noticeable when running coverage because it's slower.
• The value of eth_blockNumber and the Solidity global block.number are now identical, instead of off-by-one.

Thanks to @barakman for reporting and keeping on top of these.

v0.6.x

Hello 🐘

This release resolves several long-standing issues:

• Instrumentation speeds have improved dramatically (i.e by orders of magnitude, thanks to the LeapDao fork.)
• Parsing is now managed by @federicobond's wonderful solidity-parser-antlr and current with the latest solidity syntax. (Thanks to @area)
• testrpc-sc has been rebased to ganache-cli 6.4.5 It's petersburg enabled and runs with the --allowUnlimitedContractSize and --emitFreeLogs flags already on by default. Coverage no longer distorts gas measurements within solidity code.
• all PRs are E2E tested against the latest state of the openzeppelin-solidity repo.

Apologies to all for the gap in development here in 2018/2019 - regular maintainer @cgewecke vanished without trace or explanation for months.

Credits and Kudos to...
JoinColony's @area, LeapDao's @pinkiebell, HQ20's @obernardovieira and the contributors to #318 for getting SC to work with newer Solidity and the Petersburg fork.

Also to Ripio's @Agusx1211 for getting --emitFreeLogs into ethereumjs-vm. Brilliant.

Hoping to implement some changes here in July/August to instrument files without using events and make the tool artifact-type neutral. Generating coverage for solidity is a little tricky and it's not clear anyone has discovered a perfect way to do it. If this package isn't working for you, it's well worth checking out two other options:

• Leon Logvinov's sol-cover built for 0xProject
• Embark's coverage tool

-cg