You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
... to allow external standards to more easily test their sniffs without using the PHPCS native unit test framework.
Advantages this will/can provide:
Tests will keep working once PHPCS 4.0 comes out.
As of PHPCS 4.0, the distribution package of PHPCS won't ship with the test directories, so the abstract testsuite and sniff classes won't be available. This can be worked-around by using --prefer-source in the CI composer install if needs be.
Also see: Remove tests from the composer package squizlabs/PHP_CodeSniffer#1908 and squizlabs/PHP_CodeSniffer@cf91994
With the PHPCS native setup, you can only use the PHPUnit --filter option to filter by standard. Just running the tests for one individual sniff or for a category is difficult to do.
The base sniff test class I envision would allow for the PHPUnit --filter option to work properly again as expected.
No need anymore to work around the PHPCS native TestSuite which loads every single standard and tries to find the tests for it. The current way to ignore standards using PHPCS_IGNORE_TESTS works, but is fiddly and requires too much manual maintenance.
Things to consider adding on top of the functionality which is in the PHPCS native base sniff test class:
Allow for tests to provide a list of all error codes included in the sniff and verify that all error codes are tested (have at least one error/warning reported)
This could be used by standards which don't check code coverage to at least verify that all error codes are tested.
Allow for testing that the messages thrown are actually the messages expected.
Just counting the number of messages may still hide changed sniff behaviour if one message is replaced by another.
Will probably need an array with line -> errorcode -> nr of errors.
Allow for testing expected metrics.
Not a good idea if those would accidentally become inaccurate, so would be great to be able to safeguard these via tests.
This could be set up as plug and play, optional extra functionality to add to any testcase via a trait with a test method and an abstract data provider method.
Allow for testing the contents of select error messages.
This is useful if the replacement values aren't just simple values, i.e. the sniff contains logic to create the replacement values which is a little more complex.
This could be set up as plug and play, optional extra functionality to add to any testcase via a trait with a test method and an abstract data provider method.
Fail tests if the sniff under test contains fixers and the test case file would trigger those (= would yield a diff), but there is no .fixed version available for the test case file.
Implementation considerations
Implementation wise switching to the new PHPCSUtils test class should be as easy as changing the use statement of a test class and updating the CI command (removing the calling of the PHPCS TestSuite file).
Using new/additional functionality should be optional and elective.
The text was updated successfully, but these errors were encountered:
... to allow external standards to more easily test their sniffs without using the PHPCS native unit test framework.
Advantages this will/can provide:
As of PHPCS 4.0, the distribution package of PHPCS won't ship with the test directories, so the abstract testsuite and sniff classes won't be available. This can be worked-around by using
--prefer-source
in the CI composer install if needs be.Also see: Remove tests from the composer package squizlabs/PHP_CodeSniffer#1908 and squizlabs/PHP_CodeSniffer@cf91994
--filter
option to filter by standard. Just running the tests for one individual sniff or for a category is difficult to do.The base sniff test class I envision would allow for the PHPUnit
--filter
option to work properly again as expected.PHPCS_IGNORE_TESTS
works, but is fiddly and requires too much manual maintenance.Compatibility with PHPUnit 8.x/9.xThis has since been sorted in PHPCS itself via Tests: allow the test suite to run on PHPUnit 8.x and 9.x PHP_CodeSniffer#59.
Things to consider adding on top of the functionality which is in the PHPCS native base sniff test class:
This could be used by standards which don't check code coverage to at least verify that all error codes are tested.
Just counting the number of messages may still hide changed sniff behaviour if one message is replaced by another.
Will probably need an array with
line -> errorcode -> nr of errors
.Not a good idea if those would accidentally become inaccurate, so would be great to be able to safeguard these via tests.
This could be set up as plug and play, optional extra functionality to add to any testcase via a trait with a test method and an abstract data provider method.
This is useful if the replacement values aren't just simple values, i.e. the sniff contains logic to create the replacement values which is a little more complex.
This could be set up as plug and play, optional extra functionality to add to any testcase via a trait with a test method and an abstract data provider method.
.fixed
version available for the test case file.Implementation considerations
use
statement of a test class and updating the CI command (removing the calling of the PHPCS TestSuite file).Using new/additional functionality should be optional and elective.
The text was updated successfully, but these errors were encountered: