From e0cfb3065ca08c64d0812b51e7127840cd6ae9ee Mon Sep 17 00:00:00 2001 From: William Woodall Date: Tue, 17 Dec 2019 15:29:10 -0800 Subject: [PATCH 01/14] initial commit of new rep-2004 Signed-off-by: William Woodall --- rep-2004.rst | 527 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 527 insertions(+) create mode 100644 rep-2004.rst diff --git a/rep-2004.rst b/rep-2004.rst new file mode 100644 index 000000000..093b55575 --- /dev/null +++ b/rep-2004.rst @@ -0,0 +1,527 @@ +REP: 2004 +Title: Package Quality Categories +Author: William Woodall +Status: Active +Type: Informational +Content-Type: text/x-rst +Created: 17-Dec-2019 +Post-History: 17-Dec-2019 + + +Abstract +======== + +This REP describes a set of categories which are meant to convey the quality, or at least the maturity, of packages in the ROS ecosystem. +Inclusion in one category or another is based on the policies to which the packages adhere. +The categories are meant to give some expectation as to the quality of a package and allows the maintainers to be more strict with some packages and less so with others. + +The purpose of these categories is not to enforce quality, but to set expectations for consumers of the packages and to encourage maintainers of the packages to document how their package's policies achieve that quality level. +The documented policies allow consumers of the packages to consider any caveats for the package or its dependencies when deciding whether or not the package meets the standards for their project. + + +Specification +============= + +The categories also provide rough goals for packages to strive towards, encouraging better quality across the ecosystem. + +There are four quality levels described below, each roughly described as: + +* Quality Level 1: + + * highest quality level + * packages which are needed for production systems + * e.g. ``rclcpp``, ``urdf``, ``tf2``, etc. + +* Quality Level 2: + + * high quality packages which are either: + + * on the way to level 1 or + * are general solutions used by many people, but are only sometimes used for production systems + + * e.g. ``navigation2``, ``rosbag2``, etc. + +* Quality Level 3: + + * tooling quality packages + * e.g. ``ros2cli``, ``rviz``, ``rqt``, etc. + +* Quality Level 4: + + * demos, tutorials, and experiments + * e.g. research packages, ``demo_nodes_cpp``, ``examples_rclcpp_minimal_publisher``, etc. + +While each quality level will have different requirements, it's always possible to overachieve in certain requirements even if other requirements prevent a package from moving up to the next quality level. + +Quality Level 1 +^^^^^^^^^^^^^^^ + +This category should be used for packages which are required for a reasonable ROS system in a production environment. +That is to say that after you remove development tools, build tools, and introspection tools, these packages are still left over as requirements for a basic ROS system to run. +However, that does not mean that packages that would not normally fit this description should never be called 'Level 1'. +If there is a need for a particular package in a reasonable production scenario, then that package should be considered for this category as well. +However, packages which we consider essential to getting a robot up and running quickly, but perhaps is a generic solution to the problem should probably not start out as 'Level 1' due to the high effort in getting a package to 'Level 1' and maintaining it there. + +For example, the packages which provide intra-process communication, inter-process communication, generated message runtime code, node lifecycle, etc. should probably all be considered for 'Level 1'. +However, a package which provides pose estimation (like ``robot_pose_ekf``\ ) is a generic solution for something that most people need, but is often replaced with a domain specific solution in production, and therefore it should probably not start out as 'Level 1'. +However, it may upgrade to it at a later date, if it proves to be a solution that people want to use in their products. + +Tools, like ``rostopic``\ , generally do not fall into this category either, but are not categorically excluded. +For example, it may be the case the tool which launches and verifies a ROS graph (``ros2launch``\ ) may need to be considered 'Level 1' for use in production systems. + +Package Requirements +~~~~~~~~~~~~~~~~~~~~ + +*Note: bullets below that start with [ROS Core], will be the prescription for what we do in the core packages in order to meet the associated requirements* + +Requirements to be considered a 'Level 1' package: + +* Version Policy: + + * Must have a version policy (e.g. ``semver``) + * Must be at a stable version (e.g. for ``semver`` that means have a version >= 1.0.0) + * Must have a strictly declared public API + * Must have a policy for API stability + * Must have a policy for ABI stability + * Must have a policy that keeps API and ABI stability within a released ROS Distribution + * [ROS Core] will use ``semver``, will maintain API and ABI stability according to ``semver`` and will be ABI (and therefore API) stable within a ROS distribution + +* Change Control Process: + + * Must have all code changes occur through a change request (e.g. pull request, merge request, etc.) + * Must have peer review policy for all change requests (e.g. require one or more reviewer) + * Must have Continuous Integration (CI) policy for all change requests + * Must have documentation policy for all change requests + * [ROS Core]: + + * All changes will go through a pull request + * All pull requests will require at least one reviewer who did not author the pr (package may choose to increase this number) + * All pull requests will be tested via CI, and on all tier 1 platforms (if applicable) + * Any required changes to documentation (API documentation, feature documentation, release notes, etc.) must be proposed before merging related changes + +* Documentation: + + * Must have documentation for each "feature" (e.g. for ``rclcpp``: create a node, publish a message, spin, etc.) + * Must have documentation for each item in the public API (e.g. functions, classes, etc.) + * Must have a declared license or set of licenses + * Must have a copyright statement in each source file + * Must have a "quality declaration" document, which declares the quality level and justifies how the package meets each of the requirements + + * Must have a section in the repository's ``README`` which contains the "quality declaration" or links to it + * Must register with a centralized list of 'Level 1' packages, if one exists, to allow for peer review of the claim + + * [ROS Core]: + + * Must have automated checks for copyright statements and licenses + * Must use the Apache 2.0 license, unless the package has an existing permissive license (e.g. rviz uses three-clause BSD) + +* Testing: + + * Must have system tests which cover all items in the "feature" documentation + * Must have system, integration, and/or unit tests which cover all of the public API + * Code coverage: + + * Must have code coverage tracking for the package + * Must have and enforce a code coverage policy for new changes + * [ROS Core]: + + * Must provide line coverage + * Must achieve a line coverage above 95% + * May pick a lower percentage target with justification, but must document it prominently + * May provide branch coverage + * May exclude code from coverage (test code, debug code, etc.) + * Must require coverage to increase or stay the same before merging a change, but... + * May accept a change that decreases coverage with proper justification (e.g. deleting code that was previously covered can cause the percentage to drop) + + * Performance: + + * Must have performance tests (exceptions allowed if they don't make sense to have) + * Must have a performance regression policy (i.e. blocking either changes or releases on unexpected performance regressions) + * [ROS Core]: + + * May have performance tests, strongly recommended, but for some packages it doesn't make sense + * If there are performance tests, must choose to either check each change or before each release or both + * If there are performance tests, must require justification for merging a change or making a release that lowers performance + + * Linters and Static Analysis + + * Must have a code style and enforce it. + * Must use static analysis tools where applicable. + * [ROS Core]: + + * Must use ROS code style and use linters from ``ament_lint_common`` to enforce it + * Must use all linters/static analysis that are part of ``ament_lint_common`` + +* Dependencies: + + * Must not have direct runtime "ROS" dependencies which are not 'Level 1' dependencies, but... + * May have optional direct runtime "ROS" dependencies which are not 'Level 1', e.g. tracing or debugging features that can be disabled + * Must have justification for why each direct runtime "non-ROS" dependency is equivalent to a 'Level 1' package in terms of quality + +* Platform Support: + + * Must support all tier 1 platforms for ROS 2, as defined in `REP-2000 `_ + +If the above points are satisfied then a package can be considered 'Level 1'. +Below are some details on the above points. + +Version Policy +"""""""""""""" + +The most important thing is to have some version policy which developers may use to anticipate and understand changes to the version of the package. +We recommend the use of ``semver`` as it covers all the important points that a version policy should cover, is well thought out, and is popular in the open source community broadly. + +The policy should link changes to API and ABI to the version scheme. + +Additionally, specifically for the ROS ecosystem, the policy should state that API and ABI will be maintained within a stable ROS distribution. +For ``semver``, this means only patch and minor increases only into an existing ROS distribution. + +Public API +"""""""""" + +The package should also state what the public API includes, and/or state what parts of the API are excluded intentionally. + +For C++, it's somewhat obvious that all installed headers are part of the public API, but it's acceptable to have parts of the accessible API not be stable. +For example, having an "experimental" namespace or a "detail" namespace which does not adhere to the API and ABI stability rules is allowed, but they must be clearly documented as such. +Changes to these excluded API's, especially something like a "detail" namespace, should still not break API or ABI for other public API's indirectly. + +For Python, it's more important to explicitly declare which parts of the API is public, because all modules are typically installed and accessible to users. +One easy thing to do is to say all of the API is public and therefore API stable, but "impl" or "detail" namespaces can be used if needed, they just need to be clearly documented as not public and therefore not stable. + +There are also other, non-API, things which should be considered and optionally documented as part of the "stable interface" of the package. +This includes, but isn't limited to, message definitions, command line tools (arguments and output format), ROS names (topic, service, node, etc.), and behaviors of the applications. + +For yet other languages the details will be different, but the important thing is that the public API be obviously documented, and that the public API adheres to an API and ABI stability as described in the version policy, and that they are documented and tested. + +Feature Documentation +""""""""""""""""""""" + +For each feature provided by the public API of the package, or by a tool in the package, there must be corresponding user documentation. +The term "feature", and the scope of the documentation, is intentionally vague because it's difficult to quantitatively measure this metric. +However, the spirit of this requirement is that, for a 'Level 1' quality package, all of the things a user might do with the package needs at least basic documentation or a snippet of code as an example on how to use it. +The `roscpp Overview `_ from the ROS 1 wiki is a good example of this kind of documentation. + +Feature Testing and Code Coverage Policy +"""""""""""""""""""""""""""""""""""""""" + +This policy should aim for a "high" coverage standard, but the exact number and rules will vary depending on the package in question. +The policy may be influenced by factors like: + +- what programming languages are being used, and whether or not there are multiple languages in use +- what coverage information is available (statement vs. line vs. branch vs condition/path coverage) +- what strategy is preferred for dealing with difficult to reach statements/branches + +This StackOverflow question is a good summary of the issues: + +https://stackoverflow.com/questions/90002/what-is-a-reasonable-code-coverage-for-unit-tests-and-why + +In particular, this answer does a good job of summarizing the issue: + +https://stackoverflow.com/a/34698711/671658 + +Importantly, this answer points out that tracking and enforcing code coverage statistics is strictly empirical (rather than theoretical) and that there are different reasons for using them. +Among those reasons listed is "To satisfy stakeholders", which is the main goal of requiring a code coverage policy for these high quality packages. +It is summarized nicely: + + For many projects, there are various actors who have an interest in software quality who may not be involved in the day-to-day development of the software (managers, technical leads, etc.) + Saying "we're going to write all the tests we really need" is not convincing: + They either need to trust entirely, or verify with ongoing close oversight (assuming they even have the technical understanding to do so.) + Providing measurable standards and explaining how they reasonably approximate actual goals is better. + +The other two reasons "To normalize team behavior" and "To keep yourself honest" are nice reasons to have code coverage goals, but are out of scope for this document. + +The general recommendation is to have at least line coverage and aim to achieve and maintain a high percentage of coverage (e.g. above 90%). +This at least gives you and your stakeholders some confidence that all feature have basic tests. +Any assurances beyond that would require branch coverage statistics and independent investigation of the tests and how they test the code. + +Performance Testing +""""""""""""""""""" + +There are some cases where performance testing does not make sense to have. +For example, it may be a good idea to have performance tests for a code generator (like ``rosidl_generator_cpp``), but it is not strictly required since its performance does not affect a runtime production system, and so in that case the package could claim to be 'Level 1' without performance tests if properly justified in the "quality declaration". + +However, if performance is a reasonable concern for use in a production system, then there must be performance tests and they should be used in conjunction with a regression policy which aims to prevent new versions of the package to be considerably slower without cause. +Note, the performance regression policy should not prevent regressions, but instead should aim to detect them and either address them directly, plan to address them in the future, or when unavoidable (e.g. fixing a bug required more resources to be safe) explain why the regression has occurred in the memorandum of the change request that introduced it. + +Dependencies +"""""""""""" + +Each package should examine their direct runtime dependencies for their quality levels. +Packages should not claim a quality level higher than their dependencies, unless it can be reasonably explained why they do not affect the quality of the package in question. + +An example of this would be build or "build tool" dependencies, which are only used during build time and do not impact the runtime quality of the package. +This would not include, however, build dependencies which, for example, contribute only headers to a C++ library or a static library, as the quality of those headers or static library also impact the quality of the runtime product directly. +This would include, for another example, something like CMake, which in most ways does not impact the quality of the product. + +There's obviously a lot of ambiguity in this area, as you could argue for or against a variety of dependencies and how they impact the package. +However, the point is to require the maintainers of the package to examine each dependency, justify why they do or do not impact the quality, and document that so that peer reviewers and consumers of the package can make their own evaluation. + +Dependencies which are other "ROS" packages should have these quality standards applied to them and should meet or exceed the quality level claimed by the package in question. + +Dependencies which are not other "ROS" packages should be individually examined for quality. +You may either try to apply the requirements for the quality levels described here, or you may wish to simply argue the quality without using these requirements as a ruler. +In either case, for each direct "non-ROS" dependency your "quality declaration" should include a justification as to why it is acceptable to depend on this software and still claim your package's level of quality. +This may simply be text justification, or it may link to other analysis or discussions had by community members rationalizing the choice. +The important point is that each dependency is considered, justified, and that the justification is documented, so that users of the package can read the justification and decide for themselves if it is acceptable or not. + +Any important caveats or justified exceptions for your dependencies should be mentioned (or referenced) in your own package's "quality declaration" document. + +For example, if your package depends on ``rclcpp``, and ``rclcpp`` claims 'level 1' quality with the caveat that this requires you use an rmw implementation that also meets the 'level 1' quality standard, then your package's "quality declaration" document should mention this as well. +Perhaps just saying that one of your dependencies, ``rclcpp``, has some caveats and then link to ``rclcpp``'s own "quality declaration". + +In this way, caveats and justifications that may be important for peer reviewers and consumers of your package to understand can "bubble up" from any part of the system. + +The goal here is for the maintainer of a package to "make the case" to potential users or stakeholders that their dependencies are at least as high quality as the package in question, and to make a best effort attempt to make them aware of any issues or caveats. +It's up to those users and stakeholders to evaluate that justification and to look at the dependencies themselves as well. + +Claiming a Quality Level and Documenting Package Policies +""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +Each package claiming a quality level should have a "quality declaration" documented somewhere. +This declaration should include a claimed quality level and then should have a section for each of the requirements in that claimed quality level justifying how the package meets each of those requirements. + +Sometimes the justification will be a link to a policy documented in the package itself or it may link to a common policy used by a group of packages. +If there is additional evidence that these policies are being followed, that should be included as well, e.g. a link to the coverage statistics for the package to show that coverage is being tracked and maintained. +Other times, justification will be an explanation as to why a requirement was not met or does not apply, e.g. if performance tests do not make sense for the package in question, it should be satisfactorily explained. + +There is no enforcement or checking of these claims, but instead it's just sufficient to present this information to potential users. +If the users feel that the justifications are insufficient or incorrect, they can open issues against the repository and resolve it with the maintainers. + +There should be one or more communal lists of 'Level 1' (and maybe 'Level 2' or 'Level 3') quality level packages. +These lists should be modified via change requests (maybe a text document in a repository) so that there can be peer review. +This document will not prescribe how or where these lists should be hosted, but one thought is that the list could live on the main ROS 2 documentation website. + +Quality Level 2 +^^^^^^^^^^^^^^^ + +These are packages which need to be solidly developed and might be used in production environments, but are not strictly required, or are commonly replaced by custom solutions. +This can also include packages which are not yet up to 'Level 1' but intend to be in the future. + +Package Requirements +~~~~~~~~~~~~~~~~~~~~ + +*Note: bullets below that start with [ROS Core], will be the prescription for what we do in the core packages in order to meet the associated requirements* + +Requirements to be considered a 'Level 2' package: + +* Version Policy: + + * The same as 'Level 1' packages + +* Change Control Process: + + * Must have all code changes occur through a change request (e.g. pull request, merge request, etc.) + * Must have Continuous Integration (CI) policy for all change requests + * [ROS Core]: + + * All changes will go through a pull request + * All pull requests will be tested via CI + +* Documentation: + + * Must have documentation for each "feature" (e.g. for ``rclcpp``: create a node, publish a message, spin, etc.) + * Must have a declared license or set of licenses + * Must have a copyright statement in each source file + * Must have a "quality declaration" document, which declares the quality level and justifies how the package meets each of the requirements + + * Must have a section in the repository's ``README`` which contains the "quality declaration" or links to it + * Must register with a centralized list of 'Level 2' packages, if one exists, to allow for peer review of the claim + + * [ROS Core]: + + * Must have automated checks for copyright statements and licenses + * Must use the Apache 2.0 license, unless the package has an existing permissive license (e.g. rviz uses three-clause BSD) + +* Testing: + + * Must have system tests which cover all items in the "feature" documentation + * Code coverage: + + * Must have code coverage tracking for the package + * [ROS Core]: + + * Must provide line coverage statistics + * May provide branch coverage + * May exclude code from coverage (test code, debug code, etc.) + + * Linters and Static Analysis + + * Must have a code style and enforce it. + * Must use static analysis tools where applicable. + * [ROS Core]: + + * Must use ROS code style and use linters from ``ament_lint_common`` to enforce it + * Must use all linters/static analysis that are part of ``ament_lint_common`` + +* Dependencies: + + * Must not have direct runtime "ROS" dependencies which are not 'Level 2' dependencies, but... + * May have optional direct runtime "ROS" dependencies which are not 'Level 2', e.g. tracing or debugging features that can be disabled + * Must have justification for why each direct runtime "non-ROS" dependency is equivalent to a 'Level 2' package in terms of quality + +* Platform Support: + + * Must support all tier 1 platforms for ROS 2, as defined in `REP-2000 `_ + +If the above points are satisfied then a package can be considered 'Level 2'. +Refer to the detailed description of the requirements in the Quality Level 1 section above for more information. + +Quality Level 3 +^^^^^^^^^^^^^^^ + +These are packages which are useful for development purposes or introspection, but are not recommended for use in embedded products or mission critical scenarios. +These packages are more lax on documentation, testing, and scope of public API's in order to make development time lower or foster addition of new features. + +Package Requirements +~~~~~~~~~~~~~~~~~~~~ + +*Note: bullets below that start with [ROS Core], will be the prescription for what we do in the core packages in order to meet the associated requirements* + +Requirements to be considered a 'Level 3' package: + +* Version Policy: + + * The same as 'Level 1' packages, except: + + * No public API needs to be explicitly declared, though this can make it harder to maintain API and ABI stability + * No requirement to keep API/ABI stability within a stable ROS release, but it is recommended still + +* Change Control Process: + + * Must have all code changes occur through a change request (e.g. pull request, merge request, etc.) + * Must have Continuous Integration (CI) policy for all change requests + * [ROS Core]: + + * All changes will go through a pull request + * All pull requests will be tested via CI + +* Documentation: + + * Must have a declared license or set of licenses + * Must have a copyright statement in each source file + * May have a "quality declaration" document, which declares the quality level and justifies how the package meets each of the requirements + + * Must have a section in the repository's ``README`` which contains the "quality declaration" or links to it + * May register with a centralized list of 'Level 3' packages, if one exists, to allow for peer review of the claim + + * [ROS Core]: + + * Must have automated checks for copyright statements and licenses + * Must use the Apache 2.0 license, unless the package has an existing permissive license (e.g. rviz uses three-clause BSD) + +* Testing: + + * No explicit testing requirements, though covering some if not all of the features with tests is recommended + +* Dependencies: + + * May have direct runtime "ROS" dependencies which are not 'Level 3' dependencies, but they should be documented + +* Platform Support: + + * Must support all tier 1 platforms for ROS 2, as defined in `REP-2000 `_ + +If the above points are satisfied then a package can be considered 'Level 3'. +Refer to the detailed description of the requirements in the Quality Level 1 section above for more information. + +Quality Level 4 +^^^^^^^^^^^^^^^ + +These are demos, tutorials, or experiments. +They don't have strict requirements, but are not excluded from having good documentation or tests. +For example, this might be a tutorial package which is not intended for reuse but has excellent documentation because it serves primarily as an example to others. + +Package Requirements +~~~~~~~~~~~~~~~~~~~~ + +*Note: bullets below that start with [ROS Core], will be the prescription for what we do in the core packages in order to meet the associated requirements* + +Requirements to be considered a 'Level 4' package: + +* Version Policy: + + * No requirements, but having a policy is still recommended (e.g. ``semver``), even if the version is not yet stable (e.g. >= 1.0.0 for ``semver``) + +* Change Control Process: + + * No explicit change control process required, but still recommended + +* Documentation: + + * Must have a declared license or set of licenses + * Must have a copyright statement in each source file + * [ROS Core]: + + * Must have automated checks for copyright statements and licenses + * Must use the Apache 2.0 license, unless the package has an existing permissive license (e.g. rviz uses three-clause BSD) + +* Testing: + + * No explicit testing requirements, though covering some if not all of the features with tests is recommended + +* Dependencies: + + * No restrictions + +* Platform Support: + + * May support all tier 1 platforms for ROS 2, as defined in `REP-2000 `_ + +Any package that does not claim to be 'Level 3' or higher is automatically 'Level 4'. +Refer to the detailed description of the requirements in the Quality Level 1 section above for more information. + +Quality Level 5 +^^^^^^^^^^^^^^^ + +Packages in this category simply do not meet even the 'Level 4' requirements, and for that reason should not be used. +The rationale being that all packages should have at least a declare license or licenses and should include copyright statements in each file. + +Repository Organization +^^^^^^^^^^^^^^^^^^^^^^^ + +Since these categories are applied on a per package basis, and since there may be more than one package per source repository, it's recommended that the strictest set of policies apply to the whole repository. +This is recommended, rather than trying to mix processes depending on which packages are changed in a given change request (pull request or merge request, etc.). +If this is too onerous, then it's recommended to split lower quality packages out into a separate repository. + + +Motivation +========== + +TBD + + +Rationale +========= + +TBD + + +Backwards Compatibility +======================= + +Since there are no preceeding standards in this space, at least in the ROS community, of which we are aware, there are no backwards compatibility concerns. + + +Reference Implementation +======================== + +TBD: Link to "ROS Core" perscription for how to actually achieve each quality level. + + +References and Footnotes +======================== + +.. [1] Initial discussions about this REP: + https://github.com/ros2/ros2_documentation/pull/460 + + +Copyright +========= + +Copyright 2019 Open Source Robotics Foundation, Inc. + + +License +======= + +Creative Commons Attribution 4.0 International From 7640cce93f5c0b140faa8a919beb1a2870f52896 Mon Sep 17 00:00:00 2001 From: William Woodall Date: Tue, 17 Dec 2019 15:48:57 -0800 Subject: [PATCH 02/14] update my (wjwwood's) email addresses, because the rep script only allows one per person Signed-off-by: William Woodall --- rep-0133.rst | 2 +- rep-2004.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/rep-0133.rst b/rep-0133.rst index db274489c..5da1d3e85 100644 --- a/rep-0133.rst +++ b/rep-0133.rst @@ -1,6 +1,6 @@ REP: 133 Title: Separation of build environment and source tree tools -Author: Tully Foote , Dirk Thomas, Thibault Kruse , William Woodall +Author: Tully Foote , Dirk Thomas, Thibault Kruse , William Woodall Status: Active Type: Informational Content-Type: text/x-rst diff --git a/rep-2004.rst b/rep-2004.rst index 093b55575..740befd13 100644 --- a/rep-2004.rst +++ b/rep-2004.rst @@ -1,6 +1,6 @@ REP: 2004 Title: Package Quality Categories -Author: William Woodall +Author: William Woodall Status: Active Type: Informational Content-Type: text/x-rst From c2ba03bac23b8f313e47820833fad000dbbe058b Mon Sep 17 00:00:00 2001 From: William Woodall Date: Wed, 18 Dec 2019 16:51:20 -0800 Subject: [PATCH 03/14] fixup copyright/license statements Signed-off-by: William Woodall --- rep-2004.rst | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/rep-2004.rst b/rep-2004.rst index 740befd13..68230b19d 100644 --- a/rep-2004.rst +++ b/rep-2004.rst @@ -518,10 +518,4 @@ References and Footnotes Copyright ========= -Copyright 2019 Open Source Robotics Foundation, Inc. - - -License -======= - -Creative Commons Attribution 4.0 International +This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. From c8bd456f531acc4863d0bb8888c28b10f9271492 Mon Sep 17 00:00:00 2001 From: Marya Belanger Date: Fri, 6 Mar 2020 15:05:51 -0800 Subject: [PATCH 04/14] REP 2004 extras (#235) * REP 2004 extras Signed-off-by: maryaB-osr * possible chart structure and license question Signed-off-by: maryaB-osr * comment out chart Signed-off-by: maryaB-osr * restructure and edits Signed-off-by: maryaB-osr * fixed a header Signed-off-by: maryaB-osr * add chart Signed-off-by: maryaB-osr * q. level 3 testing in wrong order Signed-off-by: maryaB-osr * level 3 3.v.a mistake Signed-off-by: maryaB-osr * moved chart to be under level 1 makes it easier to read the numbers Signed-off-by: maryaB-osr * wjwwood review Signed-off-by: maryaB-osr * edits / chart check Signed-off-by: maryaB-osr * fix python incompatibility Signed-off-by: maryaB-osr --- .travis.yml | 3 +- rep-2004.rst | 711 +++++++++++++++++++++++++++++++-------------------- 2 files changed, 435 insertions(+), 279 deletions(-) diff --git a/.travis.yml b/.travis.yml index 5fcad770c..094f0d9b3 100644 --- a/.travis.yml +++ b/.travis.yml @@ -5,7 +5,8 @@ python: # command to install dependencies install: - pip install docutils - - pip install xmlschema + # xmlschema 1.1 dropped support for Python 2 + - if [ $TRAVIS_PYTHON_VERSION == "2.7" ]; then pip install "xmlschema<1.1"; else pip install xmlschema; fi # command to run tests script: - make diff --git a/rep-2004.rst b/rep-2004.rst index 68230b19d..9f2d2259c 100644 --- a/rep-2004.rst +++ b/rep-2004.rst @@ -11,32 +11,37 @@ Post-History: 17-Dec-2019 Abstract ======== -This REP describes a set of categories which are meant to convey the quality, or at least the maturity, of packages in the ROS ecosystem. -Inclusion in one category or another is based on the policies to which the packages adhere. -The categories are meant to give some expectation as to the quality of a package and allows the maintainers to be more strict with some packages and less so with others. +This REP describes a set of categories meant to convey the quality or maturity of packages in the ROS ecosystem. +Inclusion in a category, or quality level, is based on the policies to which a package adheres. +The categories address version, change control, documentation, testing, dependency and platform support policies. -The purpose of these categories is not to enforce quality, but to set expectations for consumers of the packages and to encourage maintainers of the packages to document how their package's policies achieve that quality level. -The documented policies allow consumers of the packages to consider any caveats for the package or its dependencies when deciding whether or not the package meets the standards for their project. +Motivation +========== + +The purpose of this REP is to provide standard guidelines regarding package quality, for both maintainers and consumers. +The categories are meant to set quality expectations for package consumers, not to enforce quality rules on maintainers. +Package maintainers are responsible for establishing processes and documenting how their package's policies achieve a certain quality level. +With the help of documented policies, package consumers can better evaluate whether a package or its dependencies meet the standards for use in their projects. + +By providing these rough goals for package maintainers to strive towards, the categories will also encourage better quality across the ROS ecosystem. Specification ============= -The categories also provide rough goals for packages to strive towards, encouraging better quality across the ecosystem. - -There are four quality levels described below, each roughly described as: +There are four quality levels below, each roughly summarized as: * Quality Level 1: * highest quality level - * packages which are needed for production systems + * packages that are needed for production systems * e.g. ``rclcpp``, ``urdf``, ``tf2``, etc. * Quality Level 2: - * high quality packages which are either: + * high quality packages that are either: - * on the way to level 1 or + * on the way to 'Level 1', or * are general solutions used by many people, but are only sometimes used for production systems * e.g. ``navigation2``, ``rosbag2``, etc. @@ -57,239 +62,388 @@ Quality Level 1 ^^^^^^^^^^^^^^^ This category should be used for packages which are required for a reasonable ROS system in a production environment. -That is to say that after you remove development tools, build tools, and introspection tools, these packages are still left over as requirements for a basic ROS system to run. -However, that does not mean that packages that would not normally fit this description should never be called 'Level 1'. +These are the packages that remain as requirements for a basic ROS system to run after you remove development tools, build tools, and introspection tools. + +Packages that would not normally fit this description can still be called 'Level 1'. If there is a need for a particular package in a reasonable production scenario, then that package should be considered for this category as well. -However, packages which we consider essential to getting a robot up and running quickly, but perhaps is a generic solution to the problem should probably not start out as 'Level 1' due to the high effort in getting a package to 'Level 1' and maintaining it there. +However, packages which we consider essential to getting a robot up and running quickly, but perhaps provide a generic solution to a problem should probably not start out as 'Level 1' due to the high effort required to get a package to 'Level 1' and maintain it there. -For example, the packages which provide intra-process communication, inter-process communication, generated message runtime code, node lifecycle, etc. should probably all be considered for 'Level 1'. +For example, packages which provide intra-process communication, inter-process communication, generated message runtime code, node lifecycle, etc. should probably all be considered for 'Level 1'. However, a package which provides pose estimation (like ``robot_pose_ekf``\ ) is a generic solution for something that most people need, but is often replaced with a domain specific solution in production, and therefore it should probably not start out as 'Level 1'. -However, it may upgrade to it at a later date, if it proves to be a solution that people want to use in their products. +Such a package may upgrade to 'Level 1' at a later date, if it proves to be a solution that people want to use in their products. Tools, like ``rostopic``\ , generally do not fall into this category either, but are not categorically excluded. -For example, it may be the case the tool which launches and verifies a ROS graph (``ros2launch``\ ) may need to be considered 'Level 1' for use in production systems. +For example, it may be the case that the tool which launches and verifies a ROS graph (``ros2launch``\ ) may need to be considered 'Level 1' for use in production systems. Package Requirements ~~~~~~~~~~~~~~~~~~~~ -*Note: bullets below that start with [ROS Core], will be the prescription for what we do in the core packages in order to meet the associated requirements* - Requirements to be considered a 'Level 1' package: -* Version Policy: - - * Must have a version policy (e.g. ``semver``) - * Must be at a stable version (e.g. for ``semver`` that means have a version >= 1.0.0) - * Must have a strictly declared public API - * Must have a policy for API stability - * Must have a policy for ABI stability - * Must have a policy that keeps API and ABI stability within a released ROS Distribution - * [ROS Core] will use ``semver``, will maintain API and ABI stability according to ``semver`` and will be ABI (and therefore API) stable within a ROS distribution - -* Change Control Process: - - * Must have all code changes occur through a change request (e.g. pull request, merge request, etc.) - * Must have peer review policy for all change requests (e.g. require one or more reviewer) - * Must have Continuous Integration (CI) policy for all change requests - * Must have documentation policy for all change requests - * [ROS Core]: - - * All changes will go through a pull request - * All pull requests will require at least one reviewer who did not author the pr (package may choose to increase this number) - * All pull requests will be tested via CI, and on all tier 1 platforms (if applicable) - * Any required changes to documentation (API documentation, feature documentation, release notes, etc.) must be proposed before merging related changes +#. **Version Policy:** -* Documentation: + #. Must have a version policy (e.g. ``semver``) + #. Must be at a stable version (e.g. for ``semver`` that means version >= 1.0.0) + #. Must have a strictly declared public API + #. Must have a policy for API stability + #. Must have a policy for ABI stability + #. Must have a policy that keeps API and ABI stability within a released ROS distribution - * Must have documentation for each "feature" (e.g. for ``rclcpp``: create a node, publish a message, spin, etc.) - * Must have documentation for each item in the public API (e.g. functions, classes, etc.) - * Must have a declared license or set of licenses - * Must have a copyright statement in each source file - * Must have a "quality declaration" document, which declares the quality level and justifies how the package meets each of the requirements +#. **Change Control Process:** - * Must have a section in the repository's ``README`` which contains the "quality declaration" or links to it - * Must register with a centralized list of 'Level 1' packages, if one exists, to allow for peer review of the claim + #. Must have all code changes occur through a change request (e.g. pull request, merge request, etc.) + #. Must have confirmation of contributor origin (e.g. `DCO `_, CLA, etc.) + #. Must have peer review policy for all change requests (e.g. require one or more reviewers) + #. Must have Continuous Integration (CI) policy for all change requests + #. Must have documentation policy for all change requests - * [ROS Core]: +#. **Documentation:** - * Must have automated checks for copyright statements and licenses - * Must use the Apache 2.0 license, unless the package has an existing permissive license (e.g. rviz uses three-clause BSD) + #. Must have documentation for each "feature" (e.g. for ``rclcpp``: create a node, publish a message, spin, etc.) + #. Must have documentation for each item in the public API (e.g. functions, classes, etc.) + #. Must have a declared license or set of licenses + #. Must have a copyright statement in each source file + #. Must have a "quality declaration" document, which declares the quality level and justifies how the package meets each of the requirements -* Testing: + #. *Must have a section in the repository's ``README`` which contains the "quality declaration" or links to it* + #. *Must register with a centralized list of 'Level N' packages, if one exists, to allow for peer review of the claim* - * Must have system tests which cover all items in the "feature" documentation - * Must have system, integration, and/or unit tests which cover all of the public API - * Code coverage: +#. **Testing:** - * Must have code coverage tracking for the package - * Must have and enforce a code coverage policy for new changes - * [ROS Core]: + #. Must have system tests which cover all items in the "feature" documentation + #. Must have system, integration, and/or unit tests which cover all of the public API + #. Code Coverage: - * Must provide line coverage - * Must achieve a line coverage above 95% - * May pick a lower percentage target with justification, but must document it prominently - * May provide branch coverage - * May exclude code from coverage (test code, debug code, etc.) - * Must require coverage to increase or stay the same before merging a change, but... - * May accept a change that decreases coverage with proper justification (e.g. deleting code that was previously covered can cause the percentage to drop) + #. *Must have code coverage tracking for the package* + #. *Must have and enforce a code coverage policy for new changes* - * Performance: + #. Performance: - * Must have performance tests (exceptions allowed if they don't make sense to have) - * Must have a performance regression policy (i.e. blocking either changes or releases on unexpected performance regressions) - * [ROS Core]: + #. *Must have performance tests (exceptions allowed if they don't make sense to have)* + #. *Must have a performance regression policy (i.e. blocking either changes or releases on unexpected performance regressions)* - * May have performance tests, strongly recommended, but for some packages it doesn't make sense - * If there are performance tests, must choose to either check each change or before each release or both - * If there are performance tests, must require justification for merging a change or making a release that lowers performance + #. Linters and Static Analysis - * Linters and Static Analysis + #. *Must have a code style and enforce it* + #. *Must use static analysis tools where applicable* - * Must have a code style and enforce it. - * Must use static analysis tools where applicable. - * [ROS Core]: +#. **Dependencies:** - * Must use ROS code style and use linters from ``ament_lint_common`` to enforce it - * Must use all linters/static analysis that are part of ``ament_lint_common`` + #. Must not have direct runtime "ROS" dependencies which are not at the same level as the package in question ('Level N'), but... + #. May have optional direct runtime "ROS" dependencies which are not 'Level N', e.g. tracing or debugging features that can be disabled + #. Must have justification for why each direct runtime "non-ROS" dependency is equivalent to a 'Level N' package in terms of quality -* Dependencies: +#. **Platform Support:** - * Must not have direct runtime "ROS" dependencies which are not 'Level 1' dependencies, but... - * May have optional direct runtime "ROS" dependencies which are not 'Level 1', e.g. tracing or debugging features that can be disabled - * Must have justification for why each direct runtime "non-ROS" dependency is equivalent to a 'Level 1' package in terms of quality - -* Platform Support: - - * Must support all tier 1 platforms for ROS 2, as defined in `REP-2000 `_ + #. Must support all tier 1 platforms for ROS 2, as defined in `REP-2000 `_ If the above points are satisfied then a package can be considered 'Level 1'. -Below are some details on the above points. +Refer to the detailed description of the requirements below the chart for more information. + +Quality Level Comparison Chart +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The chart below compares Quality Levels 1 through 5 relative to the 'Level 1' requirements' numbering scheme above. + +✓ = required + +● = recommended + +.. list-table:: Quality Levels + :widths: 5 10 10 10 10 10 + :header-rows: 1 + :stub-columns: 1 + :align: left + + + * - + - Level 1 + - Level 2 + - Level 3 + - Level 4 + - Level 5 + * - 1.i + - ✓ + - ✓ + - ✓ + - ● + - + * - 1.ii + - ✓ + - ✓ + - ✓ + - + - + * - 1.iii + - ✓ + - ✓ + - ● + - + - + * - 1.iv + - ✓ + - ✓ + - ✓ + - + - + * - 1.v + - ✓ + - ✓ + - ✓ + - + - + * - 1.vi + - ✓ + - ✓ + - ● + - + - + * - 2.i + - ✓ + - ✓ + - ✓ + - ● + - + * - 2.ii + - ✓ + - ✓ + - + - + - + * - 2.iii + - ✓ + - + - + - + - + * - 2.iv + - ✓ + - ✓ + - ✓ + - + - + * - 2.v + - ✓ + - + - + - + - + * - 3.i + - ✓ + - ✓ + - + - + - + * - 3.ii + - ✓ + - + - + - + - + * - 3.iii + - ✓ + - ✓ + - ✓ + - ✓ + - + * - 3.iv + - ✓ + - ✓ + - ✓ + - ✓ + - + * - 3.v + - ✓ + - ✓ + - ● + - + - + * - 3.v.a + - ✓ + - ✓ + - ✓ + - + - + * - 3.v.b + - ✓ + - ✓ + - ● + - + - + * - 4.i + - ✓ + - ✓ + - ● + - ● + - + * - 4.ii + - ✓ + - + - + - + - + * - 4.iii.a + - ✓ + - ✓ + - + - + - + * - 4.iii.b + - ✓ + - + - + - + - + * - 4.iv.a + - ✓ + - + - + - + - + * - 4.iv.b + - ✓ + - + - + - + - + * - 4.v.a + - ✓ + - ✓ + - + - + - + * - 4.v.b + - ✓ + - ✓ + - + - + - + * - 5.i + - ✓ + - ✓ + - + - + - + * - 5.ii + - ● + - ● + - ● + - + - + * - 5.iii + - ✓ + - ✓ + - + - + - + * - 6.i + - ✓ + - ✓ + - ✓ + - ● + - Version Policy -"""""""""""""" +^^^^^^^^^^^^^^ The most important thing is to have some version policy which developers may use to anticipate and understand changes to the version of the package. -We recommend the use of ``semver`` as it covers all the important points that a version policy should cover, is well thought out, and is popular in the open source community broadly. +For example, ``semver`` covers all the important points that a version policy should cover, is well thought out, and is popular in the open source community. -The policy should link changes to API and ABI to the version scheme. +The version policy should link public API changes, which may also impact ABI, to changes in the version number. -Additionally, specifically for the ROS ecosystem, the policy should state that API and ABI will be maintained within a stable ROS distribution. -For ``semver``, this means only patch and minor increases only into an existing ROS distribution. +For the ROS ecosystem, the version policy needs to state that API and ABI will be maintained within a stable ROS distribution. +Following ``semver``, this means only patch and minor increases are allowed into an existing ROS distribution. Public API -"""""""""" +~~~~~~~~~~ -The package should also state what the public API includes, and/or state what parts of the API are excluded intentionally. +The package documentation should state what the public API includes, and/or state what parts of the API are excluded intentionally. -For C++, it's somewhat obvious that all installed headers are part of the public API, but it's acceptable to have parts of the accessible API not be stable. +For C++, it's assumed that all installed headers are part of the public API, but it's acceptable to have parts of the accessible API not be stable. For example, having an "experimental" namespace or a "detail" namespace which does not adhere to the API and ABI stability rules is allowed, but they must be clearly documented as such. -Changes to these excluded API's, especially something like a "detail" namespace, should still not break API or ABI for other public API's indirectly. +Changes to these excluded APIs, especially something like a "detail" namespace, should still not break API or ABI for other public APIs indirectly. -For Python, it's more important to explicitly declare which parts of the API is public, because all modules are typically installed and accessible to users. -One easy thing to do is to say all of the API is public and therefore API stable, but "impl" or "detail" namespaces can be used if needed, they just need to be clearly documented as not public and therefore not stable. +For Python, it's more important to explicitly declare which parts of the API are public, because all modules are typically installed and accessible to users. +One easy thing to do is to say all of the API is public and therefore API stable, but ``impl`` or ``detail`` namespaces can be used if needed, they just need to be clearly documented as "not public" and therefore not stable. -There are also other, non-API, things which should be considered and optionally documented as part of the "stable interface" of the package. -This includes, but isn't limited to, message definitions, command line tools (arguments and output format), ROS names (topic, service, node, etc.), and behaviors of the applications. +There are also other, non-API, things which should be considered and documented as part of the "stable interface" of the package. +These could include, but aren't limited to, message definitions, command line tools (arguments and output format), ROS names (topic, service, node, etc.), and behaviors of the applications. -For yet other languages the details will be different, but the important thing is that the public API be obviously documented, and that the public API adheres to an API and ABI stability as described in the version policy, and that they are documented and tested. +For other languages the details will be different, but the important thing is that the public API be obviously documented, and the public API adheres to a documented and tested API and ABI stability policy, as described in the version policy. Feature Documentation -""""""""""""""""""""" +^^^^^^^^^^^^^^^^^^^^^ For each feature provided by the public API of the package, or by a tool in the package, there must be corresponding user documentation. The term "feature", and the scope of the documentation, is intentionally vague because it's difficult to quantitatively measure this metric. -However, the spirit of this requirement is that, for a 'Level 1' quality package, all of the things a user might do with the package needs at least basic documentation or a snippet of code as an example on how to use it. +In general, for a 'Level 1' quality package, all of the things a user might do with the package need at least basic documentation or a snippet of code as an example on how to use it. The `roscpp Overview `_ from the ROS 1 wiki is a good example of this kind of documentation. -Feature Testing and Code Coverage Policy -"""""""""""""""""""""""""""""""""""""""" - -This policy should aim for a "high" coverage standard, but the exact number and rules will vary depending on the package in question. -The policy may be influenced by factors like: +Quality Declaration and Claim +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -- what programming languages are being used, and whether or not there are multiple languages in use -- what coverage information is available (statement vs. line vs. branch vs condition/path coverage) -- what strategy is preferred for dealing with difficult to reach statements/branches +Each package claiming a quality level should include a "quality declaration" document. +This declaration should include a claimed quality level and then should have a section for each of the requirements in that claimed quality level justifying how the package meets each of those requirements. -This StackOverflow question is a good summary of the issues: +Sometimes the justification will be a link to a policy documented in the package itself or it may link to a common policy used by a group of packages. +If there is additional evidence that these policies are being followed, that should be included as well, e.g. a link to the coverage statistics for the package to show that coverage is being tracked and maintained. +Other times, justification will be an explanation as to why a requirement was not met or does not apply, e.g. if performance tests do not make sense for the package in question, it should be satisfactorily explained. -https://stackoverflow.com/questions/90002/what-is-a-reasonable-code-coverage-for-unit-tests-and-why +There is no enforcement or checking of these claims, but instead it's just sufficient to present this information to potential users. +If the users feel that the justifications are insufficient or incorrect, they can open issues against the repository and resolve it with the maintainers. -In particular, this answer does a good job of summarizing the issue: +There should be one or more communal lists of 'Level 1' (and maybe 'Level 2' or 'Level 3') quality level packages. +These lists should be modified via change requests (maybe a text document in a repository) so that there can be peer review. +This REP will not prescribe how or where these lists should be hosted, but one possibility is an informational REP, continually updated and versioned with each new ROS distribution. -https://stackoverflow.com/a/34698711/671658 +Feature Testing and Code Coverage Policy +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Importantly, this answer points out that tracking and enforcing code coverage statistics is strictly empirical (rather than theoretical) and that there are different reasons for using them. -Among those reasons listed is "To satisfy stakeholders", which is the main goal of requiring a code coverage policy for these high quality packages. -It is summarized nicely: +This policy should aim for a "high" coverage standard, but the exact number and rules will vary depending on the package in question. +The policy may be influenced by factors like: - For many projects, there are various actors who have an interest in software quality who may not be involved in the day-to-day development of the software (managers, technical leads, etc.) - Saying "we're going to write all the tests we really need" is not convincing: - They either need to trust entirely, or verify with ongoing close oversight (assuming they even have the technical understanding to do so.) - Providing measurable standards and explaining how they reasonably approximate actual goals is better. +- what programming languages are being used, and whether or not there are multiple languages in use +- what coverage information is available (statement vs. line vs. branch vs condition/path coverage) +- what strategy is preferred for dealing with difficult-to-reach statements/branches -The other two reasons "To normalize team behavior" and "To keep yourself honest" are nice reasons to have code coverage goals, but are out of scope for this document. +Tracking and enforcing code coverage statistics is strictly empirical and there are different reasons for using them. +Among those reasons is satisfying stakeholders [2]_, which is the main goal of requiring a code coverage policy for high quality packages. +A set of measurable standards and explanations of the goals they equate to is likely the most convincing method of proving to stakeholders that the package is properly tested. The general recommendation is to have at least line coverage and aim to achieve and maintain a high percentage of coverage (e.g. above 90%). -This at least gives you and your stakeholders some confidence that all feature have basic tests. -Any assurances beyond that would require branch coverage statistics and independent investigation of the tests and how they test the code. +This at least gives you and your stakeholders some confidence that all features have basic tests. +Any assurances beyond that would require branch coverage statistics and independent investigation of the tests. Performance Testing -""""""""""""""""""" +^^^^^^^^^^^^^^^^^^^ There are some cases where performance testing does not make sense to have. For example, it may be a good idea to have performance tests for a code generator (like ``rosidl_generator_cpp``), but it is not strictly required since its performance does not affect a runtime production system, and so in that case the package could claim to be 'Level 1' without performance tests if properly justified in the "quality declaration". -However, if performance is a reasonable concern for use in a production system, then there must be performance tests and they should be used in conjunction with a regression policy which aims to prevent new versions of the package to be considerably slower without cause. -Note, the performance regression policy should not prevent regressions, but instead should aim to detect them and either address them directly, plan to address them in the future, or when unavoidable (e.g. fixing a bug required more resources to be safe) explain why the regression has occurred in the memorandum of the change request that introduced it. +However, if performance is a reasonable concern for use in a production system, then there must be performance tests and they should be used in conjunction with a regression policy which aims to prevent unintended performance degradation. +Note, the performance regression policy should not prevent regressions, but instead should aim to detect them and either address them directly, plan to address them in the future, or, when unavoidable (e.g. fixing a bug required more resources to be safe), explain why the regression has occurred in the memorandum of the change request that introduced it. Dependencies -"""""""""""" +^^^^^^^^^^^^ -Each package should examine their direct runtime dependencies for their quality levels. +Each package should examine the quality levels of their direct runtime dependencies. Packages should not claim a quality level higher than their dependencies, unless it can be reasonably explained why they do not affect the quality of the package in question. -An example of this would be build or "build tool" dependencies, which are only used during build time and do not impact the runtime quality of the package. -This would not include, however, build dependencies which, for example, contribute only headers to a C++ library or a static library, as the quality of those headers or static library also impact the quality of the runtime product directly. -This would include, for another example, something like CMake, which in most ways does not impact the quality of the product. - -There's obviously a lot of ambiguity in this area, as you could argue for or against a variety of dependencies and how they impact the package. -However, the point is to require the maintainers of the package to examine each dependency, justify why they do or do not impact the quality, and document that so that peer reviewers and consumers of the package can make their own evaluation. +An example of an exception would be build or "build tool" dependencies, which are only used during build time and do not affect the runtime quality of the package. +This would not include, however, build dependencies which, for example, contribute only headers to a C++ library or a static library, as the quality of those headers or static library also affect the quality of the runtime product directly. +Conversely, this would include something like CMake, which in most ways does not affect the quality of the product. Dependencies which are other "ROS" packages should have these quality standards applied to them and should meet or exceed the quality level claimed by the package in question. - Dependencies which are not other "ROS" packages should be individually examined for quality. You may either try to apply the requirements for the quality levels described here, or you may wish to simply argue the quality without using these requirements as a ruler. -In either case, for each direct "non-ROS" dependency your "quality declaration" should include a justification as to why it is acceptable to depend on this software and still claim your package's level of quality. -This may simply be text justification, or it may link to other analysis or discussions had by community members rationalizing the choice. -The important point is that each dependency is considered, justified, and that the justification is documented, so that users of the package can read the justification and decide for themselves if it is acceptable or not. - -Any important caveats or justified exceptions for your dependencies should be mentioned (or referenced) in your own package's "quality declaration" document. - -For example, if your package depends on ``rclcpp``, and ``rclcpp`` claims 'level 1' quality with the caveat that this requires you use an rmw implementation that also meets the 'level 1' quality standard, then your package's "quality declaration" document should mention this as well. -Perhaps just saying that one of your dependencies, ``rclcpp``, has some caveats and then link to ``rclcpp``'s own "quality declaration". - -In this way, caveats and justifications that may be important for peer reviewers and consumers of your package to understand can "bubble up" from any part of the system. - -The goal here is for the maintainer of a package to "make the case" to potential users or stakeholders that their dependencies are at least as high quality as the package in question, and to make a best effort attempt to make them aware of any issues or caveats. -It's up to those users and stakeholders to evaluate that justification and to look at the dependencies themselves as well. - -Claiming a Quality Level and Documenting Package Policies -""""""""""""""""""""""""""""""""""""""""""""""""""""""""" - -Each package claiming a quality level should have a "quality declaration" documented somewhere. -This declaration should include a claimed quality level and then should have a section for each of the requirements in that claimed quality level justifying how the package meets each of those requirements. - -Sometimes the justification will be a link to a policy documented in the package itself or it may link to a common policy used by a group of packages. -If there is additional evidence that these policies are being followed, that should be included as well, e.g. a link to the coverage statistics for the package to show that coverage is being tracked and maintained. -Other times, justification will be an explanation as to why a requirement was not met or does not apply, e.g. if performance tests do not make sense for the package in question, it should be satisfactorily explained. +The argument could be text justification, or it could link to other analyses or discussions had by community members rationalizing the choice. -There is no enforcement or checking of these claims, but instead it's just sufficient to present this information to potential users. -If the users feel that the justifications are insufficient or incorrect, they can open issues against the repository and resolve it with the maintainers. +For example, if your package depends on ``rclcpp``, and ``rclcpp`` claims 'Level 1' quality with the caveat that this requires you use an rmw implementation that also meets the 'Level 1' quality standard, then your package's "quality declaration" document should mention this as well. +This could be accomplished simply by saying that one of your dependencies, ``rclcpp``, has some caveats and then linking to ``rclcpp``'s own "quality declaration". -There should be one or more communal lists of 'Level 1' (and maybe 'Level 2' or 'Level 3') quality level packages. -These lists should be modified via change requests (maybe a text document in a repository) so that there can be peer review. -This document will not prescribe how or where these lists should be hosted, but one thought is that the list could live on the main ROS 2 documentation website. +There's obviously a lot of ambiguity in this area, as you could argue for or against a variety of dependencies and how they affect the quality of a package. +The goal is for the maintainers of a package to "make the case" that their dependencies are at least as high quality as the package in question. +They should examine each dependency, and document any important caveats or justified exceptions in the package's "quality declaration" document so peer reviewers and consumers of the package can make their own evaluations. Quality Level 2 ^^^^^^^^^^^^^^^ @@ -300,71 +454,52 @@ This can also include packages which are not yet up to 'Level 1' but intend to b Package Requirements ~~~~~~~~~~~~~~~~~~~~ -*Note: bullets below that start with [ROS Core], will be the prescription for what we do in the core packages in order to meet the associated requirements* - Requirements to be considered a 'Level 2' package: -* Version Policy: - - * The same as 'Level 1' packages - -* Change Control Process: - - * Must have all code changes occur through a change request (e.g. pull request, merge request, etc.) - * Must have Continuous Integration (CI) policy for all change requests - * [ROS Core]: +#. **Version Policy:** - * All changes will go through a pull request - * All pull requests will be tested via CI + #. The same as 'Level 1' packages -* Documentation: +#. **Change Control Process:** - * Must have documentation for each "feature" (e.g. for ``rclcpp``: create a node, publish a message, spin, etc.) - * Must have a declared license or set of licenses - * Must have a copyright statement in each source file - * Must have a "quality declaration" document, which declares the quality level and justifies how the package meets each of the requirements + #. Must have all code changes occur through a change request (e.g. pull request, merge request, etc.) + #. Must have confirmation of contributor origin (e.g. `DCO `_, CLA, etc.) + #. Must have Continuous Integration (CI) policy for all change requests - * Must have a section in the repository's ``README`` which contains the "quality declaration" or links to it - * Must register with a centralized list of 'Level 2' packages, if one exists, to allow for peer review of the claim +#. **Documentation:** - * [ROS Core]: + #. Must have documentation for each "feature" (e.g. for ``rclcpp``: create a node, publish a message, spin, etc.) + #. Must have a declared license or set of licenses + #. Must have a copyright statement in each source file + #. Must have a "quality declaration" document, which declares the quality level and justifies how the package meets each of the requirements - * Must have automated checks for copyright statements and licenses - * Must use the Apache 2.0 license, unless the package has an existing permissive license (e.g. rviz uses three-clause BSD) + #. *Must have a section in the repository's ``README`` which contains the "quality declaration" or links to it* + #. *Must register with a centralized list of 'Level 2' packages, if one exists, to allow for peer review of the claim* -* Testing: +#. **Testing:** - * Must have system tests which cover all items in the "feature" documentation - * Code coverage: + #. Must have system tests which cover all items in the "feature" documentation + #. Code Coverage: - * Must have code coverage tracking for the package - * [ROS Core]: + #. *Must have code coverage tracking for the package* - * Must provide line coverage statistics - * May provide branch coverage - * May exclude code from coverage (test code, debug code, etc.) + #. Linters and Static Analysis - * Linters and Static Analysis + #. *Must have a code style and enforce it* + #. *Must use static analysis tools where applicable* - * Must have a code style and enforce it. - * Must use static analysis tools where applicable. - * [ROS Core]: +#. **Dependencies:** - * Must use ROS code style and use linters from ``ament_lint_common`` to enforce it - * Must use all linters/static analysis that are part of ``ament_lint_common`` + #. Must not have direct runtime "ROS" dependencies which are not 'Level 2' dependencies, but... + #. May have optional direct runtime "ROS" dependencies which are not 'Level 2', e.g. tracing or debugging features that can be disabled + #. Must have justification for why each direct runtime "non-ROS" dependency is equivalent to a 'Level 2' package in terms of quality -* Dependencies: +#. **Platform Support:** - * Must not have direct runtime "ROS" dependencies which are not 'Level 2' dependencies, but... - * May have optional direct runtime "ROS" dependencies which are not 'Level 2', e.g. tracing or debugging features that can be disabled - * Must have justification for why each direct runtime "non-ROS" dependency is equivalent to a 'Level 2' package in terms of quality - -* Platform Support: - - * Must support all tier 1 platforms for ROS 2, as defined in `REP-2000 `_ + #. Must support all tier 1 platforms for ROS 2, as defined in `REP-2000 `_ If the above points are satisfied then a package can be considered 'Level 2'. -Refer to the detailed description of the requirements in the Quality Level 1 section above for more information. +Refer to the detailed description of the requirements following the Quality Level 1 section above for more information. Quality Level 3 ^^^^^^^^^^^^^^^ @@ -375,54 +510,43 @@ These packages are more lax on documentation, testing, and scope of public API's Package Requirements ~~~~~~~~~~~~~~~~~~~~ -*Note: bullets below that start with [ROS Core], will be the prescription for what we do in the core packages in order to meet the associated requirements* - Requirements to be considered a 'Level 3' package: -* Version Policy: - - * The same as 'Level 1' packages, except: - - * No public API needs to be explicitly declared, though this can make it harder to maintain API and ABI stability - * No requirement to keep API/ABI stability within a stable ROS release, but it is recommended still +#. **Version Policy:** -* Change Control Process: + #. The same as 'Level 1' packages, except: - * Must have all code changes occur through a change request (e.g. pull request, merge request, etc.) - * Must have Continuous Integration (CI) policy for all change requests - * [ROS Core]: + #. *No public API needs to be explicitly declared, though this can make it harder to maintain API and ABI stability* + #. *No requirement to keep API/ABI stability within a stable ROS release, but it is still recommended* - * All changes will go through a pull request - * All pull requests will be tested via CI +#. **Change Control Process:** -* Documentation: + #. Must have all code changes occur through a change request (e.g. pull request, merge request, etc.) + #. Must have Continuous Integration (CI) policy for all change requests - * Must have a declared license or set of licenses - * Must have a copyright statement in each source file - * May have a "quality declaration" document, which declares the quality level and justifies how the package meets each of the requirements +#. **Documentation:** - * Must have a section in the repository's ``README`` which contains the "quality declaration" or links to it - * May register with a centralized list of 'Level 3' packages, if one exists, to allow for peer review of the claim + #. Must have a declared license or set of licenses + #. Must have a copyright statement in each source file + #. May have a "quality declaration" document, which declares the quality level and justifies how the package meets each of the requirements - * [ROS Core]: + #. *Must have a section in the repository's ``README`` which contains the "quality declaration" or links to it* + #. *May register with a centralized list of 'Level 3' packages, if one exists, to allow for peer review of the claim* - * Must have automated checks for copyright statements and licenses - * Must use the Apache 2.0 license, unless the package has an existing permissive license (e.g. rviz uses three-clause BSD) +#. **Testing:** -* Testing: + #. No explicit testing requirements, though covering some if not all of the features with tests is recommended - * No explicit testing requirements, though covering some if not all of the features with tests is recommended +#. **Dependencies:** -* Dependencies: + #. May have direct runtime "ROS" dependencies which are not 'Level 3' dependencies, but they should still be documented in the quality declaration - * May have direct runtime "ROS" dependencies which are not 'Level 3' dependencies, but they should be documented +#. **Platform Support:** -* Platform Support: - - * Must support all tier 1 platforms for ROS 2, as defined in `REP-2000 `_ + #. Must support all tier 1 platforms for ROS 2, as defined in `REP-2000 `_ If the above points are satisfied then a package can be considered 'Level 3'. -Refer to the detailed description of the requirements in the Quality Level 1 section above for more information. +Refer to the detailed description of the requirements following the Quality Level 1 section above for more information. Quality Level 4 ^^^^^^^^^^^^^^^ @@ -434,79 +558,108 @@ For example, this might be a tutorial package which is not intended for reuse bu Package Requirements ~~~~~~~~~~~~~~~~~~~~ -*Note: bullets below that start with [ROS Core], will be the prescription for what we do in the core packages in order to meet the associated requirements* - Requirements to be considered a 'Level 4' package: -* Version Policy: - - * No requirements, but having a policy is still recommended (e.g. ``semver``), even if the version is not yet stable (e.g. >= 1.0.0 for ``semver``) +#. **Version Policy:** -* Change Control Process: + #. No requirements, but having a policy is still recommended (e.g. ``semver``), even if the version is not yet stable (e.g. >= 1.0.0 for ``semver``) - * No explicit change control process required, but still recommended +#. **Change Control Process:** -* Documentation: + #. No explicit change control process required, but still recommended - * Must have a declared license or set of licenses - * Must have a copyright statement in each source file - * [ROS Core]: +#. **Documentation:** - * Must have automated checks for copyright statements and licenses - * Must use the Apache 2.0 license, unless the package has an existing permissive license (e.g. rviz uses three-clause BSD) + #. Must have a declared license or set of licenses + #. Must have a copyright statement in each source file -* Testing: +#. **Testing:** - * No explicit testing requirements, though covering some if not all of the features with tests is recommended + #. No explicit testing requirements, though covering some if not all of the features with tests is recommended -* Dependencies: +#. **Dependencies:** - * No restrictions + #. No restrictions -* Platform Support: +#. **Platform Support:** - * May support all tier 1 platforms for ROS 2, as defined in `REP-2000 `_ + #. May support all tier 1 platforms for ROS 2, as defined in `REP-2000 `_ Any package that does not claim to be 'Level 3' or higher is automatically 'Level 4'. -Refer to the detailed description of the requirements in the Quality Level 1 section above for more information. +Refer to the detailed description of the requirements following the Quality Level 1 section above for more information. Quality Level 5 ^^^^^^^^^^^^^^^ -Packages in this category simply do not meet even the 'Level 4' requirements, and for that reason should not be used. -The rationale being that all packages should have at least a declare license or licenses and should include copyright statements in each file. +Packages in this category cannot even meet the simple 'Level 4' requirements, and for that reason should not be used. +All packages should have at least a declared license or set of licenses and should include copyright statements in each file. Repository Organization ^^^^^^^^^^^^^^^^^^^^^^^ Since these categories are applied on a per package basis, and since there may be more than one package per source repository, it's recommended that the strictest set of policies apply to the whole repository. -This is recommended, rather than trying to mix processes depending on which packages are changed in a given change request (pull request or merge request, etc.). +This is recommended rather than trying to mix processes depending on which packages are changed in a given change request (pull request or merge request, etc.). If this is too onerous, then it's recommended to split lower quality packages out into a separate repository. +Reference Implementation +======================== + +The `ROS 2 Developer Guide `_ describes the policies we implement to achieve Quality Level 1 for ROS Core packages. -Motivation -========== +The `rcutils package's quality declaration `_ is one example of the conditions of this REP in practice on a non-trivial package. -TBD +.. update link when that draft is merged +Quality Declaration Template +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Rationale -========= +.. code-block:: markdown -TBD + This document is a declaration of software quality for the `` package, based on the guidelines in [REP-2004](https://www.ros.org/reps/rep-2004.html). + # `` Quality Declaration -Backwards Compatibility -======================= + The package `` claims to be in the **Quality Level N** category. -Since there are no preceeding standards in this space, at least in the ROS community, of which we are aware, there are no backwards compatibility concerns. + Below are the rationales, notes, and caveats for this claim, organized by each requirement listed in the [Package Requirements for Quality Level N in REP-2004](https://www.ros.org/reps/rep-2004.html). + ## Version Policy -Reference Implementation -======================== + ### Version Scheme + + ### API Stability Within a Released ROS Distribution + + ### ABI Stability Within a Released ROS Distribution + + ### Public API Declaration + + ## Change Control Process + + ## Documentation + + ### Feature Documentation + + ### Public API Documentation + + ### License + + ### Copyright Statements + + ## Testing + + ### Feature Testing + + ### Public API Testing + + ### Coverage + + ### Performance + + ### Linters and Static Analysis -TBD: Link to "ROS Core" perscription for how to actually achieve each quality level. + ## Dependencies + ## Platform Support References and Footnotes ======================== @@ -514,6 +667,8 @@ References and Footnotes .. [1] Initial discussions about this REP: https://github.com/ros2/ros2_documentation/pull/460 +.. [2] What is a reasonable code coverage % for unit tests (and why)? + https://stackoverflow.com/a/34698711/671658 Copyright ========= From d82c4b2ddac6721fa87e17dcb9e2ae1327de2d80 Mon Sep 17 00:00:00 2001 From: William Woodall Date: Thu, 12 Mar 2020 12:17:42 -0700 Subject: [PATCH 05/14] mention level 5 in specification intro; fix typos Signed-off-by: William Woodall --- rep-2004.rst | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/rep-2004.rst b/rep-2004.rst index 9f2d2259c..c91bcf2a2 100644 --- a/rep-2004.rst +++ b/rep-2004.rst @@ -29,7 +29,7 @@ By providing these rough goals for package maintainers to strive towards, the ca Specification ============= -There are four quality levels below, each roughly summarized as: +There are five quality levels below, each roughly summarized as: * Quality Level 1: @@ -56,6 +56,11 @@ There are four quality levels below, each roughly summarized as: * demos, tutorials, and experiments * e.g. research packages, ``demo_nodes_cpp``, ``examples_rclcpp_minimal_publisher``, etc. +* Quality Level 5: + + * default quality level + * e.g. for code that fails to meet requirements in 'Level 4' + While each quality level will have different requirements, it's always possible to overachieve in certain requirements even if other requirements prevent a package from moving up to the next quality level. Quality Level 1 @@ -122,7 +127,7 @@ Requirements to be considered a 'Level 1' package: #. *Must have performance tests (exceptions allowed if they don't make sense to have)* #. *Must have a performance regression policy (i.e. blocking either changes or releases on unexpected performance regressions)* - #. Linters and Static Analysis + #. Linters and Static Analysis: #. *Must have a code style and enforce it* #. *Must use static analysis tools where applicable* From 8dce8a6f46890210d844a39474e6f18ecef95e3c Mon Sep 17 00:00:00 2001 From: Marya Belanger Date: Tue, 24 Mar 2020 15:05:56 -0400 Subject: [PATCH 06/14] Review revisions rep-2004 (#243) Signed-off-by: maryaB-osr --- rep-2004.rst | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/rep-2004.rst b/rep-2004.rst index c91bcf2a2..02f5e5c72 100644 --- a/rep-2004.rst +++ b/rep-2004.rst @@ -140,7 +140,9 @@ Requirements to be considered a 'Level 1' package: #. **Platform Support:** - #. Must support all tier 1 platforms for ROS 2, as defined in `REP-2000 `_ + #. Must support all target platforms for the package's ecosystem. + + * For ROS 2 this means supporting all tier 1 platforms, as defined in `REP-2000 `_ If the above points are satisfied then a package can be considered 'Level 1'. Refer to the detailed description of the requirements below the chart for more information. @@ -250,7 +252,7 @@ The chart below compares Quality Levels 1 through 5 relative to the 'Level 1' re - ✓ - ✓ - ✓ - - + - ● * - 3.iv - ✓ - ✓ @@ -501,7 +503,9 @@ Requirements to be considered a 'Level 2' package: #. **Platform Support:** - #. Must support all tier 1 platforms for ROS 2, as defined in `REP-2000 `_ + #. Must support all target platforms for the package's ecosystem. + + * For ROS 2 this means supporting all tier 1 platforms, as defined in `REP-2000 `_ If the above points are satisfied then a package can be considered 'Level 2'. Refer to the detailed description of the requirements following the Quality Level 1 section above for more information. @@ -548,7 +552,9 @@ Requirements to be considered a 'Level 3' package: #. **Platform Support:** - #. Must support all tier 1 platforms for ROS 2, as defined in `REP-2000 `_ + #. Must support all target platforms for the package's ecosystem. + + * For ROS 2 this means supporting all tier 1 platforms, as defined in `REP-2000 `_ If the above points are satisfied then a package can be considered 'Level 3'. Refer to the detailed description of the requirements following the Quality Level 1 section above for more information. @@ -588,7 +594,9 @@ Requirements to be considered a 'Level 4' package: #. **Platform Support:** - #. May support all tier 1 platforms for ROS 2, as defined in `REP-2000 `_ + #. Must support all target platforms for the package's ecosystem. + + * For ROS 2 this means supporting all tier 1 platforms, as defined in `REP-2000 `_ Any package that does not claim to be 'Level 3' or higher is automatically 'Level 4'. Refer to the detailed description of the requirements following the Quality Level 1 section above for more information. @@ -597,7 +605,7 @@ Quality Level 5 ^^^^^^^^^^^^^^^ Packages in this category cannot even meet the simple 'Level 4' requirements, and for that reason should not be used. -All packages should have at least a declared license or set of licenses and should include copyright statements in each file. +'Level 5' has no requirements, but we still strongly recommend all packages have a license and copyright statement, ideally in each file. Repository Organization ^^^^^^^^^^^^^^^^^^^^^^^ From e421be8db3ef6fa65b29f9325ce635afdaecbf16 Mon Sep 17 00:00:00 2001 From: Marya Belanger Date: Wed, 25 Mar 2020 18:57:40 -0400 Subject: [PATCH 07/14] Add item numbers to quality declaration template (#244) * Add item numbers to quality declaration template Signed-off-by: maryaB-osr * list all requirements Signed-off-by: maryaB-osr * link to requirements in chart Signed-off-by: maryaB-osr * fix list indentation Signed-off-by: maryaB-osr --- rep-2004.rst | 413 +++++++++++++++++++++++++++++++++------------------ 1 file changed, 266 insertions(+), 147 deletions(-) diff --git a/rep-2004.rst b/rep-2004.rst index 02f5e5c72..7e89d5b65 100644 --- a/rep-2004.rst +++ b/rep-2004.rst @@ -85,62 +85,161 @@ Package Requirements Requirements to be considered a 'Level 1' package: -#. **Version Policy:** +.. _Version Policy: - #. Must have a version policy (e.g. ``semver``) - #. Must be at a stable version (e.g. for ``semver`` that means version >= 1.0.0) - #. Must have a strictly declared public API - #. Must have a policy for API stability - #. Must have a policy for ABI stability - #. Must have a policy that keeps API and ABI stability within a released ROS distribution +1. **Version Policy:** -#. **Change Control Process:** + .. _1.i: - #. Must have all code changes occur through a change request (e.g. pull request, merge request, etc.) - #. Must have confirmation of contributor origin (e.g. `DCO `_, CLA, etc.) - #. Must have peer review policy for all change requests (e.g. require one or more reviewers) - #. Must have Continuous Integration (CI) policy for all change requests - #. Must have documentation policy for all change requests + i. Must have a version policy (e.g. ``semver``) -#. **Documentation:** + .. _1.ii: - #. Must have documentation for each "feature" (e.g. for ``rclcpp``: create a node, publish a message, spin, etc.) - #. Must have documentation for each item in the public API (e.g. functions, classes, etc.) - #. Must have a declared license or set of licenses - #. Must have a copyright statement in each source file - #. Must have a "quality declaration" document, which declares the quality level and justifies how the package meets each of the requirements + ii. Must be at a stable version (e.g. for ``semver`` that means version >= 1.0.0) - #. *Must have a section in the repository's ``README`` which contains the "quality declaration" or links to it* - #. *Must register with a centralized list of 'Level N' packages, if one exists, to allow for peer review of the claim* + .. _1.iii: -#. **Testing:** + iii. Must have a strictly declared public API - #. Must have system tests which cover all items in the "feature" documentation - #. Must have system, integration, and/or unit tests which cover all of the public API - #. Code Coverage: + .. _1.iv: - #. *Must have code coverage tracking for the package* - #. *Must have and enforce a code coverage policy for new changes* + iv. Must have a policy for API stability - #. Performance: + .. _1.v: - #. *Must have performance tests (exceptions allowed if they don't make sense to have)* - #. *Must have a performance regression policy (i.e. blocking either changes or releases on unexpected performance regressions)* + v. Must have a policy for ABI stability - #. Linters and Static Analysis: + .. _1.vi: - #. *Must have a code style and enforce it* - #. *Must use static analysis tools where applicable* + vi. Must have a policy that keeps API and ABI stability within a released ROS distribution -#. **Dependencies:** +.. _Change Control Process: - #. Must not have direct runtime "ROS" dependencies which are not at the same level as the package in question ('Level N'), but... - #. May have optional direct runtime "ROS" dependencies which are not 'Level N', e.g. tracing or debugging features that can be disabled - #. Must have justification for why each direct runtime "non-ROS" dependency is equivalent to a 'Level N' package in terms of quality +2. **Change Control Process:** -#. **Platform Support:** + .. _2.i: - #. Must support all target platforms for the package's ecosystem. + i. Must have all code changes occur through a change request (e.g. pull request, merge request, etc.) + + .. _2.ii: + + ii. Must have confirmation of contributor origin (e.g. `DCO `_, CLA, etc.) + + .. _2.iii: + + iii. Must have peer review policy for all change requests (e.g. require one or more reviewers) + + .. _2.iv: + + iv. Must have Continuous Integration (CI) policy for all change requests + + .. _2.v: + + v. Must have documentation policy for all change requests + +.. _Documentation: + +3. **Documentation:** + + .. _3.i: + + i. Must have documentation for each "feature" (e.g. for ``rclcpp``: create a node, publish a message, spin, etc.) + + .. _3.ii: + + ii. Must have documentation for each item in the public API (e.g. functions, classes, etc.) + + .. _3.iii: + + iii. Must have a declared license or set of licenses + + .. _3.iv: + + iv. Must have a copyright statement in each source file + + .. _3.v: + + v. Must have a "quality declaration" document, which declares the quality level and justifies how the package meets each of the requirements + + .. _3.v.a: + + a. *Must have a section in the repository's ``README`` which contains the "quality declaration" or links to it* + + .. _3.v.b: + + b. *Must register with a centralized list of 'Level N' packages, if one exists, to allow for peer review of the claim* + +.. _Testing: + +4. **Testing:** + + .. _4.i: + + i. Must have system tests which cover all items in the "feature" documentation + + .. _4.ii: + + ii. Must have system, integration, and/or unit tests which cover all of the public API + + .. _4.iii: + + iii. Code Coverage: + + .. _4.iii.a: + + a. *Must have code coverage tracking for the package* + + .. _4.iii.b: + + b. *Must have and enforce a code coverage policy for new changes* + + .. _4.iv: + + iv. Performance: + + .. _4.iv.a: + + a. *Must have performance tests (exceptions allowed if they don't make sense to have)* + + .. _4.iv.b: + + b. *Must have a performance regression policy (i.e. blocking either changes or releases on unexpected performance regressions)* + + .. _4.v: + + v. Linters and Static Analysis: + + .. _4.v.a: + + a. *Must have a code style and enforce it* + + .. _4.v.b: + + b. *Must use static analysis tools where applicable* + +.. _Dependencies: + +5. **Dependencies:** + + .. _5.i: + + i. Must not have direct runtime "ROS" dependencies which are not at the same level as the package in question ('Level N'), but... + + .. _5.ii: + + ii. May have optional direct runtime "ROS" dependencies which are not 'Level N', e.g. tracing or debugging features that can be disabled + + .. _5.iii: + + iii. Must have justification for why each direct runtime "non-ROS" dependency is equivalent to a 'Level N' package in terms of quality + +.. _Platform Support: + +6. **Platform Support:** + + .. _6.i: + + i. Must support all target platforms for the package's ecosystem. * For ROS 2 this means supporting all tier 1 platforms, as defined in `REP-2000 `_ @@ -169,181 +268,181 @@ The chart below compares Quality Levels 1 through 5 relative to the 'Level 1' re - Level 3 - Level 4 - Level 5 - * - 1.i + * - 1.i_ - ✓ - ✓ - ✓ - ● - - * - 1.ii + * - 1.ii_ - ✓ - ✓ - ✓ - - - * - 1.iii + * - 1.iii_ - ✓ - ✓ - ● - - - * - 1.iv + * - 1.iv_ - ✓ - ✓ - ✓ - - - * - 1.v + * - 1.v_ - ✓ - ✓ - ✓ - - - * - 1.vi + * - 1.vi_ - ✓ - ✓ - ● - - - * - 2.i + * - 2.i_ - ✓ - ✓ - ✓ - ● - - * - 2.ii + * - 2.ii_ - ✓ - ✓ - - - - * - 2.iii + * - 2.iii_ - ✓ - - - - - * - 2.iv + * - 2.iv_ - ✓ - ✓ - ✓ - - - * - 2.v + * - 2.v_ - ✓ - - - - - * - 3.i + * - 3.i_ - ✓ - ✓ - - - - * - 3.ii + * - 3.ii_ - ✓ - - - - - * - 3.iii + * - 3.iii_ - ✓ - ✓ - ✓ - ✓ - ● - * - 3.iv + * - 3.iv_ - ✓ - ✓ - ✓ - ✓ - - * - 3.v + * - 3.v_ - ✓ - ✓ - ● - - - * - 3.v.a + * - 3.v.a_ - ✓ - ✓ - ✓ - - - * - 3.v.b + * - 3.v.b_ - ✓ - ✓ - ● - - - * - 4.i + * - 4.i_ - ✓ - ✓ - ● - ● - - * - 4.ii + * - 4.ii_ - ✓ - - - - - * - 4.iii.a + * - 4.iii.a_ - ✓ - ✓ - - - - * - 4.iii.b + * - 4.iii.b_ - ✓ - - - - - * - 4.iv.a + * - 4.iv.a_ - ✓ - - - - - * - 4.iv.b + * - 4.iv.b_ - ✓ - - - - - * - 4.v.a + * - 4.v.a_ - ✓ - ✓ - - - - * - 4.v.b + * - 4.v.b_ - ✓ - ✓ - - - - * - 5.i + * - 5.i_ - ✓ - ✓ - - - - * - 5.ii + * - 5.ii_ - ● - ● - ● - - - * - 5.iii + * - 5.iii_ - ✓ - ✓ - - - - * - 6.i + * - 6.i_ - ✓ - ✓ - ✓ @@ -463,47 +562,47 @@ Package Requirements Requirements to be considered a 'Level 2' package: -#. **Version Policy:** +1. **Version Policy:** - #. The same as 'Level 1' packages + i. The same as 'Level 1' packages -#. **Change Control Process:** +2. **Change Control Process:** - #. Must have all code changes occur through a change request (e.g. pull request, merge request, etc.) - #. Must have confirmation of contributor origin (e.g. `DCO `_, CLA, etc.) - #. Must have Continuous Integration (CI) policy for all change requests + i. Must have all code changes occur through a change request (e.g. pull request, merge request, etc.) + ii. Must have confirmation of contributor origin (e.g. `DCO `_, CLA, etc.) + iii. Must have Continuous Integration (CI) policy for all change requests -#. **Documentation:** +3. **Documentation:** - #. Must have documentation for each "feature" (e.g. for ``rclcpp``: create a node, publish a message, spin, etc.) - #. Must have a declared license or set of licenses - #. Must have a copyright statement in each source file - #. Must have a "quality declaration" document, which declares the quality level and justifies how the package meets each of the requirements + i. Must have documentation for each "feature" (e.g. for ``rclcpp``: create a node, publish a message, spin, etc.) + ii. Must have a declared license or set of licenses + iii. Must have a copyright statement in each source file + iv. Must have a "quality declaration" document, which declares the quality level and justifies how the package meets each of the requirements - #. *Must have a section in the repository's ``README`` which contains the "quality declaration" or links to it* - #. *Must register with a centralized list of 'Level 2' packages, if one exists, to allow for peer review of the claim* + a. *Must have a section in the repository's ``README`` which contains the "quality declaration" or links to it* + b. *Must register with a centralized list of 'Level 2' packages, if one exists, to allow for peer review of the claim* -#. **Testing:** +4. **Testing:** - #. Must have system tests which cover all items in the "feature" documentation - #. Code Coverage: + i. Must have system tests which cover all items in the "feature" documentation + ii. Code Coverage: - #. *Must have code coverage tracking for the package* + a. *Must have code coverage tracking for the package* - #. Linters and Static Analysis + iii. Linters and Static Analysis - #. *Must have a code style and enforce it* - #. *Must use static analysis tools where applicable* + a. *Must have a code style and enforce it* + b. *Must use static analysis tools where applicable* -#. **Dependencies:** +5. **Dependencies:** - #. Must not have direct runtime "ROS" dependencies which are not 'Level 2' dependencies, but... - #. May have optional direct runtime "ROS" dependencies which are not 'Level 2', e.g. tracing or debugging features that can be disabled - #. Must have justification for why each direct runtime "non-ROS" dependency is equivalent to a 'Level 2' package in terms of quality + i. Must not have direct runtime "ROS" dependencies which are not 'Level 2' dependencies, but... + ii. May have optional direct runtime "ROS" dependencies which are not 'Level 2', e.g. tracing or debugging features that can be disabled + iii. Must have justification for why each direct runtime "non-ROS" dependency is equivalent to a 'Level 2' package in terms of quality -#. **Platform Support:** +6. **Platform Support:** - #. Must support all target platforms for the package's ecosystem. + i. Must support all target platforms for the package's ecosystem. * For ROS 2 this means supporting all tier 1 platforms, as defined in `REP-2000 `_ @@ -521,38 +620,38 @@ Package Requirements Requirements to be considered a 'Level 3' package: -#. **Version Policy:** +1. **Version Policy:** - #. The same as 'Level 1' packages, except: + i. The same as 'Level 1' packages, except: - #. *No public API needs to be explicitly declared, though this can make it harder to maintain API and ABI stability* - #. *No requirement to keep API/ABI stability within a stable ROS release, but it is still recommended* + a. *No public API needs to be explicitly declared, though this can make it harder to maintain API and ABI stability* + b. *No requirement to keep API/ABI stability within a stable ROS release, but it is still recommended* -#. **Change Control Process:** +2. **Change Control Process:** - #. Must have all code changes occur through a change request (e.g. pull request, merge request, etc.) - #. Must have Continuous Integration (CI) policy for all change requests + i. Must have all code changes occur through a change request (e.g. pull request, merge request, etc.) + ii. Must have Continuous Integration (CI) policy for all change requests -#. **Documentation:** +3. **Documentation:** - #. Must have a declared license or set of licenses - #. Must have a copyright statement in each source file - #. May have a "quality declaration" document, which declares the quality level and justifies how the package meets each of the requirements + i. Must have a declared license or set of licenses + ii. Must have a copyright statement in each source file + iii. May have a "quality declaration" document, which declares the quality level and justifies how the package meets each of the requirements - #. *Must have a section in the repository's ``README`` which contains the "quality declaration" or links to it* - #. *May register with a centralized list of 'Level 3' packages, if one exists, to allow for peer review of the claim* + a. *Must have a section in the repository's ``README`` which contains the "quality declaration" or links to it* + b. *May register with a centralized list of 'Level 3' packages, if one exists, to allow for peer review of the claim* -#. **Testing:** +4. **Testing:** - #. No explicit testing requirements, though covering some if not all of the features with tests is recommended + i. No explicit testing requirements, though covering some if not all of the features with tests is recommended -#. **Dependencies:** +5. **Dependencies:** - #. May have direct runtime "ROS" dependencies which are not 'Level 3' dependencies, but they should still be documented in the quality declaration + i. May have direct runtime "ROS" dependencies which are not 'Level 3' dependencies, but they should still be documented in the quality declaration -#. **Platform Support:** +6. **Platform Support:** - #. Must support all target platforms for the package's ecosystem. + i. Must support all target platforms for the package's ecosystem. * For ROS 2 this means supporting all tier 1 platforms, as defined in `REP-2000 `_ @@ -571,30 +670,30 @@ Package Requirements Requirements to be considered a 'Level 4' package: -#. **Version Policy:** +1. **Version Policy:** - #. No requirements, but having a policy is still recommended (e.g. ``semver``), even if the version is not yet stable (e.g. >= 1.0.0 for ``semver``) + i. No requirements, but having a policy is still recommended (e.g. ``semver``), even if the version is not yet stable (e.g. >= 1.0.0 for ``semver``) -#. **Change Control Process:** +2. **Change Control Process:** - #. No explicit change control process required, but still recommended + i. No explicit change control process required, but still recommended -#. **Documentation:** +3. **Documentation:** - #. Must have a declared license or set of licenses - #. Must have a copyright statement in each source file + i. Must have a declared license or set of licenses + ii. Must have a copyright statement in each source file -#. **Testing:** +4. **Testing:** - #. No explicit testing requirements, though covering some if not all of the features with tests is recommended + i. No explicit testing requirements, though covering some if not all of the features with tests is recommended -#. **Dependencies:** +5. **Dependencies:** - #. No restrictions + i. No restrictions -#. **Platform Support:** +6. **Platform Support:** - #. Must support all target platforms for the package's ecosystem. + i. Must support all target platforms for the package's ecosystem. * For ROS 2 this means supporting all tier 1 platforms, as defined in `REP-2000 `_ @@ -623,10 +722,12 @@ The `rcutils package's quality declaration ` package, based on the guidelines in [REP-2004](https://www.ros.org/reps/rep-2004.html). @@ -636,43 +737,61 @@ Quality Declaration Template Below are the rationales, notes, and caveats for this claim, organized by each requirement listed in the [Package Requirements for Quality Level N in REP-2004](https://www.ros.org/reps/rep-2004.html). - ## Version Policy + ## Version Policy [1] + + ### Version Scheme [1.i] + + ### Version Stability [1.ii] + + ### Public API Declaration [1.iii] + + ### API Stability Within a Released ROS Distribution [1.iv]/[1.vi] + + ### ABI Stability Within a Released ROS Distribution [1.v]/[1.vi] + + ## Change Control Process [2] + + ### Change Requests [2.i] + + ### Contributor Origin [2.ii] + + ### Peer Review Policy [2.iii] - ### Version Scheme + ### Continuous Integration [2.iv] - ### API Stability Within a Released ROS Distribution + ### Documentation Policy [2.v] - ### ABI Stability Within a Released ROS Distribution + ## Documentation [3] - ### Public API Declaration + ### Feature Documentation [3.i] - ## Change Control Process + ### Public API Documentation [3.ii] - ## Documentation + ### License [3.iii] - ### Feature Documentation + ### Copyright Statement [3.iv] - ### Public API Documentation + ## Testing [4] - ### License + ### Feature Testing [4.i] - ### Copyright Statements + ### Public API Testing [4.ii] - ## Testing + ### Coverage [4.iii] - ### Feature Testing + ### Performance [4.iv] - ### Public API Testing + ### Linters and Static Analysis [4.v] - ### Coverage + ## Dependencies [5] - ### Performance + ### Direct Runtime ROS Dependencies [5.i] - ### Linters and Static Analysis + ### Optional Direct Runtime ROS Dependencies [5.ii] - ## Dependencies + ### Direct Runtime non-ROS Dependency [5.iii] - ## Platform Support + ## Platform Support [6] References and Footnotes ======================== From d1074e43f25f957d75f50dbfda94ab10d86bcbfd Mon Sep 17 00:00:00 2001 From: Kyle Fazzari Date: Tue, 21 Apr 2020 11:10:35 -0700 Subject: [PATCH 08/14] Add security section (#247) Signed-off-by: Kyle Fazzari --- rep-2004.rst | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/rep-2004.rst b/rep-2004.rst index 7e89d5b65..f407c89bd 100644 --- a/rep-2004.rst +++ b/rep-2004.rst @@ -243,6 +243,12 @@ Requirements to be considered a 'Level 1' package: * For ROS 2 this means supporting all tier 1 platforms, as defined in `REP-2000 `_ +7. **Security** + + .. _7.i: + + i. Must have a declared Vulnerability Disclosure Policy and adhere to a response schedule for addressing security vulnerabilities + If the above points are satisfied then a package can be considered 'Level 1'. Refer to the detailed description of the requirements below the chart for more information. @@ -448,6 +454,12 @@ The chart below compares Quality Levels 1 through 5 relative to the 'Level 1' re - ✓ - ● - + * - 7.i_ + - ✓ + - ✓ + - ✓ + - + - Version Policy ^^^^^^^^^^^^^^ @@ -606,6 +618,10 @@ Requirements to be considered a 'Level 2' package: * For ROS 2 this means supporting all tier 1 platforms, as defined in `REP-2000 `_ +7. **Security** + + i. Must have a declared Vulnerability Disclosure Policy and adhere to a response schedule for addressing security vulnerabilities + If the above points are satisfied then a package can be considered 'Level 2'. Refer to the detailed description of the requirements following the Quality Level 1 section above for more information. @@ -655,6 +671,10 @@ Requirements to be considered a 'Level 3' package: * For ROS 2 this means supporting all tier 1 platforms, as defined in `REP-2000 `_ +7. **Security** + + i. Must have a declared Vulnerability Disclosure Policy and adhere to a response schedule for addressing security vulnerabilities + If the above points are satisfied then a package can be considered 'Level 3'. Refer to the detailed description of the requirements following the Quality Level 1 section above for more information. @@ -697,6 +717,10 @@ Requirements to be considered a 'Level 4' package: * For ROS 2 this means supporting all tier 1 platforms, as defined in `REP-2000 `_ +7. **Security** + + i. No restrictions + Any package that does not claim to be 'Level 3' or higher is automatically 'Level 4'. Refer to the detailed description of the requirements following the Quality Level 1 section above for more information. From eb5ba5a38ad466b918d79e49db984c8f0c40b0d9 Mon Sep 17 00:00:00 2001 From: William Woodall Date: Mon, 27 Apr 2020 10:21:18 -0700 Subject: [PATCH 09/14] Update rep-2004.rst MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Alejandro Hernández Cordero --- rep-2004.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/rep-2004.rst b/rep-2004.rst index f407c89bd..376c1c5d9 100644 --- a/rep-2004.rst +++ b/rep-2004.rst @@ -817,6 +817,10 @@ Quality Declaration Template ## Platform Support [6] + ## Security [7] + + ### Vulnerability Disclosure Policy [7.i] + References and Footnotes ======================== From d7a359160cba9864e7bd95e81bc43080996bdb55 Mon Sep 17 00:00:00 2001 From: William Woodall Date: Mon, 27 Apr 2020 10:28:34 -0700 Subject: [PATCH 10/14] Update rep-2004.rst Co-Authored-By: Marya Belanger --- rep-2004.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rep-2004.rst b/rep-2004.rst index 376c1c5d9..2ef3c5b6d 100644 --- a/rep-2004.rst +++ b/rep-2004.rst @@ -457,7 +457,7 @@ The chart below compares Quality Levels 1 through 5 relative to the 'Level 1' re * - 7.i_ - ✓ - ✓ - - ✓ + - ● - - From 4a3bbf6c296a725fda742ff8e695e2e1f6f59aa9 Mon Sep 17 00:00:00 2001 From: William Woodall Date: Mon, 27 Apr 2020 10:30:44 -0700 Subject: [PATCH 11/14] Update rep-2004.rst Co-Authored-By: Kyle Fazzari --- rep-2004.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rep-2004.rst b/rep-2004.rst index 2ef3c5b6d..bd60f70be 100644 --- a/rep-2004.rst +++ b/rep-2004.rst @@ -673,7 +673,7 @@ Requirements to be considered a 'Level 3' package: 7. **Security** - i. Must have a declared Vulnerability Disclosure Policy and adhere to a response schedule for addressing security vulnerabilities + i. Should have a declared Vulnerability Disclosure Policy and adhere to a response schedule for addressing security vulnerabilities If the above points are satisfied then a package can be considered 'Level 3'. Refer to the detailed description of the requirements following the Quality Level 1 section above for more information. From 0282e1c9a3a6787b865ddbc169f8a072cb2c1ad0 Mon Sep 17 00:00:00 2001 From: Marya Belanger Date: Wed, 29 Apr 2020 11:29:15 -0400 Subject: [PATCH 12/14] Adjusting 3.v.b requirement (#253) Signed-off-by: maryaB-osr --- rep-2004.rst | 28 +++++++++++++++++++++------- 1 file changed, 21 insertions(+), 7 deletions(-) diff --git a/rep-2004.rst b/rep-2004.rst index bd60f70be..1e1cb97e4 100644 --- a/rep-2004.rst +++ b/rep-2004.rst @@ -167,7 +167,11 @@ Requirements to be considered a 'Level 1' package: .. _3.v.b: - b. *Must register with a centralized list of 'Level N' packages, if one exists, to allow for peer review of the claim* + b. *Should register with a centralized list of 'Level N' packages, if one exists, to allow for peer review of the claim* + + .. _3.v.c: + + c. *Must reference any 'Level N' lists the package belongs to, and/or any other peer review processes undergone* .. _Testing: @@ -377,9 +381,15 @@ The chart below compares Quality Levels 1 through 5 relative to the 'Level 1' re - - * - 3.v.b_ + - ● + - ● + - ● + - + - + * - 3.v.c_ + - ✓ - ✓ - ✓ - - ● - - * - 4.i_ @@ -507,11 +517,11 @@ Sometimes the justification will be a link to a policy documented in the package If there is additional evidence that these policies are being followed, that should be included as well, e.g. a link to the coverage statistics for the package to show that coverage is being tracked and maintained. Other times, justification will be an explanation as to why a requirement was not met or does not apply, e.g. if performance tests do not make sense for the package in question, it should be satisfactorily explained. -There is no enforcement or checking of these claims, but instead it's just sufficient to present this information to potential users. +There is no enforcement or checking of these claims, but instead it's sufficient to present this information to potential users. If the users feel that the justifications are insufficient or incorrect, they can open issues against the repository and resolve it with the maintainers. -There should be one or more communal lists of 'Level 1' (and maybe 'Level 2' or 'Level 3') quality level packages. -These lists should be modified via change requests (maybe a text document in a repository) so that there can be peer review. +There should be one or more communal lists of 'Level 1' (and maybe 'Level 2' and 'Level 3') quality level packages for maintainers to register their packages with to seek peer review. +These lists should be modified via change requests (maybe a text document in a repository). This REP will not prescribe how or where these lists should be hosted, but one possibility is an informational REP, continually updated and versioned with each new ROS distribution. Feature Testing and Code Coverage Policy @@ -592,7 +602,8 @@ Requirements to be considered a 'Level 2' package: iv. Must have a "quality declaration" document, which declares the quality level and justifies how the package meets each of the requirements a. *Must have a section in the repository's ``README`` which contains the "quality declaration" or links to it* - b. *Must register with a centralized list of 'Level 2' packages, if one exists, to allow for peer review of the claim* + b. *Should register with a centralized list of 'Level 2' packages, if one exists, to allow for peer review of the claim* + c. *Must reference any 'Level 2' lists the package belongs to, and/or any other peer review processes undergone* 4. **Testing:** @@ -656,6 +667,7 @@ Requirements to be considered a 'Level 3' package: a. *Must have a section in the repository's ``README`` which contains the "quality declaration" or links to it* b. *May register with a centralized list of 'Level 3' packages, if one exists, to allow for peer review of the claim* + c. *Must reference any 'Level 3' lists the package belongs to, and/or any other peer review processes undergone* 4. **Testing:** @@ -795,6 +807,8 @@ Quality Declaration Template ### Copyright Statement [3.iv] + ### Lists and Peer Review [3.v.c] + ## Testing [4] ### Feature Testing [4.i] @@ -818,7 +832,7 @@ Quality Declaration Template ## Platform Support [6] ## Security [7] - + ### Vulnerability Disclosure Policy [7.i] References and Footnotes From cfa44ee57537c79775f747729d9642e9bdae0291 Mon Sep 17 00:00:00 2001 From: Marya Belanger Date: Thu, 30 Apr 2020 16:12:52 -0400 Subject: [PATCH 13/14] Optional chart symbol + "higher quality" distinction --- rep-2004.rst | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/rep-2004.rst b/rep-2004.rst index 1e1cb97e4..6bc4ba607 100644 --- a/rep-2004.rst +++ b/rep-2004.rst @@ -265,6 +265,8 @@ The chart below compares Quality Levels 1 through 5 relative to the 'Level 1' re ● = recommended +○ = optional + .. list-table:: Quality Levels :widths: 5 10 10 10 10 10 :header-rows: 1 @@ -371,7 +373,7 @@ The chart below compares Quality Levels 1 through 5 relative to the 'Level 1' re * - 3.v_ - ✓ - ✓ - - ● + - ○ - - * - 3.v.a_ @@ -383,7 +385,7 @@ The chart below compares Quality Levels 1 through 5 relative to the 'Level 1' re * - 3.v.b_ - ● - ● - - ● + - ○ - - * - 3.v.c_ @@ -447,9 +449,9 @@ The chart below compares Quality Levels 1 through 5 relative to the 'Level 1' re - - * - 5.ii_ - - ● - - ● - - ● + - ○ + - ○ + - ○ - - * - 5.iii_ @@ -733,7 +735,7 @@ Requirements to be considered a 'Level 4' package: i. No restrictions -Any package that does not claim to be 'Level 3' or higher is automatically 'Level 4'. +Any package that does not claim to be 'Level 3' or higher quality is automatically 'Level 4'. Refer to the detailed description of the requirements following the Quality Level 1 section above for more information. Quality Level 5 From 99c28651093a87e27616cf8c30331e16c133a8f6 Mon Sep 17 00:00:00 2001 From: William Woodall Date: Tue, 5 May 2020 11:29:43 -0700 Subject: [PATCH 14/14] address feedback about copyright statement requirements Signed-off-by: William Woodall --- rep-2004.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/rep-2004.rst b/rep-2004.rst index 6bc4ba607..5f4615bcd 100644 --- a/rep-2004.rst +++ b/rep-2004.rst @@ -155,7 +155,7 @@ Requirements to be considered a 'Level 1' package: .. _3.iv: - iv. Must have a copyright statement in each source file + iv. Must state copyrights within the project and attribute all authors .. _3.v: @@ -600,7 +600,7 @@ Requirements to be considered a 'Level 2' package: i. Must have documentation for each "feature" (e.g. for ``rclcpp``: create a node, publish a message, spin, etc.) ii. Must have a declared license or set of licenses - iii. Must have a copyright statement in each source file + iii. Must state copyrights within the project and attribute all authors iv. Must have a "quality declaration" document, which declares the quality level and justifies how the package meets each of the requirements a. *Must have a section in the repository's ``README`` which contains the "quality declaration" or links to it* @@ -664,7 +664,7 @@ Requirements to be considered a 'Level 3' package: 3. **Documentation:** i. Must have a declared license or set of licenses - ii. Must have a copyright statement in each source file + ii. Must state copyrights within the project and attribute all authors iii. May have a "quality declaration" document, which declares the quality level and justifies how the package meets each of the requirements a. *Must have a section in the repository's ``README`` which contains the "quality declaration" or links to it* @@ -715,7 +715,7 @@ Requirements to be considered a 'Level 4' package: 3. **Documentation:** i. Must have a declared license or set of licenses - ii. Must have a copyright statement in each source file + ii. Must state copyrights within the project and attribute all authors 4. **Testing:**