Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Define the scope of web-platform-tests #215

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Define the scope of web-platform-tests #215

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
60b04c5
Add a quick and dirty draft of what might be the de-facto scope of WPT
gsnedders Dec 6, 2024
4e73ec1
Update scope-of-wpt.md
gsnedders Dec 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 60 additions & 0 deletions rfcs/scope-of-wpt.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# RFC 215: Defining the Scope of web-platform-tests

### Summary

Define the scope of web-platform-tests to specifications which might reasonably be called "web specifications"
and which are intended to be implemented by web browsers.

Additionally, define what "tentative" might reasonably be used for.

This is intended to codify existing practice,
rather than represent any change.


### Details

Historically, what's been in scope for web-platform-tests has been somewhat nebulous,
and largely based on a common understanding of participants,
especially those on the Core Team.
We should put this on surer footing,
and make it easier for others to understand what is and isn't reasonable to add.

The primary aim of web-platform-tests is to test
[web specifications](https://github.com/w3c/browser-specs/blob/11a71b738f5e41f9239fdcd2074153388c8c6b8b/README.md#spec-selection-criteria)
[intended to be implemented by browsers](https://github.com/w3c/browser-specs/blob/11a71b738f5e41f9239fdcd2074153388c8c6b8b/README.md#categories),
both RFC 2119 "must" and RFC 2119 "should"
implementation requirements.

A specification does not need to have cross-vendor support for its tests to be included in web-platform-tests.
(XXX: Should we set a bar of one-vendor support or are we okay with zero-vendor support?)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In theory I would be OK with zero-vendor support, since incubation can start outside browser vendors, but I worry that the necessary cleanup for unsuccessful incubations won't be done if nobody is footing the bill.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then again, the discussion around tentative tests below indicates that this wouldn't be a new problem.


Additionally, other features can be added as
[tentative](https://web-platform-tests.org/writing-tests/file-names.html#:~:text=.tentative,-%3A%20)
tests:

* Web browser behavior currently being explored via an
[explainer](https://tag.w3.org/explainers/),
but without a specification yet written.
(XXX: require prototyping to have started?)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think by definition prototyping has started when there are tests, but I'm not sure what adding that requirement would mean in practice. Maybe pointers to bugs on file or intents to prototype?


* Historic but unspecified features in web browsers,
especially where major browsers are interoperable,
where there exists a long-term intention to specify them.
(Note: this excludes features which have been deliberately removed from specifications.
These are explicitly out of scope.)
Comment on lines +31 to +44
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This arguably doesn't actually match up with what the documentation currently says:

Indicates that a test makes assertions not yet required by any specification, or in contradiction to some specification. This is useful when implementation experience is needed to inform the specification. It should be apparent in context why the test is tentative and what needs to be resolved to make it non-tentative.

The "in contradiction to some specification" is kinda troubling — this implies that if some spec change is made, with no expectation of implementations changing behaviour any time soon, you could change the existing tests to be tentative and add new non-tentative tests for the new behaviour. IMO, that should be against policy — anything that is tentative should be on a path to becoming non-tentative. We should probably be explicit in the scope of what can be tentative that everything which is tentative should be on a path to becoming non-tentative.

I don't think the two things listed here are a complete list of cases; we also have cases:

The other thing that stands out is:

% rg --glob '*tentative*' 'https?://((bugs\.webkit\.org|bugs\.chromium\.org|crbug\.com|issues\.chromium\.org)/|github\.com/.*/(issue|pull))' --files-without-match | wc -l
    2078

Which shows it's very often not obvious "why the test is tentative and what needs to be resolved to make it non-tentative", though that is really a separate issue. (And looking at a few random tests, the commit history often doesn't make it obvious either.)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I appreciate this analysis and I wonder if we should separate adding more clarity on tentative tests from the rest of this RFC. I.e. let's not attempt to boil the ocean :-)


Features which are optional
(in RFC 2119 terms:
where the conformance criteria is
"may" or "may not")
may be included as
[optional tests](https://web-platform-tests.org/writing-tests/file-names.html#:~:text=.optional,-%3A%20)
or
[optional subtests](https://web-platform-tests.org/writing-tests/testharness-api.html#optional-features).


### Risks

We over-constrain what is allowed in web-platform-tests,
potentially raising the bar to change what is allowed to "submit a new RFC",
leading browser vendors to reduce what they submit to the project.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that there is no bar to submitting an RFC, any and all change suggestions should be welcome. I suspect you meant that there is a risk of raising the bar to adding more tests without sending an RFC first, because adding said tests would require us to relax some of these rules?