Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase quality of compat data #3555

Closed
7 of 11 tasks
Elchi3 opened this issue Mar 1, 2019 · 156 comments
Closed
7 of 11 tasks

Increase quality of compat data #3555

Elchi3 opened this issue Mar 1, 2019 · 156 comments
Labels
KR: Real BCD 🔑 Key Result: Eliminate true/null values and replace them with "real" values. O/KR 🔑 Objectives and Key Results. Most attention will be focused on this.

Comments

@Elchi3
Copy link
Member

Elchi3 commented Mar 1, 2019

The MDN team is employed by Mozilla and has yearly goals. I'm opening this issue to inform you about this year's goal and to give status updates throughout the year.

In 2018, the team's goal for mdn/browser-compat-data has been to finish migrating compat data from the legacy wiki tables into this repository and offering the data to 3rd party tools. This goal is now almost complete (see this gist for remaining pages using the legacy tables and the list of tools using the data.) Work is underway to fully finish this. This goal is now complete.

In 2019 and going forward, the goal will be to increase the quality of the compat data found in this repository. Compat data quality can be measured in various ways (I will write about this in the wiki soon), but we're going to focus on our efforts on getting 100% real values for Firefox, IE, Edge, Chrome, Safari, mobile Safari, mobile Chrome for all web platform features (i.e. WebExtensions are excluded). Real values here means to replace all "true", "null" or no data values with either a version number or "false".

We have finished the following categories:

Beginning of the year overall baseline data (based on version 0.0.66, released on 7. Feb. 2019) for all 8 tracked browsers for web platform features: :

browser real values true values null values
All 8 browsers 52.36% 23.77% 23.87%
chrome 64.55% 24.47% 10.98%
chrome android 48.28% 32.89% 18.83%
edge 39.38% 29.75% 30.89%
firefox 73.08% 13.63% 13.28%
ie 59.35% 15.14% 25.52%
safari 48.17% 19.28% 32.55%
safari ios 39.92% 20.55% 39.54%
webview android 46.17% 34.44% 19.39%

Going forward, I will update this issue with every release of BCD, so that we can track where we are.

If you have questions about how this goal was decided, or if you have feedback, please reach out to me or other MDN team members.

@Elchi3 Elchi3 added the O/KR 🔑 Objectives and Key Results. Most attention will be focused on this. label Mar 1, 2019
@Elchi3
Copy link
Member Author

Elchi3 commented Mar 1, 2019

Status as of version 0.0.69 (released on 28th February 2019) for web platform features:

browser real values true values null values
All 8 browsers 54.49% (+2.13) 22.21% (-1.56%) 23.31% (-0.56%)
chrome 65.20% 24.05% 10.75%
chrome android 49.27% 32.31% 18.41%
edge 53.07% 20.06% 26.88%
firefox 73.44% 12.97% 13.59%
ie 59.41% 14.88% 25.72%
safari 48.44% 18.92% 32.64%
safari ios 40.19% 20.29% 39.54%
webview android 46.89% 34.19% 18.92%

I've highlighted Edge here to show the most significant changes to the prior statistics. Updating Edge data was mostly driven by @foolip. Thanks for your work, Philip!

@ddbeck
Copy link
Collaborator

ddbeck commented Mar 1, 2019

@Elchi3, this is excellent! Thank you for getting started on this effort and sharing the baseline publicly. Would you mind sharing the code you're using to generate the stats, too? Just a gist would be fine—I think generating quality stats could be baked into the npm run release-notes script and save you some time at releases

@Elchi3
Copy link
Member Author

Elchi3 commented Mar 3, 2019

Thanks Daniel! It makes total sense to have this as a script! I've opened #3557 as a first stab at this. When you commented here, I actually realized that we should review the data gathering of this as we surely want these statistics to be correct 😄

Previously, there was a script that created a csv which then was used to make charts in a sheet, so this is inspired from that prior art (see https://github.com/atopal/browser-compat-data-dashboard).

@foolip
Copy link
Collaborator

foolip commented Mar 4, 2019

Thanks for the stats @Elchi3!

I wonder how you'd like to deal with increasing the API coverage of BCD? When adding new entries entirely, it is arguably increasing the the usefulness of the data, but if data for all browsers isn't added up front it will regress these metrics slightly. One option for dealing with this is using absolute values, and aiming for what 100% would mean today.

@Elchi3
Copy link
Member Author

Elchi3 commented Mar 4, 2019

Thanks @foolip! You're right, by adding more data, this metric can regress.

I think I'd want to be ambitious and see if we can reach this goal anyway and see if can maybe disallow adding new "true" and "nulls" once we've finished parts of this. So, e.g. "all Edge data" or all "Edge CSS data" has 100% real values and then the linter would disallow adding new "true" and "null" values for these parts and so we can't regress anymore. This would force new data contributors to provide real values and I don't know how much of a burden that would be on them. It would be another option, however.

Also tagging @atopal to comment on the option of tracking absolute values of today.

@atopal
Copy link

atopal commented Mar 5, 2019

That is an interesting problem. The key result is an indicator for substantial progress against what we really care about: making the compatibility data more useful for individuals and tooling vendors. So, I think we want to reflect current reality as part of that key result and that means using the current data set at any given time. As an added benefit, I think that it will motivate us to figure out how to keep the quality up and not regress over time.

That said, we might want to set a goal that allows for some minor fluctuations. 100% is just a very nice and very round number 😃

@foolip
Copy link
Collaborator

foolip commented Mar 5, 2019

If you treat 0.7 as the expected average score of KRs (do you?) then to either 70% or ~85% real values, depending on if you treat 0% or 51.11%, would be success.

@atopal
Copy link

atopal commented Mar 6, 2019

Yes, .7 is the expected average score. I think it's fair to take 51% as the baseline. We are shooting for 100%, but a minimum of 85% would already be a significant improvement.

@Elchi3
Copy link
Member Author

Elchi3 commented Mar 8, 2019

I've updated the above stats for 0.0.66 and 0.0.69 using the latest version of the the script and here is the latest numbers for yesterday's release.

Status as of version 0.0.70 (released on 07/03/2019) for web platform features:

browser real values true values null values
All 8 browsers 55.02% (+0.53%) 22.08% (-0.13%) 22.90% (-0.41%)
chrome 65.68% 23.92% 10.40%
chrome android 49.91% 31.99% 18.09%
edge 53.46% 19.96% 26.59%
firefox 73.60% 12.90% 13.50%
ie 60.19% 14.79% 25.02%
safari 49.03% 18.88% 32.08%
safari ios 40.74% 20.34% 38.93%
webview android 47.51% 33.90% 18.59%

@jpmedley
Copy link
Contributor

jpmedley commented Mar 15, 2019 via email

@atopal
Copy link

atopal commented Mar 15, 2019

Yes, it's available here: #3557

@jpmedley
Copy link
Contributor

jpmedley commented Mar 21, 2019 via email

@Elchi3
Copy link
Member Author

Elchi3 commented Mar 21, 2019

[Apologies to everyone for the lack of reviews from my side last week and this week. I was on vacation and then at a workshop presenting about this project :)]

Status as of version 0.0.72 (released on 21/03/2019) for web platform features:

browser real values true values null values
All 8 browsers 55.29% (+0.27%) 22.26% (+0.18%) 22.46% (-0.44%)
chrome 65.99% 23.90% 10.11%
chrome android 50.51% 31.92% 17.57%
edge 53.27% 20.28% 26.46%
firefox 73.65% 13.30% 13.06%
ie 60.60% 14.81% 24.59%
safari 49.40% 19.10% 31.49%
safari ios 41.11% 20.48% 38.43%
webview android 47.77% 34.27% 17.96%

@Elchi3
Copy link
Member Author

Elchi3 commented Mar 28, 2019

Status as of version 0.0.73 (released on 28/03/2019) for web platform features:

browser real values true values null values
All 8 browsers 55.87% (+0.58%) 22.49% (+0.23%) 21.64% (-0.84%)
chrome 66.15% 24.43% 9.42%
chrome android 51.11% 32.40% 16.49%
edge 55.70% 20.01% 24.30%
firefox 73.89% 13.52% 12.59%
ie 60.66% 14.81% 24.53%
safari 49.77% 19.16% 31.07%
safari ios 41.59% 20.64% 37.79%
webview android 48.09% 34.95% 16.96%

@Elchi3
Copy link
Member Author

Elchi3 commented Apr 4, 2019

Status as of version 0.0.74 (released on 04/04/2019) for web platform features:

browser real values true values null values
All 8 browsers 55.92% (+0.05%) 24.45% (+1.96%) 19.64% (-2%)
chrome 66.37% 24.31% 9.32%
chrome android 51.50% 32.26% 16.24%
edge 55.59% 19.92% 24.50%
firefox 73.71% 13.61% 12.68%
ie 60.52% 14.75% 24.74%
safari 49.77% 25.91% 24.33%
safari ios 41.55% 30.00% 28.49%
webview android 48.37% 34.81% 16.82%

In this release we were able to move 2% of the values from null to true and I want to thank @foolip and @vinyldarkscratch for their work specifically.

@foolip
Copy link
Collaborator

foolip commented Apr 4, 2019

Thanks @Elchi3, that's great to see!

@queengooborg
Copy link
Collaborator

queengooborg commented Apr 4, 2019

💜

All null and true values, begone!

@connorshea
Copy link
Contributor

connorshea commented Apr 8, 2019

Someone should write a script that can generate a line graph of these percentages over time :) (bonus points if you go back all the way to ~0.50 as well)

@connorshea
Copy link
Contributor

@vinyldarkscratch did you generate that gist with a script, and if so could you share it? :)

@queengooborg
Copy link
Collaborator

I did, yeah -- I wrote linter/test-version-bool.js initially as a test against version_added: true, hence the name, but then adapted it to allow for any value. I basically just take the output and run some Sublime Text magic to extract the specific features afterwards (not the best, but I tend to unintentionally do things manually :P).

@Elchi3
Copy link
Member Author

Elchi3 commented Apr 11, 2019

Status as of version 0.0.75 (released on 11/04/2019) for web platform features:

browser real values true values null values
All 8 browsers 55.96% (+0.04%) 25.85% (+1.40%) 18.19% (-1.45%)
chrome 66.35% 26.36% 7.29%
chrome android 51.54% 34.96% 13.50%
edge 55.56% 20.56% 23.89%
firefox 73.70% 16.43% 9.87%
ie 60.58% 14.69% 24.73%
safari 49.87% 25.87% 24.28%
safari ios 41.64% 29.95% 28.44%
webview android 48.42% 38.02% 13.56%

@Elchi3
Copy link
Member Author

Elchi3 commented Apr 18, 2019

Status as of version 0.0.76 (released on 18/04/2019) for web platform features:

browser real values true values null values
All 8 browsers 56.18% (+0.22%) 25.69% (-0.16%) 18.14% (-0.05%)
chrome 66.56% 26.21% 7.23%
chrome android 51.82% 34.83% 13.35%
edge 55.85% 20.60% 23.56%
firefox 74.44% 15.58% 9.98%
ie 60.64% 14.58% 24.78%
safari 49.89% 25.80% 24.31%
safari ios 41.71% 29.88% 28.44%
webview android 48.55% 38.01% 13.44%

@queengooborg queengooborg pinned this issue Apr 18, 2019
@github-actions
Copy link

github-actions bot commented Feb 3, 2023

Status as of version 5.2.34 (released on 2023-02-03) for web platform features:

browser real values ranged values true values null values
total 94.91% 1.40% 2.38% 1.31%
chrome 88.67% 0.00% 8.72% 2.61%
chrome android 98.31% 0.00% 1.65% 0.04%
edge 97.85% 2.11% 0.01% 0.02%
firefox 95.23% 0.00% 2.34% 2.42%
safari 92.85% 0.30% 3.21% 3.63%
safari ios 99.47% 0.19% 0.23% 0.11%
webview android 92.07% 7.69% 0.16% 0.09%

@github-actions
Copy link

Status as of version 5.2.35 (released on 2023-02-10) for web platform features:

browser real values ranged values true values null values
total 94.92% 1.40% 2.38% 1.31%
chrome 88.67% 0.00% 8.72% 2.61%
chrome android 98.31% 0.00% 1.65% 0.04%
edge 97.87% 2.10% 0.01% 0.02%
firefox 95.23% 0.00% 2.34% 2.42%
safari 92.85% 0.30% 3.21% 3.63%
safari ios 99.47% 0.19% 0.23% 0.11%
webview android 92.07% 7.69% 0.16% 0.09%

@github-actions
Copy link

Status as of version 5.2.36 (released on 2023-02-17) for web platform features:

browser real values ranged values true values null values
total 94.94% 1.40% 2.36% 1.30%
chrome 88.78% 0.00% 8.64% 2.58%
chrome android 98.31% 0.00% 1.65% 0.04%
edge 97.78% 2.19% 0.01% 0.02%
firefox 95.28% 0.00% 2.32% 2.40%
safari 92.93% 0.30% 3.18% 3.59%
safari ios 99.47% 0.19% 0.23% 0.11%
webview android 92.08% 7.67% 0.16% 0.09%

@github-actions
Copy link

Status as of version 5.2.37 (released on 2023-02-21) for web platform features:

browser real values ranged values true values null values
total 94.98% 1.40% 2.33% 1.30%
chrome 89.01% 0.00% 8.42% 2.58%
chrome android 98.31% 0.00% 1.65% 0.04%
edge 97.80% 2.17% 0.01% 0.02%
firefox 95.29% 0.00% 2.32% 2.39%
safari 92.93% 0.30% 3.18% 3.59%
safari ios 99.47% 0.19% 0.23% 0.11%
webview android 92.09% 7.67% 0.16% 0.09%

@github-actions
Copy link

Status as of version 5.2.38 (released on 2023-02-23) for web platform features:

browser real values ranged values true values null values
total 94.99% 1.40% 2.32% 1.29%
chrome 89.03% 0.00% 8.40% 2.57%
chrome android 98.32% 0.00% 1.64% 0.04%
edge 97.80% 2.16% 0.01% 0.02%
firefox 95.30% 0.00% 2.31% 2.39%
safari 92.95% 0.30% 3.17% 3.58%
safari ios 99.47% 0.19% 0.23% 0.11%
webview android 92.11% 7.65% 0.16% 0.09%

@github-actions
Copy link

Status as of version 5.2.39 (released on 2023-02-28) for web platform features:

browser real values ranged values true values null values
total 94.99% 1.40% 2.32% 1.30%
chrome 89.02% 0.00% 8.40% 2.57%
chrome android 98.32% 0.00% 1.65% 0.04%
edge 97.80% 2.17% 0.01% 0.02%
firefox 95.30% 0.00% 2.31% 2.39%
safari 92.94% 0.30% 3.18% 3.58%
safari ios 99.47% 0.19% 0.23% 0.11%
webview android 92.10% 7.66% 0.16% 0.09%

@github-actions
Copy link

github-actions bot commented Mar 8, 2023

Status as of version 5.2.40 (released on 2023-03-07) for web platform features:

browser real values ranged values true values null values
total 94.98% 1.40% 2.32% 1.30%
chrome 89.00% 0.00% 8.42% 2.59%
chrome android 98.32% 0.00% 1.65% 0.04%
edge 97.81% 2.15% 0.01% 0.02%
firefox 95.31% 0.00% 2.31% 2.38%
safari 92.93% 0.30% 3.17% 3.59%
safari ios 99.47% 0.19% 0.23% 0.11%
webview android 92.10% 7.66% 0.16% 0.09%

@github-actions
Copy link

Status as of version 5.2.41 (released on 2023-03-10) for web platform features:

browser real values ranged values true values null values
total 94.98% 1.39% 2.32% 1.30%
chrome 89.00% 0.00% 8.42% 2.59%
chrome android 98.32% 0.00% 1.65% 0.04%
edge 97.81% 2.15% 0.01% 0.02%
firefox 95.31% 0.00% 2.31% 2.38%
safari 92.93% 0.30% 3.17% 3.60%
safari ios 99.47% 0.19% 0.23% 0.11%
webview android 92.10% 7.65% 0.16% 0.09%

@github-actions
Copy link

Status as of version 5.2.42 (released on 2023-03-14) for web platform features:

browser real values ranged values true values null values
total 94.99% 1.39% 2.32% 1.30%
chrome 89.01% 0.00% 8.41% 2.58%
chrome android 98.32% 0.00% 1.64% 0.04%
edge 97.82% 2.15% 0.01% 0.02%
firefox 95.32% 0.00% 2.31% 2.37%
safari 92.94% 0.30% 3.17% 3.59%
safari ios 99.47% 0.19% 0.23% 0.11%
webview android 92.11% 7.64% 0.16% 0.09%

@github-actions
Copy link

Status as of version 5.2.43 (released on 2023-03-17) for web platform features:

browser real values ranged values true values null values
total 95.00% 1.39% 2.32% 1.29%
chrome 89.02% 0.00% 8.39% 2.58%
chrome android 98.32% 0.00% 1.64% 0.04%
edge 97.85% 2.11% 0.01% 0.02%
firefox 95.33% 0.00% 2.30% 2.37%
safari 92.94% 0.31% 3.16% 3.58%
safari ios 99.47% 0.19% 0.23% 0.11%
webview android 92.11% 7.64% 0.16% 0.09%

@github-actions
Copy link

Status as of version 5.2.44 (released on 2023-03-21) for web platform features:

browser real values ranged values true values null values
total 95.00% 1.39% 2.32% 1.29%
chrome 89.02% 0.00% 8.39% 2.58%
chrome android 98.32% 0.00% 1.64% 0.04%
edge 97.85% 2.11% 0.01% 0.02%
firefox 95.33% 0.00% 2.30% 2.37%
safari 92.94% 0.31% 3.16% 3.58%
safari ios 99.48% 0.19% 0.23% 0.10%
webview android 92.11% 7.64% 0.16% 0.09%

@github-actions
Copy link

Status as of version 5.2.45 (released on 2023-03-24) for web platform features:

browser real values ranged values true values null values
total 95.00% 1.39% 2.32% 1.29%
chrome 89.02% 0.00% 8.39% 2.58%
chrome android 98.32% 0.00% 1.64% 0.04%
edge 97.85% 2.11% 0.01% 0.02%
firefox 95.33% 0.00% 2.30% 2.37%
safari 92.94% 0.31% 3.16% 3.58%
safari ios 99.48% 0.19% 0.23% 0.10%
webview android 92.11% 7.64% 0.16% 0.09%

@github-actions
Copy link

Status as of version 5.2.46 (released on 2023-03-28) for web platform features:

browser real values ranged values true values null values
total 95.00% 1.39% 2.32% 1.29%
chrome 89.02% 0.00% 8.39% 2.58%
chrome android 98.32% 0.00% 1.64% 0.04%
edge 97.85% 2.11% 0.01% 0.02%
firefox 95.33% 0.00% 2.30% 2.37%
safari 92.94% 0.31% 3.16% 3.58%
safari ios 99.48% 0.19% 0.23% 0.10%
webview android 92.11% 7.64% 0.16% 0.09%

@github-actions
Copy link

Status as of version 5.2.47 (released on 2023-03-31) for web platform features:

browser real values ranged values true values null values
total 95.00% 1.39% 2.32% 1.29%
chrome 89.03% 0.00% 8.39% 2.58%
chrome android 98.32% 0.00% 1.64% 0.04%
edge 97.85% 2.11% 0.01% 0.02%
firefox 95.33% 0.00% 2.30% 2.37%
safari 92.95% 0.31% 3.16% 3.58%
safari ios 99.48% 0.19% 0.23% 0.10%
webview android 92.12% 7.64% 0.16% 0.09%

@github-actions
Copy link

github-actions bot commented Apr 4, 2023

Status as of version 5.2.48 (released on 2023-04-04) for web platform features:

browser real values ranged values true values null values
total 95.08% 1.37% 2.28% 1.27%
chrome 89.20% 0.00% 8.26% 2.54%
chrome android 98.35% 0.00% 1.61% 0.04%
edge 97.89% 2.08% 0.01% 0.02%
firefox 95.41% 0.00% 2.26% 2.33%
safari 93.06% 0.30% 3.11% 3.53%
safari ios 99.48% 0.19% 0.23% 0.10%
webview android 92.25% 7.51% 0.15% 0.08%

@github-actions
Copy link

github-actions bot commented Apr 7, 2023

Status as of version 5.2.49 (released on 2023-04-07) for web platform features:

browser real values ranged values true values null values
total 95.09% 1.36% 2.27% 1.27%
chrome 89.21% 0.00% 8.25% 2.54%
chrome android 98.35% 0.00% 1.61% 0.04%
edge 97.89% 2.08% 0.01% 0.02%
firefox 95.41% 0.00% 2.26% 2.33%
safari 93.07% 0.30% 3.11% 3.52%
safari ios 99.48% 0.19% 0.23% 0.10%
webview android 92.26% 7.50% 0.15% 0.08%

@github-actions
Copy link

Status as of version 5.2.50 (released on 2023-04-11) for web platform features:

browser real values ranged values true values null values
total 95.09% 1.36% 2.27% 1.27%
chrome 89.22% 0.00% 8.24% 2.54%
chrome android 98.35% 0.00% 1.61% 0.04%
edge 97.89% 2.08% 0.01% 0.02%
firefox 95.41% 0.00% 2.25% 2.33%
safari 93.07% 0.30% 3.11% 3.52%
safari ios 99.48% 0.19% 0.23% 0.10%
webview android 92.27% 7.49% 0.15% 0.08%

@github-actions
Copy link

Status as of version 5.2.51 (released on 2023-04-18) for web platform features:

browser real values ranged values true values null values
total 95.09% 1.36% 2.27% 1.27%
chrome 89.22% 0.00% 8.24% 2.54%
chrome android 98.35% 0.00% 1.61% 0.04%
edge 97.89% 2.08% 0.01% 0.02%
firefox 95.42% 0.00% 2.25% 2.33%
safari 93.08% 0.30% 3.11% 3.51%
safari ios 99.48% 0.19% 0.23% 0.10%
webview android 92.27% 7.49% 0.15% 0.08%

@github-actions
Copy link

Status as of version 5.2.52 (released on 2023-04-21) for web platform features:

browser real values ranged values true values null values
total 95.10% 1.36% 2.27% 1.27%
chrome 89.23% 0.00% 8.23% 2.53%
chrome android 98.35% 0.00% 1.61% 0.04%
edge 97.89% 2.07% 0.01% 0.02%
firefox 95.42% 0.00% 2.25% 2.33%
safari 93.09% 0.30% 3.10% 3.51%
safari ios 99.49% 0.19% 0.23% 0.10%
webview android 92.28% 7.48% 0.15% 0.08%

@github-actions
Copy link

Status as of version 5.2.53 (released on 2023-04-25) for web platform features:

browser real values ranged values true values null values
total 95.08% 1.36% 2.27% 1.29%
chrome 89.19% 0.00% 8.23% 2.58%
chrome android 98.36% 0.00% 1.60% 0.04%
edge 97.89% 2.07% 0.01% 0.02%
firefox 95.38% 0.00% 2.25% 2.37%
safari 93.05% 0.30% 3.10% 3.55%
safari ios 99.49% 0.19% 0.23% 0.10%
webview android 92.28% 7.48% 0.15% 0.08%

@queengooborg
Copy link
Collaborator

For quite some time, this goal has been very low on our priority list, as other goals have been more important, in particular the goal to ensure that our version numbers are accurate. As such, I'm going to close this issue.

@queengooborg
Copy link
Collaborator

Work on this issue had resumed once again in the form of a project managed by Open Web Docs -- see openwebdocs/project#206 for more details!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
KR: Real BCD 🔑 Key Result: Eliminate true/null values and replace them with "real" values. O/KR 🔑 Objectives and Key Results. Most attention will be focused on this.
Projects
None yet
Development

No branches or pull requests

13 participants
@ddbeck @Elchi3 @foolip @atopal @connorshea @queengooborg @jpmedley and others