Implement --minimum-confidence parameter for vCA reward command #156
Conversation
|
Here is a spreadsheet showing the difference of output: https://docs.google.com/spreadsheets/d/1Sam0fcEIRdx7flujAiKzS0zcM60UfdQupJBy8altbzw/edit#gid=0 Note that I used the f8-vca-merged.csv that was removed a few commits ago. |
|
I've updated the spreadsheet and also added the output of the original tool without this patch into a new tab to show that it is identical when using |
Value in range [0.5, 1] The minimum consensus for a vCA ranking to be excluded from eligible rankings when in disagreement from simple majority. Simple majority is 50%. Qualified majority is 70%. Using 70% avoids punishing vCAs where the consensus is not clear. 70% is because when #vca == 3 consensus is only 66% and thus in this case, where there is just 1 vote in disagreement, all 3 vCAs get rewarded.
2072
force-pushed
the
vca_rewards_minimum_consensus
branch
from
d2cce74 to
c4e6dfc
Compare
|
Hey @2072, thank you for your PR! Or is what you are proposing different from that? Also, I'm lacking a bit of context, is there a broader consensus behind this change or is it just your (valid) suggestion? |
No this is different, it's about not making a review not legitimate when the vcas are on a tie on whether it's good or FO. Which in turn prevents these vCAs from seeing their agreement rate falling below one of these tunable threshold you mentioned and being unduly punished for a crime they did not commit.
This change was discussed and accepted by the community but we were told it was too late to be implemented for this fund given the delay and that more data was needed. So I took the opportunity and did it myself so you get the change and the data :-) Here is the google doc where this change was discussed: https://docs.google.com/document/d/1bN3L0-R7ntmKKY9KwQTeevQLuDExyJ1NkH7b7-uOXtE/edit#heading=h.dmlc2526vg0q |
…sus_per_review() into calc_final_ranking_with_consensus_per_review() This also fixes a small bug where the consensus was lowered on Excellent assessments because the Good ones count was not taken into account, This had a low impact since Excellent assessments are much rarer. (The final_ranking_is_correct test caught this.)
|
OK, all your requested changes are done, this allowed me to catch a small issue on the consensus of Excellent assessment where the number of Good where not taken into account (very low impact on rewards du to the rareness of Excellent and the rareness of positive disagreement on those). |
The documentation added is enough to clear any doubts and "confidence" does not imply as much as does the term "consensus" which prevent false assumptions.
2072
changed the title
Thanks! |
This PR implements a new command line parameter:
--minimum-confidencewhich takes a value between0.5and1It represents the minimum confidence for a vCA ranking to be excluded from eligible rankings when in disagreement from simple majority.
Confidence is either
#FO / #Rankingsor(#Excellent + #Good) / #Rankingsdepending on the final ranking.Simple majority is 50%. Qualified majority is 70%. Using 70% avoids punishing vCAs where this confidence is not clear.
70% is because when
#vCA == 3confidence is only 66% and thus in this case, where there is just 1 vote in disagreement, all 3 vCAs get rewarded.Note: to get the previous behavior just set this parameter to
0.5:--minimum-confidence 0.5