course: when doing peer grading, provide an official way for students to enter a score

[williamstein]

Right now when doing peer grading there is no CANONICAL "guard rails" way for students to enter the score they are assigning for their peers. That's partly because students using cocalc to do the grading are just looking at completely arbitrary documents in a directory, and there's nothing whatever that is changed about their cocalc project to indicate that it is a student project. Also the content they are peer grading can be absolutely anything -- jupyter notebooks, markdown files, latex documents, etc.

The way peer grading is used by everybody so far is that you create a grading rubrik or guidelines file, and the students enter grades there. A TA or instructor then collects peer grading and reviews and assembles the grades together into the actual grade that the student gets. This depends on some manual work and some conventions. When I'm teaching, I always want to have my TA (or myself) review all the peer grading before assigning a grade to a student. However their could be settings (e.g., large understaffed classes) where maybe peer grading with no sign off by the instructor is OK, and then the whole process could be automated.

I've labeled this unclear and also NOT easy, because there are many different ways this could be implemented, and it's not at all clear what the right choices are. Also, changing cocalc so that student projects behave in a special way that is different than other cocalc project, specifically when doing peer grading, is itself nontrivial and full of subtle questions.

[novoselt]

One of the most concerning points here is that student graders may be very different in terms of grading quality, starting from "as good as an instructor and even more detailed" to "absolutely nothing, not even opening the file to be graded". The automatic way to fight this is to assign grading to many students and then have an algorithm to derive the actual grade that instructors may want to customize. Personally, if I couldn't oversee peer grading and had to rely on automation, I would stick with nbgrader...

[williamstein]

Personally, if I couldn't oversee peer grading and had to rely on automation, I would stick with nbgrader...

I've used peer grading a lot in teaching undergrads, but was always in a fortunate position to be able to spend them time to properly oversee it. This was absolutely critical to making it work well, in the same way that you need editors for journals.