Reviewer Credits

[withanage]

Review functionality

Orcid_Peer_Review.mp4

Reviewer Anonymity metadata transfer concept

Information	Orcid Field	Anonymous	Open
1. Date of submission	created-date	YYYY	YYYY-MM-DD
2. Date of acceptance (review)	review-completion-date	YYYY	YYYY-MM-DD
3. Date of publication	created-date	YYYY	YYYY-MM-DD
7. Article URL	review-url	❌	✔️

Review Functionalitry

Plugin Settings

Orcid_Settings.mp4

Links

• Definition: https://support.orcid.org/hc/en-us/articles/360006971333-Peer-Review
• API : https://api.orcid.org/v3.0/#!/Development_Member_API_v3.0/viewPeerReviewv3
• SANDBOX API https://api.sandbox.orcid.org/v3.0/#!/Development_Member_API_v3.0/createPeerReviewv3
• Peer Review Orcid https://github.com/ORCID/ORCID-Source/blob/master/orcid-api-web/tutorial/peer-review.md
• Niso group https://www.niso.org/standards-committees/peer-review-taxonomy
• marc codes https://www.loc.gov/marc/countries/countries_name.html

Orcid Review display

• In user profiles, peer-review will be visible only when the user has enabled it. ref

Specification (from ORCID)

a. Information about the reviewer

1. Role (required): The individual’s role in the review process, e.g. chair, editor, member, organizer, reviewer 🆗

2. Identifier (required): The reviewer’s ORCID iD
   full orcid URL including https 🆗

b. Information about the organizer

• Convening organization (required): This describes the organization which organized the review - a journal publisher, conference organizer, funding agency, faculty, etc.

3. We can use the Publisher name under Settings- > Journal -> masterhead -> publisher ❓

Location

4. City (required): The city where the organizer is based

• Region: The region where the organizer is based
• Country (required): The country where the organizer is based

If I am correct, the only location where the journal address is default maintained is under principal contact. There also we have the mailing address. This can be a little tricky to parse, if the journal managers do not provide it correctly. Country is straight-forward, but the city can be not easy to find.

Another possibility is to get the location from the issn. Haven't explored it whether they have an API or does it make sense ?
e.g. https://portal.issn.org/resource/ISSN/0006-1387 ❓

5. Identifier (required): The persistent identifier for the organization

ISSN of the journal or ROR id of the press / hosting organization ❓

c. Information about the review

6. Group (required): An identifier used to group together the reviews on the ORCID record; it describes the group of which the review is part. This could be the name of a journal (Journal of Scientific Investigations) or an organization (Scholarly Publisher, State University)

User -> affiliation ❓

7. Type (required): The type of review activity, e.g., a review, evaluation

Review 🆗

8. Date (required): When the review was completed. This can be broad (2008) or specific

Reveiw data 🆗

9. Review container name: The name of main object of which the review is part, e.g. journal, conference, grant review panel, etc.

journal name 🆗

10. Review identifier (required): A unique resolvable identifier provided by the source for the review itself. This is used to prevent duplication of review activity. Reviews can have more than one identifier, and multiple reviews with the same identifier will group together

Do we have a reviewer URL for the front-end through a plugin may be ❓

References:

This plugin sends reviewer id, submission-id,data and (due and submission) to reveiwercredits.com

https://www.reviewercredits.com/integrations-for-journals-and-manuscript-platforms/

Related to 10

11, Review URL: A link to the representation of the review online

d. Information about the review subject

• Review subject name: The title of the item reviewed
• Review subject type: The type of item that was reviewed, e.g. article, grant, monograph
• Review subject identifier: A unique resolvable identifier for the reviewed item, e.g. a DOI
• Review subject URL: A link to the reviewed item online

OJS supports all 🆗

References

• https://support.orcid.org/hc/en-us/articles/360006971333-Peer-Review

TODO

• Check the review round
• Check visibility
• Sending emails
• Update with put code
• Deleting submission
• Country codes as plugin settings
• [ ]

Related tickets

• Allow a journal's editorial to authenticate and display ORCIDs  pkp-lib#7406
• Reviewers should be asked to authenticate an ORCID before reviewing pkp-lib#7405
• Journals should be able to demonstrate that their reviewers are qualified pkp-lib#7404

[withanage]

Hi @asmecher,

I have gone through the requirements from orcid.org of adding o reviewer information for the orcidPlugin.

Most of the requirements seem to be supported directly from OJS, but for the points 4 and 10, I think we have to think for a solution, what is the best possiblie solution. i am not sure , if 4 or 10 was already adressed by any plugins ?

[asmecher]

Thanks, @withanage!

Most of the requirements seem to be supported directly from OJS, but for the points 4 and 10, I think we have to think for a solution, what is the best possible solution. i am not sure , if 4 or 10 was already addressed by any plugins ?

See below:

City (required): The city where the organizer is based

Hmm, I'm surprised this is a requirement! I don't think we have any other requirements for a field like this in the software, so until then, it should be a setup field in the plugin, I think.

Region: The region where the organizer is based

(this is optional)

Country (required): The country where the organizer is based

See pkp/pkp-lib#6099, which adds a "country" field -- this will be released in OJS/OMP/OPS 3.4.0. The country can also be determined from the ISSN via the Wordcat API, but better skip that if it's not required.

Review identifier (required): A unique resolvable identifier provided by the source for the review itself. This is used to prevent duplication of review activity. Reviews can have more than one identifier, and multiple reviews with the same identifier will group together

This, I think, best corresponds to the review_round_id in OJS -- i.e. reviews in that round would be grouped together with the same number. I trust this ID doesn't have to be globally unique? Plus, if we later wanted to support exposing review information about a round via the API or front end (for open reviews), it's the review_round_id we'd likely use.

[withanage]

A functional alpha implementation of peer-review support

[NateWr]

This looks great, @withanage! I noticed that reviews are deposited when the article is published. However, reviewers should get credit whenever a review is completed, even if the submission is rejected. So I think that the ORCID deposit ought to occur when the reviewer submits their review, or when the editor confirms it. Or deposits could be made through a weekly scheduled task that deposited any completed assignments on a published or declined submission.

[withanage]

Hi @NateWr

I noticed that reviews are deposited when the article is published. However, reviewers should get credit whenever a review is completed, even if the submission is rejected. So I think that the ORCID deposit ought to occur when the reviewer submits their review, or when the editor confirms it

Thanks, I personally would prefer when the reviewer submits it (in the OJS 3.3. version, if the reviewer has already stored authenticated orcid credentials and for future versions a more better, email based approach ) What do you think ?

[withanage]

Hi @asmecher and @NateWr ,

By looking at the NISO recommendations, first comments from the group and our current OJS reviewer model: I tried to map the reviewers model in the Ticket header #181. Please have a look and if you have any suggestions, let me know.

Also the questions came to me, at any stage if we should also delete a orcid peer-review ? The new API allows that, if we have a put-code in the database and I haven't programmed it yet for deleting functionality for author orcid assignments too.

[NateWr]

I personally would prefer when the reviewer submits it (in the OJS 3.3. version, if the reviewer has already stored authenticated orcid credentials and for future versions a more better, email based approach ) What do you think ?

I think that is probably best. I wonder about cases where a reviewer submits the form but hasn't actually completed a review. For example, maybe a reviewer submits the form to say "sorry i can't review". Also, if it's the journal that is bestowing "credit" for a review, the editor might not want to grant credit for an inadequate review (unhelpful, mean-spirited, not detailed enough). I guess there's a lot to explore here about reviewer credit. Who is the motivated party for this ticket? Who has asked us for this feature? We may need to go back and explore this in more detail.

I tried to map the reviewers model in the Ticket header

Can you describe the reviewer mapping table in more detail? I'm not sure what I'm looking at there.

Also the questions came to me, at any stage if we should also delete a orcid peer-review ?

I suspect that will need to be handled, but I can think of lots of problems arising just from software maintenance over time. For example, a journal decides to use OJS just for workflow, not for publishing, and decides to "clean up" their old published submissions by deleting them from the system. If we automated deletes, we'd end up removing valid review entries.

How does ORCID handle disputes on its end? Or situations where the depositing authority is no longer available to curate the depositing data?

[withanage]

Hi @NateWr

Can you describe the reviewer mapping table in more detail? I'm not sure what I'm looking at there.

I have taken the out mapping part here for disucussion. My suggestion was that may be the numbered metadata fields can be published openly for the various reviewer types without disclosing the anonymity.

Mapping

Identity transparency	Accept Submission	Revisions Required	Resubmit for Review	Resubmit Elsewhere	Decline Submission	See Comments
Anonymous Reviewer/Anonymous Author (Double anonymized)	5	5	5	5	5	x
Anonymous Reviewer/Disclosed Author (Single anonymized)	5	5	5	5	5	x
Open (All identities visible)	1,2,3,5,6,7	1,2,3,5,6,7	1,2,3,5,6,7	1,2,3,5,6,7	1,2,3,5,6,7	x

Metadata

Information	Orcid Field
1. Date of submission	created-date
2. Date of acceptance (review)	review-completion-date
3. Date of publication	created-date
4. Number of reviewer reports submitted in first round	❌
5. Number of revision rounds	source-name
6. Article Group (ISSN)	review-group-id
7. Article URL	review-url

How does ORCID handle disputes on its end? Or situations where the depositing authority is no longer available to curate the depositing data?

I am not yet aware, how they handle that. I would try to get feed back from orcid employees about that ?

[NateWr]

@withanage What do the recommendation columns mean in the table? Does it matter what decision the reviewer takes?

[withanage]

@withanage What do the recommendation columns mean in the table? Does it matter what decision the reviewer takes?

@NateWr : actually other than the comments , it does not matter, I added only for completeness. For the comments, I was not 100% sure ,cause the reviewer may opt for not reviewing by giving a reason as a comment. I think in this case, it may be better as you suggested #181 (comment) to trigger the orcid API communication, at the moment when an editor reads review and confirms it.

[NateWr]

Yeah, I think that reviewers often use See Comments even when they include a review, but don't want to choose a pre-selected recommendation. I think we should expect to send it on to ORCID unless the editor does something.

In terms of what data to send to ORCID, I feel like we should probably send all the data in all cases. It should be up to ORCID to decide what to show depending on whether the review is anonymous or not. For example, they'll still need the date in order to show the year in which a review occurred, as can be seen in this example: https://orcid.org/0000-0001-7795-4652

Ultimately, the anonymity is about the article that was reviewed and the precise date. So maybe the metadata table should look like this:

Information	Orcid Field	Anonymous	Open
1. Date of submission	created-date	YYYY	YYYY-MM-DD
2. Date of acceptance (review)	review-completion-date	YYYY	YYYY-MM-DD
3. Date of publication	created-date	YYYY	YYYY-MM-DD
7. Article URL	review-url	❌	✔️

[withanage]

@NateWr thanks for the abstraction. Yes, I missed the review-url totally and I will take i the above table as the specification.

[withanage]

@NateWr

How does ORCID handle disputes on its end? Or situations where the depositing authority is no longer available to curate the depositing data?

Quote from the answer:

We do not provide specific instructions on how to proceed if the article is removed. Other than that, the member organization could delete the peer review if they have permission to do so (/activities/update). And at the following link you can find our Dispute Procedure

In a quick browse and from the email, we could automatically delete them, if the submission is removed. I will have a look into it in more detail and come with a proposal.

[NateWr]

we could automatically delete them, if the submission is removed. I will have a look into it in more detail and come with a proposal.

I suspect it's going to be very rare that a submission will go through reviews but then those reviews should be withdrawn from orcid. I know I raised this issue, but on second thought, I think we can leave this to ORCID to sort out. People delete submissions for lots of reasons in OJS, and most of the time the review credit should remain with ORCID.

[klausru]

Dear @withanage
can you confirm that the delivery of Reviewer Credits towards ORCID really works for ORCID Members in production mode? From reading the above I wasn't sure if this feature is mature and out of alpha status.
In addition to that: We are planning to become member of ORCID. Is there any script or trigger to deliver back issues / reviewer data to ORCID from past reviews? I wouldn't want unpublish - republish each and every article.
Thank you so much and all the best

[withanage]

Hi @klausru ,

can you confirm that the delivery of Reviewer Credits towards ORCID really works for ORCID Members in production mode? From reading the above I wasn't sure if this feature is mature and out of alpha status.

The functionality can be used for production purposes.

In addition to that: We are planning to become member of ORCID. Is there any script or trigger to deliver back issues / reviewer data to ORCID from past reviews? I wouldn't want unpublish - republish each and every article.

no, unfortunately there isn't a script for re-publish in the current version and is also not planned currently. may be a small utility function can be writtten for that purpose. There are some examples here, how those scripts look like.

https://github.com/pkp/ojs/tree/stable-3_3_0/tools