Metadata Pre-Flight Checks for Publishing

[AhemNason]

The body of this largely came from a result of the 2023 PKP Sprint in Hanover. In brief, I can summarize a justification here at the top. Otherwise, everything below the header is as it appeared in the sprint proposal. Currently, in OJS, publishing an issue is very easy and happens with little fanfare. You click publish. The work is published. But publishing is a big deal, and a single button click can shotgun poor metadata to an increasingly wide variety of downstream destinations. Asking users if they’ve reviewed all the metadata for their issue before they formally publish as a workflow or wizard would provide editors with an opportunity to see what they might have missed, forgotten, or overlooked. If you want to review your publication metadata now, you can preview the issue but it won’t show you everything that’s going to be available in metadata. And if you were to go look at every article, that’s a heap of clicks to get to the production page for each individual article in an issue. A pre-flight check could flag, for example, missing affiliation fields, references, funding, multilingual fields, licenses, DOI assignment, or any other number of potential things. Like a sort of last-minute sanity check on publication that could encourage editors to double check instead of publishing and then returning back to issues to un-publish and publish materials.

Metadata Sprint Group

Members

Mike Nason | PKP & UNB
Xenia van Edig | TIB
Erik Hanson | PKP
Patrick Vale | Crossref
Stephan Henn | UCL Cologne
Marco Tullney | TIB
Oliver Colberg | UCL Cologne

Automatic metadata checks before sending e.g. to Crossref (AKA Automatic Pre-Flight Metadata Checks)

Background | Why is this Important?

This should be self-evident, we think. Metadata pushed out of OJS is false/bad/compromised in a variety of ways.

Summary of Issues | Brainstorming

Approaches to sorting our or preventing false/erroneous metadata being pushed to Crossref. This is already on the CRAFTOA todo list. https://github.com/withanage/publicationValidator

Maybe worth looking at:
Shi, J., Nason, M., Tullney, M., & Alperin, J. P. (2023, August 17). Identifying Metadata Quality Issues Across Cultures. https://doi.org/10.31235/osf.io/6fykh.

• language detection
  • facebook, for example, had an API for detecting language in forms
  • "this work is supposed to be metadata in english, but it seems like this is french."
  • people may have good intentions with their metadata,
• what kind of metadata would benefit from a test before submission?
• what subgroup of these might have a method for automatic checking?
  • "usually this sort of metadata looks like x… but in this case you've entered y"
• at the moment, Dulip's plugin is only looking for missing metadata
• in some cases, metadata isn't consistent in the same issue.
  • capitalization in author names… affiliation… title… probably harder to check but still problematic

What sorts of metadata would benefit from metadata checks?

• Valid XML coming out of the system.
• Attached galleys
• Language (checking to see if metadata exists for all languages and the check for whether the language matches the locale/field).
• Affiliation/ROR ID
  • Does affiliation exist in general?
  • Adding departmental affiliation is possible via the institution_department in the Crossref api.
• Encoding issues. (from pasting via word, PDF)
  • Exists everywhere, including references…
• Single number page ranges?
• Publication dates way in the future or super far in the past…
• Information entered differently than with all the other articles (capitalization of author names, affiliations, titles eg.), in general: homogeneous metadata

When might we check?

• Submission
  • Is there a way to tie this into the citation plugin? Rendering of the citation to show people on submission what it would look like.
• When scheduled before publication
• On issue publication?
• any time you push the work to the next stage of the workflow?
  • if you see this too many times you might just ignore it.

Maybe there should be a report as part of the publication workflow… a pre-flight check to show all the metadata for the works about to be published.

• And, ideally, someone writing a plugin for plan S or Crossref or whatever, could hook into this workflow.
• What would we need for a pre-flight check?

When are things supposed to happen?

• As a principle, we aim at a tool that enables checks (and confirmation by an author/editor) of metadata that often is wrong at different stages of the submission and publishing process
• Can we have a preflight that can be run at different stages, e.g. whenever someone hands something over to another person/stage?
  • Too often could annoy people, essential are submission and pre-publication
• Maybe the stages are so different that we cannot have the same processes/name, e.g.
  • during submission: extender submission overview in OJS 3.4/3.5
  • during review/editing: new preflight
  • Proof sign-off: make sure all the necessary information is available

People do not understand the ‘preferred name’ field. We need a better explanation, pre-population or removal of that field. (And if at any point we see the light and adopt a single string for names, it could be avoided.)

Goals

What are you hoping to achieve? This is where your group will describe its plans at the start of the sprint. Keep it SMART: specific, measurable, achievable, realistic, and time specific (within two day sprint, no homework!).

• Given an understanding that we couldn't possibly solve this problem or create these solutions within the given time frame, we are breaking the work (per the recommendations of Erik) into one of scoping for future development.
• We're proposing a pre-flight check function or plugin for OJS.
• We hope to discover how many of these metadata issues are actually solvable problems.
  • how might we check?
  • what's possible?
• We'd like to create an issue description to take forward to PKP for plugin-related work.

Results | Pre-flight Check Scoping Proposal

We know that OJS journals are pushing, in some cases, pretty poor metadata downstream. This conversation is often about "completeness". It is almost too easy to publish scholarly works with less than the bare minimum of article metadata. This is a bit of a double-edged sword. Obviously, you don't want to get in the way of users publishing how they want, but this also presumes that editors and authors have a good understanding of what may be missing.

We're proposing a pre-flight check function, that would allow an overview of submitted and/or pre-publication work to allow for user review. We think this makes sense in two places:

1. Submissions
2. Issue Publication

Submissions

In OJS 3.4, we provide a summary for submissions. It provides users with an overview of the content being submitted to OJS. This is actually pretty close to what we intend to propose at the submission level. We also know that Dulip is working on a publication validator as part of the CRAFTOA project that can look for or report empty fields. https://github.com/withanage/publicationValidator.

A submission pre-flight would check against the following:

• are affiliations provided for all authors?
• are author names "complete" for all authors?
• have references or abstracts been populated if required?
• is an ORCID provided if required?
• is there a title?
• is any of the metadata in ALL CAPS
• and, ideally…
  • is metadata in a given field written in the same language as expected within the form
  • is an email address valid
  • is a ROR ID provided alongside affiliations

In general, we're not proposing that users be forced to correct these issues but, instead, that they are made aware of anything missing that they may not have considered. More gentle encouragement than punitive finger wagging.

In the case of a submission pre-flight check, we're already kind of on our way here.

A screenshot of the deposit/submission review in OJS 3.4… 2023

Issue Publication

The act of publishing an issue in OJS is actually pretty unceremonious. You click publish. The issue is live. Content is distributed worldwide; metadata sent to ORCID, Crossref, DataCite, via OAI to any number of downstream locations… etc. There's an assumption that editors are reviewing their content in this space with due diligence before clicking the "publish" button. Or, maybe more importantly, an assumption that editors understand and take seriously the metadata rube-goldberg machine they're setting in motion when they publish their issues. But, OJS doesn't necessarily make it easy to review all the relevant metadata.

Problems with checking include:

• taking many clicks to review all submission metadata for every article in an issue
• no easy way to see if collected metadata is consistent across an issue
  • for example, two or three authors haven't included affiliations…
  • some articles have incomplete pagination
  • some articles have references where others do not
  • when some articles have multilingual metadata, others may not, and reviewing this is laborious or obtuse
• it's easier to spot a mistake than it is to know what might have been omitted.

A publication pre-flight check would allow editors to review, comprehensively, the work they are about to publish. It could/might/should(?) take inspiration from the submission summary that exists in 3.4. It would review the following elements and identify some or all of the following for completeness and, eventually, accuracy.

• Issue-level metadata
• Article metadata
  • Everything from Submission Pre-flight
  • Related works identifiers
    • DOIs for deposited datasets?
    • DOIs for posted preprints?
    • DOIs or record specific to peer-review?
  • Pagination
  • Galleys
  • Sections
  • Article DOI Preview
    • More useful for, say, copy editors or layout editors than an opportunity to complain about the human-readability of a suffix.
• CHECK FOR ALL CAPS
• and, ideally…
  • is metadata in a given field written in the same language as expected within the form
  • or other, programmatic/automatic format sanity checks

Proposed Behaviour

The emphasis here isn't just about what metadata is missing, but which metadata has been recorded inconsistently. "We noticed that for these articles, you have [content x] but for these articles you have [content y] or no content at all". This would address general issues around completeness, we believe. Ideally, it would be possible to also allow users to edit metadata from this space. And, extra, extra ideally, users could filter out any metadata fields they're not specifically interested in checking.

In the future, additional functionality could be added based on downstream requirements specified by relevant organizations. For example:

• crossref
• doaj
• plan s
• coalition publica

Certain intended downstream services may have specific requirements for metadata that aren't required for publication but are required for those journals to fulfill their responsibilities or agreements. These could be plugins created or maintained by those partner organizations.

Next Steps

• Implement. ( ͡° ͜ʖ ͡°) ( ͡☉ ͜ʖ ͡☉) ( ͡◉ ͜ʖ ͡◉)
• This would go under "discussions" as a github issue, but not necessarily as an issue. Could convert the discussion to an issue after the dev team has an opportunity to talk about this. Discussions a better place for this sort of proposal.

Food for thoughts/future ideas too big for this sprint

Language detection

• Goal: be able to discover if the text in a metadata field does not match the language indicated for that field
• To consider:
  • choice of software/criteria
  • How many languages are supported an
  • Service (in case we use a 3rd party service)
  • who is operating the service
  • are there rate limits
  • what are the costs
  • Service (in case someone sets up a service for OJS journals (like all of them))
  • What are the costs involved (including maintenance)
  • Who will help tens of thousands of journals when they run into trouble?
• Examples:
  • Lingua (Java) https://github.com/pemistahl/lingua
  • Apache Tika (Java) https://tika.apache.org/2.8.0/detection.html#Language_Detection

Typographical checks

• Goal: be able to check for common typographical errors in fields
• To consider:
  • copy-and-paste errors from word/PDF
  • unexpected punctuation in given name eg _\/*@()[]
  • Jr or Sr or et al. in given name
  • malformed entities eg & or &#
  • not allowed characters in the proposed DOI