Search Data Packages API returns publication date as year (i.e. YYYY format)

[clnsmth]

A user of the Search Data Packages REST API, via the EDIutils R package, reported an unexpected behavior where the returned pubdate field has values in the format YYYY rather than the expected format YYYY-MM-DD. The returned begindate and enddate fields have values in the format YYYY-MM-DD and this incongruence seems to be the source of confusion. For the original report see: ropensci/EDIutils#45.

The users behavior can be reproduced outside of EDIutils with:

https://pasta.lternet.edu/package/search/eml?q=scope:(knb-lter-fce)&fl=doi,title,packageid,begindate,enddate,pubdate

Now, the Level-1 EML <pubDate> element values of the data packages returned by the users query, lists the date in the format YYYY-MM-DD, so It seems there is a transformation that occurs when processing the EML for delivery through the Search Data Packages REST API.

A quick search of the PASTA+ codebase turns up a potential source of the observed behavior at:

PASTA/common/src/edu/lternet/pasta/common/eml/EmlObject.java

Line 187 in 8994894

* Returns the publication year based on the publication date in the

Thoughts @servilla @rogerdahl?

Thanks!

[servilla]

Hi @clnsmth,

reported an unexpected behavior

This behavior is completely expected; it's just not what the user wants. The PASTA pubdate has returned the "YYYY" format in the search API since its inception (2015), and yearDate is a valid (and is the default type) format for the EML pubDate field. It is possible to return the full "YYYY-MM-DD", considering that PASTA now overwrites this field with the repository upload date.

so It seems there is a transformation that occurs when processing the EML for delivery through the Search Data Packages REST API

Exactly. This is an expected transformation (above).

A quick search of the PASTA+ codebase turns up a potential source

It is this code:

PASTA/DataPackageManager/src/edu/lternet/pasta/datapackagemanager/solr/search/SimpleSolrSearch.java

Lines 382 to 384 in 8994894

	if (fieldName != null && fieldName.equalsIgnoreCase("pubdate")) {
	bestDateFormat = "YYYY";
	}

Action: we should discuss as a group the most appropriate "pubDate" to return, realizing that changing this to a full date may affect other users who rely on this field containing a "YYYY" formatted date.

[clnsmth]

Thanks for looking into this @servilla, and for linking to the code that controls the returned date format.

And yes, 100%, I should have more clearly stated that the "expected behavior" is what the user was expecting, rather than imply there was something wrong with the code.