Invalid data dump

[hagbeck]

Hi,
I'm using the following request at weekends with the additional parameter format=jsonl to get the data as JSON-lines dump:

https://lobid.org/resources/search?q=almaMmsId:*6441+AND+NOT+hasItem.heldBy.id%3A%22http%3A%2F%2Flobid.org%2Forganisations%2FDE-290%23!%22

since two weeks I'm getting only 1.1 M records instead of the 24 M requested.

What happens and how to solve this?

[TobiasNx]

@dr0i can you check this?

[dr0i]

@hagbeck can you check your provided query - as it yields more than 24M records, which would be more that 24 MB jsonl.

[dr0i]

uh, sorry @hagbeck I see, you meant 24 M records not 24 MB...

[dr0i]

We have around some hours some performance troubles - this may very well end up in not serving all the data (so this problem could be just temporary (... although this doesn't explain why you don't get the data since 2 weeks... )).
But let me get that straight: you want to get 24 M records ? Why that many? (I thought your regularly ~ 2 M would be the maximum someone would query...)

[hagbeck]

Why? It's open data ;-) I've to index the hbz Union Catalog weekly in a separate collection to our discovery service. And so far it was not an problem.

[dr0i]

Just wondering ... how long does it take and how big is the file?

[hagbeck]

The complete process including transformations takes ten hours, whereas the main part is the download. The size is currently unknown.