Query exporting only 2ˆ28 documents (data have more) even after setting query limit to 2ˆ29

[laisuchoa]

Description

We're working with a CouchDB database that holds about 272 million records. When attempting to retrieve all documents using a curl command and writing to a file, we noticed that only 268435456 records were returned, despite the database having more entries.

The curl command in question:

curl -k -X GET "https://$(cat credential_couchdb)@{host}/{db}/_all_docs?include_docs=true" > all_docs

This led us to suspect a default query limit issue, which aligns with discussions in this GitHub pull request.

To resolve this, we updated the config file to set both partition_query_limit and query_limit to 536870912 (2^29).

"query_server_config":{"partition_query_limit":"536870912","query_limit":"536870912"}

Despite this change and the offset field from the output file displaying the correct amount of total records, the output still holds 268435456 records.

Expected Behaviour

We expect the number of records in the output file to match the offset field value, not to be stuck in this default value count.

Additional Context

The CouchDB instance in question is integrated as a state database within a Hyperledger Fabric setup.
The database is updated once a day.

@lucasmation

[rnewson]

couchdb/src/couch_mrview/include/couch_mrview.hrl

Line 57 in aca60dd

-define(MAX_VIEW_LIMIT, 16#10000000).

couchdb/src/couch_mrview/include/couch_mrview.hrl

Line 72 in aca60dd

limit = ?MAX_VIEW_LIMIT,

You may need to paginate, or pass a larger ?limit= query parameter.