bug: listDocuments filtering out design docs does not adhere to provided limit

[TillaTheHun0]

listDocuments filters out design documents from Couch response, before returning the list back. This is to prevent design docs from being sent back to the consumer, thus leaking implementation details about the underlying datastore. This can result in unexpected behavior.

In Couch, an index is stored as a special document called a "design document"

Suppose I have a DB with 2k docs, and 4 indexes. If I call listDocuments with a limit of 1000, the adapter will receive 1000 documents from Couch. If there are any index design documents in that response, say 2, then the adapter will filter them out, thus returning 998 documents to caller, not 1000. The caller could interpret this as not enough documents exist in the DB, to satisfy the provided limit, even though there are 2k documents in the DB.

We'll need to figure out how to take design documents into account on listDocuments. Do we continue to filter them out? If so, do we make an additional call to the DB to "replace" any filtered out design documents? Or do we just include design documents in the response (I think this leaks details about Couch, but open to debate).

[TillaTheHun0]

One naive solution could be to fetch an additional static amount of documents, on top of the provided limit, when calling Couch. If 'x' design documents are filtered out of the first 'limit' documents, then include 'x' number of documents from the additional static amount fetched in that call.

list limit:10
0,1,2,3,4,design_doc,6,7,design_doc,9,extra_1,extra_2,extra_3
// filter design docs and append with extras result:
0,2,3,4,6,7,9,extra_1,extra_2

This would also need to make sure those additional documents also don't include design documents (recursion).

The static amount of additional docs could be a sufficiently high amount that is greater than the number of indexes, say 100, or something like that. Or could query the amount of indexes in the db, and then add that to the limit (but requires an extra call to Couch to do that)