how to achieve these features with _find API?

[fangq]

I understand that _find API offers the ability to do ad-hoc search of the entire document. I wanted to use it in the following three scenarios but could not figure out how

1. when finding a match, the "fields":[] specifier can only retrieve the content of the specified object, but not parameters, even simple ones such as the length of an array, or the key names of an objects etc. Is this possible to retrieve these properties? my content may include embedded objects and can be be fairly large and I don't want to return those.

2. my document contains special annotations, such as _DataLink_ as key name, in various depth; if I want to query how many such key exists, can I use _find? or design document? I know sqlite provides iterators such as such as json_each() or json_tree for such purpose.

3. I know this may sound a bit strange - can I use _find to search in design document views? my understand is that a design document view provides extracted queryable information, it it does not have the ability to specify complex search conditions; it would be nice to use _find to query the digested view documents - it would be both fast and flexible.

[fangq]

I am wondering if any couchdb developers can comment on my above questions? thanks

[nickva]

1. when finding a match, the "fields":[] specifier can only retrieve the content of the specified object, but not parameters, even simple ones such as the length of an array, or the key names of an objects etc. Is this possible to retrieve these properties? my content may include embedded objects and can be be fairly large and I don't want to return those.

Maybe others can correct me, but I don't think that's possible. It's the kind of stuff you could do in the map/reduce view if you process the document with a javascript function

2. my document contains special annotations, such as DataLink as key name, in various depth; if I want to query how many such key exists, can I use _find? or design document? I know sqlite provides iterators such as such as json_each() or json_tree for such purpose.

There is $keyMapMatch https://docs.couchdb.org/en/stable/api/database/find.html#combination-operators but I don't that will work for recursively searching the document to find a field at arbitrary depths. This might be a job for design view function and then combined with a _count built-in reducer to count the number of such matches

3. I know this may sound a bit strange - can I use _find to search in design document views? my understand is that a design document view provides extracted queryable information, it it does not have the ability to specify complex search conditions; it would be nice to use _find to query the digested view documents - it would be both fast and flexible.

It's not strange, it's an interesting idea. Behind the scenes they are actually implemented the other way - _find indexes are implemented internally as a _design map view. It's just not using javascript but an erlang view mapper. But that's why there is the one-btree index limitation: once an index is picked to respond it's just one index, we can't currently merge (AND or OR) results from two indexes.

[fangq]

@nickva, thank you so much for your comments, really helpful for me to understand the primary use cases of couchdb's indices.

There is $keyMapMatch https://docs.couchdb.org/en/stable/api/database/find.html#combination-operators but I don't that will work for recursively searching the document to find a field at arbitrary depths. This might be a job for design view function and then combined with a _count built-in reducer to count the number of such matches

the type of JSON documents I am working on can have various levels of depths. These documents are currently hosted on a CouchDB server, with a web interface for browsing. You can see a few examples here

https://neurojson.org/db/openneuro/ds000001
https://neurojson.org/db/openneuro/ds002680#highlight=sub-002

I am trying to aggregate some information from these JSON-encoded metadata sets. For example, for one of the needs, I am trying to extract a special annotation tag _DataLink_ which is used in these documents to encode URL pointing to external binary data files, and report the file type and sizes. However, these tags are embedded at various depths and various parent constructs. You can type datalink in the Find keyword search box to see a few examples.

I already wrote a design document script that manually traverse the top 2-3 levels. Now I understood that recursion is not possible, I will further extend my explicit traversal code to handle 4-5 levels. It should capture most of those tags.

[fangq]

It's not strange, it's an interesting idea. Behind the scenes they are actually implemented the other way - _find indexes are implemented internally as a _design map view. It's just not using javascript but an erlang view mapper. But that's why there is the one-btree index limitation: once an index is picked to respond it's just one index, we can't currently merge (AND or OR) results from two indexes.

I wonder how _find handles $and $or matches? it feels like we can use multiple conditions and their logical combinations this way.

if _find can be used to query _design views, I would imagine that it would be dramatically faster than traversing the entire document/database because _design views are much more compact. even _design view indices can not handle multiple indices, if _find can do $and, $or etc on view data, I think that also solves the problem.