Write about invariants & expectations for views

[cinnamon-bun]

It would be nice to clearly specify what a view can expect. I'm not sure this is correct:

• Messages will be processed in any order, even within a single feed (?)
• Messages will be processed exactly once each. If you have storeState and fetchState then processing will pick up where it left off last time; otherwise it starts from scratch each time.
• What happens when only part of a hypercore is downloaded, and then the rest shows up later? Does the saved progress state assume a single moving front of completed processing or does it know there could be mixed regions of processed and unprocessed messages?
• If views want to persist their internal state (such as sum in the example) they need to do that in map (and not call next() until the persist is complete)
• What happens if your map doesn't call next()? Is this an ok way to abort processing?
• Is it safe for a view to pause itself from inside map?

[cinnamon-bun]

Also

• Multiple views operate independently; you can't specify an order of views (so that one view is always ahead of another, in case they depend on each other)

[hackergrrl]

Multiple views operate independently; you can't specify an order of views (so that one view is always ahead of another, in case they depend on each other)

You can, but it's maybe not obvious just from reading APIs. If you have a view foo that depends on bar, your map function or api:{} functions can do something like

api: {
  getValue: function (core, cb) {
    core.api.bar.ready(() => {
      this.getValue(cb)
    })
  }
}

[cinnamon-bun]

Aha, that's true. I was thinking of dependencies where one view would access another's API from inside its map function during indexing.

I suppose if you want to do anything tricky with multiple views it's easiest to combine them into a single view so you can have more control over execution order.

[hackergrrl]

You can also wait for views to be ready within the map function; I just need to expose the kappa-core instance so you can call ready on it. ready also takes optional params, so you can do core.ready(['users', 'channels'], cb.

[hackergrrl]

Going over this again: documenting here until it gets put into a doc somewhere visible:

Messages will be processed in any order, even within a single feed (?)

Yes, this is good to assume. Currently it's in-order per feed, but when the stack supports sparse mode, it will not stay true.

Messages will be processed exactly once each. If you have storeState and fetchState then processing will pick up where it left off last time; otherwise it starts from scratch each time.

Sort of. The view's map function may be called multiple times with messages, such as if the map function was called but the app crashed before storeState ran. It's important to write your views in such a way that map can be called idempotently.

I think if kappa-core knew that state and view data were stored together, they could be written atomically, which would relieve downstream developers from needing to worry about idempotency.

What happens when only part of a hypercore is downloaded, and then the rest shows up later?

Right now, multifeed-index doesn't support sparse mode well. What happens is mf-index will grab the first N (opts.maxBatchSize) messages from a feed, to index. If some are missing, it bails and moves on to another feed. So: nothing will break if entries are missing in a hypercore, but noncontiguous data won't get indexed until it becomes contiguous.

If views want to persist their internal state (such as sum in the example) they need to do that in map (and not call next() until the persist is complete)

Right. Ideally in an idempotent fashion.

What happens if your map doesn't call next()? Is this an ok way to abort processing?

No. This will pause indexing for this view indefinitely. You probably don't want to do this: kappa-core has a pause / resume API for doing this. If you don't want to do any work, call next() without doing any writes anywhere, and that entry will be skipped.

Is it safe for a view to pause itself from inside map?

Yes, it should be fine. mf-index is pretty smart about concurrency stuff like this.

[hackergrrl]

• add answers to an faq.md doc

PRs welcome! :) Basically just copy+pasta'ing the Qs and As into a markdown doc!