it was confusing why a utxo standard coin was being used instead of the coin we are building, this makes a bit more sense

…erstanding

There was no point of the option wrapping. Simplifying stuff.

also does some housekeeping, file splitting and renaming

remove the intermediate bridge task and replace it with `filter_map` call.
also properly implement the shutdown handle

also wraps the streamermanager mutable data inside Arc<RwLock< so to be used in concurrent code.
RwLock was chosen here since we should be sending streaming data way more than editing the streamers

along with some cleanups and simplifications

Also use the `data_rx` for utxo balance events.
A cleanup for the scripthash notification channels/logic will follow

this is very messy though (because of the current state of electurm code) and will be reverted, just recording this in the history for now

…nterface"

This reverts commit 8972822.

made more sense to have it inside the manager. also now the manager provides the controller as a parameter in the handle method so different streamers don't have to keep a copy of ctx just to broadcast on the controller.

this is still not hooked up though. no data will be received on this rx channel till we do
`event_stream_manager.send('BALANCE:ZCoin') (or whatever the coin name is)` somewhere.

also let StreamingManager::add return StreamingManagerError instead of String

we only extracted the ctx here to use the controller. since the controller is provided as an argument in EventStreamer::handle already, no need to extract the context in these senarios

looks like there are no more errors that could arise :/

no need for the feild. looks like the only use for it was to access the channel controller.

we want to use another type of filter (to enable and disable streams dynamically while keeping the SSE connection open)

… receivers

change it only when extracting the output out

instead of applying the filter at event level, we will instead filter while broadcasting.
in here we choose which clients we want the event to be broadcasted to. This should also reduce unnecessary overhead for slow client connections who don't pick up the events quickly (they now won't receive unrelated events, thus keeping extra more space in their channels)

now clients will subscribe to some streamer.
if the streamer doesn't yet exist, it will be spawned using the client provided configuration. and the client will be registered with as a listener to that streamer (so every new event from that streamer will be sent to that client).
on the other hand, if the streamer already exists and a client is trying to add/spawn it, the client will simply be registered to the alraedy runnning streamer and no changes to the streamer configuration is made (even if the clinet's wanted config is different). now new events from this streamer will also be sent to the newly added client.

a client can shut/(stop listening to) a streamer, in this case, if the streamer has no more clients to server, it will be terminated. otherwise, it's kept alive to sever other clients.

a clinet can also terminate the sse connection (remove_client), in which case all of its streamers are shut (same as the shut point above)

this will allow us to reason about what feilds must be locked together for atomicity of streaming manager operations.

this also gets rid of the smart channel receiver that removes a channel from the controller when it drops. it's now the responsibilty of who called `create_channel` to call `remove_channel` when the channel is no longer used (that is, the responsibility of the streaming manager)

This refactor makes sure that we perform write operations on the streaming manager atomically.

We were previouly locking some fields of the manager to edit them then unlock them and lock another field. This clearly isn't atomic since there is a depedency between the data stored in the two fields and they should be set in one step with no operation in between.

One solution could have been to just wrap the whole manager with a mutex and not allow concurrent access to it. This would hurt the read perf though, which is critical for frequent send calls and broadcast calls.
Another would have been introducing a new field (mutex) called `critical` or something and lock it when we want no other writing method to execute in parallel. This is the approach taken here, but since `clinets` field is already locked with every operation (every operation mutating the state of the streaming manager must involve a client anyway: added, removed, newly listening, etc...), we now lock the `clients` field before any cirtical section till the end so we are sure no other mutating code will run in parallel.

since events now pass through the streaming manager (who has all info regarding which clients are interested in what events) we don't need such a filter.
also this filter won't work with the sub/pub arch anyways

they are the same thing, keeping the two names visible so their equivliance is clear

this is still faulty though. we need to only add a streamer on client request and not out of the blue. ctrl+f `add(0,`

changed from:
'event_streaming_configuration' : { 'worker_path': 'path/to/worker.js', ..other_params }
to
'event_stream_worker_path': 'path/to/worker.js'

If a streamer is down (we wrote an event streamer that doesn't loop forever for some reason, or maybe the streamer paniced) we should start it agian the next time it is activated by any client, so the newly listening client gets the actual stream of events.

Note that we don't keep older clients listening to that streamer, because:
1- the streamer writer probably wanted this effect, if the streamer is dead, clients should get no more events.
2- streamer initialization might fail, and if we keep the clients listening we will end up with `clients` & `streamers` maps out of sync. if we remove the clients if the streamer fails (and only if it fails), this will be a highly inconsistent side-effect.

We should make sure streamers never exit (unless this is the wanted logic).
For other cases like when the spawner used by the streamer is aborted (e.g. utxo balance streamer running on utxo coin spawner and then we disable the coin), we should make sure to use the approbriate spawner to exit only when the streamer's dependecy is no longer present.
We should expect clients to re-enable the streamer is such cases (e.g. [enable utxo coin, enable blanace streamer, disable utxo coin, enable utxo coin] scenario doesn't re-enable the balance streamer by itself, the client should do [enable balance streamer] one more time)

`ask_for_data` didn't use the timeout that was passed over.
the only way for it to timeout was for a client to reply for the asked data *after* the timeout

enabling and disabling streamers now take place in
mm2_main/rpc/streaming_activations

for modularity, we shouldn't enable streamers elsewhere so not to forget about them

I went with removing the fee estimation context and rpc methods but I
fount that it is used by mm2 itself and not just for API usage.

Maybe with some refactor we could get the mm2 usage decoupled from the
API one, and abandon the fee context (it looks possible from the
streaming API side).

they are already in a streaming_activations module (already tells what these functions are for), so cut the names short

also fixes a bug where the eth fee streamer was spawned on the ctx weakspawner

this only streams the specific event that took place along with the UUID since most events don't embed the UUID. This doesn't work like `swap_v2_rpcs::SwapRpcData` where all the past events are send in a vector

access_control_allow_origin config moved from:
'event_streaming_configuration' : { 'access_control_allow_origin': '*', ..other_params }
to
'access_control_allow_origin': '*'
i.e. moved one scope out in the mm2.json config

moved to a TODO, we need #1966 first to start working on it, otherwise it's gonna look like this: 8972822
not removing the ScripthashNotificationSender traces entierly so not to have a lot of merge conflicts with #1966, but it's now disabled (provided always as None).

they all share the same request and response and don't depend on the path

to show up in review if it's important :)

since there is no such a streamer id (the id was made up on the spot in the data asker) the client who should receive this message won't get it (not any client in fact). it seems like we can't somehow know the client, so sending the request to all clients instead.

…rror

in the streamed error message

and only clone it if the streamer exists (and make sure it injests inputs [data_in], but this really shouldn't happen as this would be a dev error

This reverts commit bfb15da.

Since we use custom enums for the streamers inputs that encapsulates other mm2 types. We ended up cloning the mm2 types to construct the enum and then clone the enums from the reference if the streamer exists. so we ended up double cloning instead of cloning just one time.

since we expect `StreamingManager::send` & `StreamingManager::broadcast` to be called alot thus holding read locks to `streamers` and/or `controller`, we might end up in a situation where a write upate can't take place because of a queue of readers

rely only on the streaming version

`gas_fee_estimator` was removed from coins file and is now set while enabling the streaming for the first time. One can choose to enable it in 'simiple' or 'provider' mode.
provider mode uses 'gas_api' defined in mm2.json, if not found we always fail back to simple again.

so that we won't create message/notification we want to send to the streamer unless the streamer actually exists and accepts messages (if it doesn't accept messages, i.e. periodic streamer, this would definitely be a dev error)

this adds tx history streaming for tx history supporting coins (except zcoin).
the streamer is fairly simple, just a forwarder that doesn't do any computation. The tx details are grapped from the tx history state machine and forwared to the streamer right before getting stored in the db.

To be able to access the `event_stream_manager` from the tx history statemachine, `MmArc` was made accessible through the coins (`MmCoin`). Chose the coins here and not directly embedding the MmArc inside the history statemachine since we will probably use the event streamer more and more in different areas of the code, so having it distributed among coins would make adding new streaming functionalies easier.
We could have also gone for cloning `event_stream_manager` to the coins instead of passing the `ctx`. Went for passing the ctx here since eth & tindermint already pass the ctx, so to avoid redundency and unify coins structure.

and not electrum. This so the error could be reported back to the client and not silently ignored.

P.S. we could support native rpc balance events using a periodic polling technique like the one we have in eth for example

this is still missing the translation from txhash to json tx details (this need to be fetched from the db).

this commit also reactivates the z balance streaming by using the streaming manager to send the balance change notification to the z balance event streamer (instead of sending the notification on the channel which no body was listening to after balance streaming refactor).

inlined the controller logic right inside the streaming manager. it was simple enough to have it's own struct + having it essentially replicated the clients hashmap for no reason

…ents

finally :)

only for native, wasm to come

outdated urls, a runtime panic, non-zero scan blocks per iteration

clean, fmt & clip

impl zcoin activation params with lightmode as default since native won't work on wasm

the ctx wasn't hooked up, unwrap panics when accessing the streaming manager in the background thread

i confused min with max when fixing this potential panic. max is the one we want here, so that when we have 0.max(1) we get 1 and not 0

without any streamer involved

this is a generic impl (and i think is better), and doesn't require a subscription (because there are no streamers running for it).

the flow goes as follows:
- the client inits a task managed request
- the client (all clients actually) receive updates once they happen (either action required, request resolution (ok, err), etc...)

later we should make it so that the client ID is provided in the init request, but optionally, and if the client provides it, only them will receive the updates, otherwise we can choose to broadcast_all or not broadcast at all.
Note that spawning a streamer for each single task and hooking it up to a single client is tedious, for this we can instead broadcast the udpate rightaway from the rpc task manager knowing the client ID using the new `broadcast_to`.

`task::` API init messages now have an optional `client_id` which can be
used to stream out task status upadates and results to the client via
event streaming.

the client must have already done the inital SSE request in case of
native and they must use the same client_id they used in that SSE init
request.