Memory Leak in ZettaHttpServer - PeerSockets

[kevinswiber]

When one Zetta peer connects to another, the receiver stores a PeerSocket in its ZettaHttpServer#peers object. When the initiating peer disconnects, its PeerSocket remains on the receiving peer. This was likely an intentional performance optimization for use cases that only ever have a few of the same peers connecting. In scenarios where hundreds, perhaps thousands of peers are connecting, the receiving peer's ZettaHttpServer#peers object continues to grow. Some of these peers may disconnect and never return.

We have a few options to address this:

1. delete the peer from ZettaHttpPeers#peers on _peer/disconnect
2. Set a buffer limit for dead PeerSocket instances. Prune this buffer on an interval.
3. The same as Option 1, except with a timeout that is fired should the initiating peer not reconnect during a specified time duration.

[kevinswiber]

@AdamMagaluk @mdobson Thoughts? This is currently an issue in our custom cloud deployment.

@landlessness, you may be experiencing some ill effects from this, too.

[kevinswiber]

A workaround, in the meantime, implementing Option 1 above:

zetta()
  .name('cloud')
  .use(function(runtime) {
    runtime.pubsub.subscribe('_peer/disconnect', function(ev, msg) {
      delete runtime.httpServer.peers[msg.peer.name];
    });
  })
  .listen(process.env.PORT || 1337);

[AdamMagaluk]

@kevinswiber The main (possibly only reason, I forget) we are keeping the peer_sockets around is to keep the list of event subscriptions from clients/apps to that peer and subscribe to them when they reconnect.

Of course this can be done differently by using the event_broker or other internals to get a list of subscriptions for that peer. We could also delete the peer once all of it's subscriptions are gone (clients disconnecting), though I don't think that would ever be the case for our deployment.

[kevinswiber]

@AdamMagaluk Thanks for the assessment. Let's abstract that away. What can we do about unbounded growth on leftover subscriptions? Do we need to institute a timeout between the lack of availability of connected device streams and the subscriptions to those streams? Is there a straightforward solution? Perhaps we could use a circular buffer that holds dead subscriptions and starts dropping them on overflow?

Unbounded growth is not only hurting memory consumption; it's also an attack vector.

[AdamMagaluk]

@kevinswiber I think the best approach would be to delete the peer_socket from .peers after a disconnect and use the data available from the clients/apps to re-subscribe to topics on the peer when a new peer_socket is created.

In this case zetta would only keep subscriptions and peers when they are currently connected. We would that just have to worry about an unbounded number of peers connecting, clients connecting, and subscriptions from clients but that solved pretty easily by some limits.

I think it would be pretty straightforward, may take a bit code and figuring out how to access the proper data but it's all there.