Zombie Sockets

[JMANN240]

I am running into a bug where the server holds sockets long after the client has disconnected. This leads to endless message buffering, ending in an InternalChannelFull error. I would make an issue, but the bug is not easily reproducible: I don't know what causes it, but it consistently happens when my server is running (usually 12+ hours after starting it).

I have spent hours looking at my code and haven't been able to find what could be introducing this bug causing some sockets to be held for longer than they should be. No deadlocks, no persistent references to sockets themselves, nothing.

I just wondered if anyone has experienced a similar issue and might how to fix it, or if anyone has any debugging ideas.

This code is here https://github.com/JMANN240/dots-not-bots

[Totodore]

Hey! Looking at you're code it seems you do not hold any reference to a socket so it might be a problem from the socketioxide lib.
Did you succeed to get zombie sockets before 12h by artificially increase app usage? Or do you think it is purely related to a specific duration.

What would be nice is to print all sockets periodically and check if they are disconnected or if there channel is full or empty. Also do you know how the sockets are disconnected, is it by brutally closing the connection, by closing it from the server or the client? This could help me to check if I can find something in the corresponding code.

[JMANN240]

Hello, thanks for your reply! I just checked, and running it locally, if I overload the server (holding down F5 to refresh the page) this happens as well. This makes me think it is not specifically related to duration, but some event that only happens after a while during normal use.

Using some logging, I was able to determine that all sockets say that they are connected, but I know that is not true. The SocketIo instance is holding sockets that should have been removed long ago, buffering messages to them (which they can't send because they aren't actually connected), and causing the internal channel to fill up.

The sockets are disconnected by just closing the connection when closing a tab or the web browser, there is no graceful disconnection code in place.

For some reason, the server is sometimes not removing sockets when they disconnect.

[JMANN240]

You can find all of the code here https://github.com/JMANN240/dots-not-bots

I added a readme for how to get it up and running, but let me know if you have any issues because I might have missed something.

[JMANN240]

Hello, I just wanted to check in and see if there was any progress made on this. If not that's ok, I just wasn't sure. Thanks.

[Totodore]

I didn't have the time but it's on my to-do list

[JMANN240]

Got it, no worries. Thanks!

[Totodore]

Hey @JMANN240 I tried your example but I could not reproduce the issue even when reducing the internal channel buffer size.

The only thing I observed is that when you smash F5 sockets don't have the time to upgrade so they stay in HTTP polling mode, therefore they are not eagerly closed after you refresh the page but only after the hearbeat_timeout. Did you also try to decrease the heartbeat timeout / heartbeat interval so that polling sockets are removed faster if they do not respond?

I tried to improve the closing mechanism for websockets on #579. Could you try to use this branch to test on your side and tell me if you still see the issue? If so, could you add the tracing to the socketioxide deps and enable logging with:
RUST_LOG=socketioxide=trace,engineioxide=trace and tracing_subscriber env filter and send it here.

Side note: There is one internal channel buffer per socket, so it is quite weird that the issue comes from here. Do you have a lot of traffic on your app? Did you try to disable polling and immediately connect through websocket with your socketio client?

Thanks for your help!

[JMANN240]

Thanks for the update.

I have not tried decreasing the heartbeat timeout, but the unresponsive socket stays around for more than 20 seconds, so I am not sure that would help.

Building my application from that branch seems to have fixed the issue! The changes you made must have fixed whatever the issue was.

Side note answer: I wouldn't say my app has an exceptional amount of traffic. And I have not tried to disable polling... just tried it and that also seems to fix the issue. It seems that my issue must be specific to HTTP polling sockets before they get upgraded to websocket.