Running a Websocket client in a different thread or process alongside other tasks/code executing in the program

[n-srinidhi]

HI!I am using Autobahn for websockets to send data to and from between a client and a server. I have a python script where the WebSocketClient is present, along with many other methods. The script is fetching data from multiple places as well as data received from the server to the client at periodic intervals, processing it and sending it forward. Problem is when I am starting the WebSocketClient process any and all code after that is blocked and doesn't execute. I have combed the links and solutions available online to resolve this but to no avail. Any help here would be really appreciated! Thank you!

[meejah]

Autobahn is event-based so you don't usually use threads to solve concurrency problems like this. You can use either Twisted or asyncio APIs to achieve multiple data streams without threads.

[oberstet]

yes, as meejah says, Autobahn uses asynchronous programming with Twisted or asyncio, and hence your program needs to be structured along that - in case you haven't found it yet, there is a ton of examples you can look and steal from here https://github.com/crossbario/crossbar-examples

[n-srinidhi]

I am doing something like this now:

if __name__ == '__main__':
        client=threading.Thread(target=ConnectToManagerServer,args=(url,host,port))
        client.daemon=True
        client.start()
        StartEngineServer()

Where ConnectToManagerServer is my autobahn web socket method with the boilerplate code to start and register the WebSocketClient.
StartEngineServer() is an independent method that is going to process the data received by the web socket, process it and send it forward.
Is this the right way to do it?

[oberstet]

no, this is wrong. don't use threads.

[n-srinidhi]

Thank you for your response. I have tried different ways but none seem to work! :( Not using threads keeps my other code from executing and only the web socket part is active.
P.S : The StartEngineServer() Method also has TCP sockets
How about I start the second web socket server after the OnConnect() callback in the previous ConnectToManagerServer() method? I will have separate client and server protocols defined within the same python script, so respective callbacks will be called when requests arrive on different ports.

[oberstet]

Not using threads keeps my other code from executing and only the web socket part is active.

You should aim for having all of your "other code" itself non-blocking and using Twisted or asyncio (eg StartEngineServer would return a Deferred or Future). This is the "right" way.

If this is not the case, e.g. your StartEngineServer is using plain Python socket API, which is blocking, things will get complicated.

One way in this case is to run StartEngineServer on a background thread and interact from that with Twisted/asyncio/WebSocket/Autobahn running on the main thread.

That interaction from the background thread with the main thread - which is running the Twisted/asyncio reactor/event loop must be done using Twisted/asyncio provided special methods (you cannot just call from one thread to the other).

E.g. in Twisted there is deferToThread and callFromThread.

I'm sorry, but I can't teach you all the details needed for making blocking code work in the context of a non-blocking asynchronous framework like Twisted or asyncio ... this is kinda advanced stuff ... but the notes should give you some clues.

Of course you could also rewrite your blocking code in StartEngineServer to proper non-blocking asynchronous code - that would be the best anyways (better than working around by using deferToThread etc), but the work this would require depends on your code.

hope this helps.

[n-srinidhi]

Understood. I'll research on this. Thanks a lot for your pointers! They have been super helpful.