Clearing buffers and releasing memory

[n-srinidhi]

Hi! I'm currently having memory overload issues while using autobahn web socket client. The program executes correctly and shuts down with the following error after a few hours:

--- Logging error ---
Traceback (most recent call last):
  File "<frozen centra_server>", line 84, in onMessage
MemoryError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\logging\__init__.py", line 1100, in emit
    msg = self.format(record)
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\logging\__init__.py", line 943, in format
    return fmt.format(record)
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\logging\__init__.py", line 681, in format
    s = self.formatMessage(record)
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\logging\__init__.py", line 650, in formatMessage
    return self._style.format(record)
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\logging\__init__.py", line 440, in format
    return self._format(record)
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\logging\__init__.py", line 436, in _format
    return self._fmt % values
MemoryError
 
File "<frozen centra_server>", line 340, in <module>
  File "<frozen centra_server>", line 330, in main
  File "<frozen centra_server>", line 270, in StartClient
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\site-packages\twisted\internet\base.py", line 693, in run
    self.mainLoop()
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\site-packages\twisted\internet\base.py", line 706, in mainLoop
    self.doIteration(t)
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\site-packages\twisted\internet\selectreactor.py", line 142, in doSelect
    _logrun(selectable, _drdw, selectable, method)
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\site-packages\twisted\python\log.py", line 96, in callWithLogger
    return callWithContext({"system": lp}, func, *args, **kw)
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\site-packages\twisted\python\log.py", line 80, in callWithContext
    return context.call({ILogContext: newCtx}, func, *args, **kw)
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\site-packages\twisted\python\context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\site-packages\twisted\python\context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\site-packages\twisted\internet\selectreactor.py", line 148, in _doReadOrWrite
    why = getattr(selectable, method)()
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\site-packages\twisted\internet\tcp.py", line 248, in doRead
    return self._dataReceived(data)
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\site-packages\twisted\internet\tcp.py", line 253, in _dataReceived
    rval = self.protocol.dataReceived(data)
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\site-packages\autobahn\twisted\websocket.py", line 348, in dataReceived
    self._dataReceived(data)
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\site-packages\autobahn\websocket\protocol.py", line 1243, in _dataReceived
    self.consumeData()
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\site-packages\autobahn\websocket\protocol.py", line 1255, in consumeData
    while self.processData() and self.state != WebSocketProtocol.STATE_CLOSED:
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\site-packages\autobahn\websocket\protocol.py", line 1619, in processData
    fr = self.onFrameEnd()
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\site-packages\autobahn\websocket\protocol.py", line 1747, in onFrameEnd
    self._onMessageEnd()
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\site-packages\autobahn\twisted\websocket.py", line 384, in _onMessageEnd
    self.onMessageEnd()
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\site-packages\autobahn\websocket\protocol.py", line 647, in onMessageEnd
    self._onMessage(payload, self.message_is_binary)
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\site-packages\autobahn\twisted\websocket.py", line 387, in _onMessage
    self.onMessage(payload, isBinary)
  File "<frozen centra_server>", line 224, in onMessage
Unable to print the message and arguments - possible formatting error.
Use the traceback above to help find the error.
Unhandled Error
Traceback (most recent call last):
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\site-packages\twisted\python\log.py", line 96, in callWithLogger
    return callWithContext({"system": lp}, func, *args, **kw)
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\site-packages\twisted\python\log.py", line 80, in callWithContext
    return context.call({ILogContext: newCtx}, func, *args, **kw)
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\site-packages\twisted\python\context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\site-packages\twisted\python\context.py", line 82, in callWithContext
    return func(*args, **kw)
--- <exception caught here> ---
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\site-packages\twisted\internet\selectreactor.py", line 148, in _doReadOrWrite
    why = getattr(selectable, method)()
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\site-packages\twisted\internet\tcp.py", line 248, in doRead
    return self._dataReceived(data)
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\site-packages\twisted\internet\tcp.py", line 253, in _dataReceived
    rval = self.protocol.dataReceived(data)
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\site-packages\autobahn\twisted\websocket.py", line 348, in dataReceived
    self._dataReceived(data)
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\site-packages\autobahn\websocket\protocol.py", line 1243, in _dataReceived
    self.consumeData()
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\site-packages\autobahn\websocket\protocol.py", line 1255, in consumeData
    while self.processData() and self.state != WebSocketProtocol.STATE_CLOSED:
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\site-packages\autobahn\websocket\protocol.py", line 1619, in processData
    fr = self.onFrameEnd()
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\site-packages\autobahn\websocket\protocol.py", line 1747, in onFrameEnd
    self._onMessageEnd()
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\site-packages\autobahn\twisted\websocket.py", line 384, in _onMessageEnd
    self.onMessageEnd()
  File "C:\Users\centraadmin\AppData\Local\Programs\Python\Python310\lib\site-packages\autobahn\websocket\protocol.py", line 644, in onMessageEnd
    payload = b''.join(self.message_data)
builtins.MemoryError:

Is there a way to clear the payload or data buffers everytime a data packet is received in the client websockets?
Note that we are receiving data every second from a web socket server (Which is based in .net).

[oberstet]

you can certainly force to collect garbage and free memory in CPython using

>>> import gc
>>> gc.collect()

with no harm.

you might also want to track down the actual source of the issue, e.g. are you using any native Python libraries, if so, which? CPyExt or CFFI based? I am asking because it is quite unlikely originating in CPython itself (my experience) but more likely in native code related stuff

what are the machine specs, and what are the Python interpreter specs btw?

you can also just buy more RAM and then hope stuff doesn't break for the duration / uptime required, and you can also install a decent OS when you are updating your machine as well;)

[n-srinidhi]

Hi, Using GC.collect() did not help with this. The machine has around 40GB RAM, I'm unable to troubleshoot if the memory error is from the web socket server side or the client program itself. How do i troubleshoot the same?

Machine specs:
Processor: Intel Gold 6144 CPU @ 3.50GHz (2 processors)
64 Bit operating System ,with 40 GB RAM

[oberstet]

did not help with this

"help" is arbitrary, hence is "did not". show some numbers. what OS and WAMP client-side / router statistics did you track? post some charts and detailed test setup and testing configuration and conditions .. post a complete reproducing test ..

64 Bit operating System

I guess you mean "windows". install a proper OS, Linux, is what I meant. this is good in general. but it likely isn't the source of your issues anyways. I am guessing, just my experience - this is without knowing specifics.

if the memory error is from the web socket server side

The issues are very unlikely from any code in AutobahnPython which has been proven in practice under massive load and in production.

How do i troubleshoot the same?

why are you asking here? this forum is about specific and directly related questions rgd Autobahn(Python) and WAMP. if you think there is a bug in Autobahn, please post code and a complete & minimal example that reproduces the issue claimed in a new issue. Thanks!