Fix CI with postgres #954
Labels
1.4.0
bug
Something isn't working
High Priority
Current Priorities & Blocking Issues
testing
testing
Comments
|
|
Cannot reproduce any more. |
github-project-automation
bot
moved this from TODO
to Done
in Fractal Project Management
github-project-automation
bot
moved this from Done
to TODO
in Fractal Project Management
|
Perhaps fixed with 1008? To be verified. EDIT: no |
|
It looks like we are really hitting this one issue of asyncpg: |
tcompa
added a commit
that referenced
this issue
Nov 21, 2023
github-project-automation
bot
moved this from TODO
to Done
in Fractal Project Management
|
Unexpectedly, this bug appeared again (ref https://github.com/fractal-analytics-platform/fractal-server/actions/runs/6966759281/job/18957429979), even after #1012. One more reason to consider #996 |
github-project-automation
bot
moved this from Done
to TODO
in Fractal Project Management
1 task
|
Possibly relevant entry in sqlalchemy 1.4.41 CHANGELOG (https://docs.sqlalchemy.org/en/20/changelog/changelog_14.html#change-1.4.41-asyncio):
|
|
(with @mfranzon) Ref: where we upgraded asyncpg from 0.27 to 0.29. Possibly related (as of 0.29): MagicStack/asyncpg#1087 and the addition of . Errors still appear when running the CI locally (see below), but they are not re-raised and the tests pass. Note the difference between Another relevant bit of info is
poetry run pytest -s --log-cli-level info tests/test_*_api.py
...
Results (78.27s):
95 passed
1 warning
ERROR:asyncio:Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7fa737542080>
transport: <_SelectorSocketTransport closing fd=18>
Traceback (most recent call last):
File "/usr/lib/python3.10/asyncio/selector_events.py", line 924, in write
n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.10/asyncio/sslproto.py", line 690, in _process_write_backlog
self._transport.write(chunk)
File "/usr/lib/python3.10/asyncio/selector_events.py", line 930, in write
self._fatal_error(exc, 'Fatal write error on socket transport')
File "/usr/lib/python3.10/asyncio/selector_events.py", line 725, in _fatal_error
self._force_close(exc)
File "/usr/lib/python3.10/asyncio/selector_events.py", line 737, in _force_close
self._loop.call_soon(self._call_connection_lost, exc)
File "/usr/lib/python3.10/asyncio/base_events.py", line 753, in call_soon
self._check_closed()
File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
ERROR:asyncio:Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7fa7378a51b0>
transport: <_SelectorSocketTransport closing fd=8>
Traceback (most recent call last):
File "/usr/lib/python3.10/asyncio/selector_events.py", line 924, in write
n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.10/asyncio/sslproto.py", line 690, in _process_write_backlog
self._transport.write(chunk)
File "/usr/lib/python3.10/asyncio/selector_events.py", line 930, in write
self._fatal_error(exc, 'Fatal write error on socket transport')
File "/usr/lib/python3.10/asyncio/selector_events.py", line 725, in _fatal_error
self._force_close(exc)
File "/usr/lib/python3.10/asyncio/selector_events.py", line 737, in _force_close
self._loop.call_soon(self._call_connection_lost, exc)
File "/usr/lib/python3.10/asyncio/base_events.py", line 753, in call_soon
self._check_closed()
File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
ERROR:asyncio:Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7fa746081cf0>
transport: <_SelectorSocketTransport closing fd=9>
Traceback (most recent call last):
File "/usr/lib/python3.10/asyncio/selector_events.py", line 924, in write
n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.10/asyncio/sslproto.py", line 690, in _process_write_backlog
self._transport.write(chunk)
File "/usr/lib/python3.10/asyncio/selector_events.py", line 930, in write
self._fatal_error(exc, 'Fatal write error on socket transport')
File "/usr/lib/python3.10/asyncio/selector_events.py", line 725, in _fatal_error
self._force_close(exc)
File "/usr/lib/python3.10/asyncio/selector_events.py", line 737, in _force_close
self._loop.call_soon(self._call_connection_lost, exc)
File "/usr/lib/python3.10/asyncio/base_events.py", line 753, in call_soon
self._check_closed()
File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
ERROR:asyncio:Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7fa737d1fe50>
transport: <_SelectorSocketTransport closing fd=10>
Traceback (most recent call last):
File "/usr/lib/python3.10/asyncio/selector_events.py", line 924, in write
n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.10/asyncio/sslproto.py", line 690, in _process_write_backlog
self._transport.write(chunk)
File "/usr/lib/python3.10/asyncio/selector_events.py", line 930, in write
self._fatal_error(exc, 'Fatal write error on socket transport')
File "/usr/lib/python3.10/asyncio/selector_events.py", line 725, in _fatal_error
self._force_close(exc)
File "/usr/lib/python3.10/asyncio/selector_events.py", line 737, in _force_close
self._loop.call_soon(self._call_connection_lost, exc)
File "/usr/lib/python3.10/asyncio/base_events.py", line 753, in call_soon
self._check_closed()
File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
ERROR:asyncio:Fatal error on SSL transport
protocol: <asyncio.sslproto.SSLProtocol object at 0x7fa737f59f60>
transport: <_SelectorSocketTransport closing fd=11>
Traceback (most recent call last):
File "/usr/lib/python3.10/asyncio/selector_events.py", line 924, in write
n = self._sock.send(data)
OSError: [Errno 9] Bad file descriptor
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.10/asyncio/sslproto.py", line 690, in _process_write_backlog
self._transport.write(chunk)
File "/usr/lib/python3.10/asyncio/selector_events.py", line 930, in write
self._fatal_error(exc, 'Fatal write error on socket transport')
File "/usr/lib/python3.10/asyncio/selector_events.py", line 725, in _fatal_error
self._force_close(exc)
File "/usr/lib/python3.10/asyncio/selector_events.py", line 737, in _force_close
self._loop.call_soon(self._call_connection_lost, exc)
File "/usr/lib/python3.10/asyncio/base_events.py", line 753, in call_soon
self._check_closed()
File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed |
|
A very promising option to play with is We ran the
This suggests that |
tcompa
added a commit
that referenced
this issue
Nov 29, 2023
1 task
|
For the record, the CI can still fail - even with the |
1 task
|
As part of sqlalchemy 2.0.2, there is this change (https://docs.sqlalchemy.org/en/20/changelog/changelog_20.html#change-72bab6f6b9eb1662c0f5c5030a44f184):
See also the corresponding issue/discussion/commit: |
|
In #1044, by setting event pool The background error logs (as shown in #954 (comment)) are still present, but they are less worrying. For the record, even if they are not always reproducible, some tests suggest that:
|
github-project-automation
bot
moved this from TODO
to Done
in Fractal Project Management
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
See failures in #953.
For the record, as of #949 we are using sqlalchemy 1.4.50 in the CI (while we were using 1.4.41 before that PR).
Ref:
The text was updated successfully, but these errors were encountered: