Watcher fails because empty vault is used during re-authentication

[aa2811]

Long story short

We are seeing issues where our watcher fails after the first credential expiration, because the empty vault is used during re-authentication.

From code reading, it looks like the _select function should block during re-authentication, but logs show this isn't happening.

We are using the piggybacking authentication with the login_with_service_account method

Kopf version

1.37.2

Kubernetes version

1.30 (AKS)

Python version

3.9.19

Code

No response

Logs

{"message": "Re-authentication has been initiated.", "timestamp": "2024-10-16T17:40:00.488688+00:00", "severity": "info"}
{"message": "Activity 'login_with_service_account' is invoked.", "timestamp": "2024-10-16T17:40:00.488932+00:00", "severity": "debug"}
{"message": "Stopping the watch-stream for servicemonitors.v1.monitoring.coreos.com cluster-wide.", "timestamp": "2024-10-16T17:40:00.489383+00:00", "severity": "debug"}
{"message": "Watcher for servicemonitors.v1.monitoring.coreos.com@none has failed: Ran out of valid credentials. Consider installing an API client library or adding a login handler. See more: https://kopf.readthedocs.io/en/stable/authentication/", "exc_info": "Traceback (most recent call last):\n  File \"/opt/python/venvs/heracles/lib/python3.9/site-packages/kopf/_cogs/aiokits/aiotasks.py\", line 96, in guard\n    await coro\n  File \"/opt/python/venvs/heracles/lib/python3.9/site-packages/kopf/_core/reactor/queueing.py\", line 175, in watcher\n    async for raw_event in stream:\n  File \"/opt/python/venvs/heracles/lib/python3.9/site-packages/kopf/_cogs/clients/watching.py\", line 86, in infinite_watch\n    async for raw_event in stream:\n  File \"/opt/python/venvs/heracles/lib/python3.9/site-packages/kopf/_cogs/clients/watching.py\", line 201, in continuous_watch\n    async for raw_input in stream:\n  File \"/opt/python/venvs/heracles/lib/python3.9/site-packages/kopf/_cogs/clients/watching.py\", line 266, in watch_objs\n    async for raw_input in api.stream(\n  File \"/opt/python/venvs/heracles/lib/python3.9/site-packages/kopf/_cogs/clients/api.py\", line 200, in stream\n    response = await request(\n  File \"/opt/python/venvs/heracles/lib/python3.9/site-packages/kopf/_cogs/clients/auth.py\", line 48, in wrapper\n    async for key, info, context in vault.extended(APIContext, 'contexts'):\n  File \"/opt/python/venvs/heracles/lib/python3.9/site-packages/kopf/_cogs/structs/credentials.py\", line 158, in extended\n    async for key, item in self._items():\n  File \"/opt/python/venvs/heracles/lib/python3.9/site-packages/kopf/_cogs/structs/credentials.py\", line 195, in _items\n    yielded_key, yielded_item = self.select()\n  File \"/opt/python/venvs/heracles/lib/python3.9/site-packages/kopf/_cogs/structs/credentials.py\", line 214, in select\n    raise LoginError(\"Ran out of valid credentials. Consider installing \"\nkopf._cogs.structs.credentials.LoginError: Ran out of valid credentials. Consider installing an API client library or adding a login handler. See more: https://kopf.readthedocs.io/en/stable/authentication/", "timestamp": "2024-10-16T17:40:00.490222+00:00", "severity": "error"}
{"message": "Activity 'login_with_service_account' succeeded.", "timestamp": "2024-10-16T17:40:00.491191+00:00", "severity": "info"}
{"message": "Re-authentication has finished.", "timestamp": "2024-10-16T17:40:00.491303+00:00", "severity": "info"}

Additional information

If it's useful, we're using aiohttp==3.10.10