KeyError: 'apiVersion' in Kopf 1.37.x with Kubernetes 1.30.x

[drexlar5]

Summary
I’m encountering a recurring KeyError: 'apiVersion' error in Kopf when it attempts to post events. This happens despite ensuring compatibility between Kopf, the Kubernetes Python client, and the Kubernetes server version.

Environment Details

• Kopf Version: 1.37.3
• Kubernetes Python Client Version: 30.5.0
• Kubernetes Server Version: v1.30.5-gke.1443001
• Kubernetes Client (kubectl) Version: v1.30.6-dispatcher

I’m running Kopf in a containerized environment, and the same issue is reproducible across different deployments.

Error Logs

Below are the logs I consistently see whenever that is a change to the custom applicationSet resource that kopf is watching:

Traceback (most recent call last):
File "/api/.venv/lib/python3.12/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 96, in guard
await coro
File "/api/.venv/lib/python3.12/site-packages/kopf/_core/engines/posting.py", line 171, in poster
await events.post_event(
File "/api/.venv/lib/python3.12/site-packages/kopf/_cogs/clients/events.py", line 35, in post_event
if ref['apiVersion'] == 'v1' and ref['kind'] == 'Event':
~~~^^^^^^^^^^^^^^
KeyError: 'apiVersion'
Traceback (most recent call last):
File "/api/.venv/bin/kopf", line 8, in
sys.exit(main())
^^^^^^
File "/api/.venv/lib/python3.12/site-packages/click/core.py", line 1157, in call
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/api/.venv/lib/python3.12/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/api/.venv/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/api/.venv/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/api/.venv/lib/python3.12/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/api/.venv/lib/python3.12/site-packages/kopf/cli.py", line 60, in wrapper
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/api/.venv/lib/python3.12/site-packages/click/decorators.py", line 92, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/api/.venv/lib/python3.12/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/api/.venv/lib/python3.12/site-packages/kopf/cli.py", line 109, in run
return running.run(
^^^^^^^^^^^^
File "/api/.venv/lib/python3.12/site-packages/kopf/_core/reactor/running.py", line 81, in run
asyncio.run(coro)
File "/usr/local/lib/python3.12/asyncio/runners.py", line 194, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/api/.venv/lib/python3.12/site-packages/kopf/_core/reactor/running.py", line 138, in operator
await run_tasks(operator_tasks, ignored=existing_tasks)
File "/api/.venv/lib/python3.12/site-packages/kopf/_core/reactor/running.py", line 419, in run_tasks
await aiotasks.reraise(root_done | root_cancelled | hung_done | hung_cancelled)
File "/api/.venv/lib/python3.12/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 226, in reraise
task.result() # can raise the regular (non-cancellation) exceptions.
^^^^^^^^^^^^^
File "/api/.venv/lib/python3.12/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 96, in guard
await coro
File "/api/.venv/lib/python3.12/site-packages/kopf/_core/engines/posting.py", line 171, in poster
await events.post_event(
File "/api/.venv/lib/python3.12/site-packages/kopf/_cogs/clients/events.py", line 35, in post_event
if ref['apiVersion'] == 'v1' and ref['kind'] == 'Event':
~~~^^^^^^^^^^^^^^
KeyError: 'apiVersion'

This error occurs during the posting of Kubernetes events, specifically when Kopf tries to access the apiVersion field of the ref object.

What I have tried

• Updated Versions for Compatibility:

  • Confirmed compatibility between Kopf, the Kubernetes Python client, and the Kubernetes server:
    • Kopf: 1.37.3
    • Kubernetes Python Client: 30.5.0
    • Kubernetes Server: v1.30.5
  • Upgraded Kopf and the Kubernetes Python client to align with Kubernetes server 1.30.x. However, the issue persists.

• Tested on Local KIND Environment:

  • Deployed the same operator on a local KIND cluster running Kubernetes server v1.31.x. The error does not occur in this setup, which suggests the issue might be specific to GKE or how it handles events.

Expected Behavior

Kopf should gracefully handle situations where the apiVersion field is missing from the ref object, or ensure that it is always populated during event posting.

Additional Notes

• The issue appears more prominently in GKE (Google Kubernetes Engine) than in KIND.
• This might be related to GKE-specific configurations or event structures.