Support defining Kubernetes resources without Kubernetes API data types and with list and dicts

[ketozhang]

Description

KubernetesPodOperator and operator's pod_override support the K8s Python API data models/types and in some cases also supports passing in Python standard data types (e.g., env_vars, cmds, etc).

It would be nice to expand the cases to allow defining all K8s spec with Python standard data types instead of using the API's data types. Users would have to sacrifice type integrity, but they have options to use type hinting (e.g., with the K8s type stubs package).

Current alternative is to rely on pod_template_dict.

Use case/motivation

• pod_override: dict | V1Pod
• ports: list[int] | list[V1ContainerPort]
• volume_mounts: list[dict] | list[V1VolumeMount]
• volumes: list[dict] | list[V1Volume]
• and other KPO fields ...

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[ketozhang]

I'd like to contribute to this, but I am unaware if this was attempted before and perhaps some limitation in the K8s API prevents this. Naively, I don't understand what possible implementation detail prevents us from doing:

One option is to rely on a private method as it's how pod_template_dict is implemented, but beyond deserializing V1Pod,

airflow/airflow/providers/cncf/kubernetes/pod_generator.py

Lines 559 to 572 in ba053f7

	@staticmethod
	def deserialize_model_dict(pod_dict: dict | None) -> k8s.V1Pod:
	"""
	Deserializes a Python dictionary to k8s.V1Pod.
	Unfortunately we need access to the private method
	``_ApiClient__deserialize_model`` from the kubernetes client.
	This issue is tracked here; https://github.com/kubernetes-client/python/issues/977.
	:param pod_dict: Serialized dict of k8s.V1Pod object
	:return: De-serialized k8s.V1Pod
	"""
	api_client = ApiClient()
	return api_client._ApiClient__deserialize_model(pod_dict, k8s.V1Pod)

There are few others options involving writing our own deserializer or just serialize to YAML and pass it similar to pod_template_file.

[dirrao]

@ketozhang
Right now the pod operator is overloaded with too many details. It would be great if you can simplify it.

[dimberman]

@ketozhang can you make a DAG example showing how this would look? We had something kind of like this in earlier versions of the k8sexecutor and it did not go very well. I'm open to hearing more but want to be careful.

[cmarteepants]

This sounds similar to how KPOs and k8sexecutor configs were defined pre-2.0. See #10393 for context for this was removed. Simplifying would be amazing, but agree with @dimberman that we need to tread carefully. I am going to convert this into a discussion so we can brainstorm examples of how this will look.

[ketozhang]

Taken a look at #10393. I couldn't find much motivation. @dimberman mentioned some previous email thread. I know it's been many years; would you happen to have a link to this or perhaps a quick summary.

PS: I haven't forgotten about making an example. TBD...