Airflow 2.9.3 , KubernetesExecutor , worker pod creation in batch.

[nitinmahawadiwar]

Hello team,
I am trying to understand how to reduce the pod creation time in case of KubernetesExecutor, specially a case when parallel tasks are more and each task runs in separate pod.

Google search results following two websites about the variable, but I couldn't find this variable in 2.9.3 source. Is this still applicable or the variable is changed/no longer supported. ?

https://hevodata.com/learn/airflow-parallelism/
AIRFLOW_KUBERNETES_WORKER_PODS_CREATION_BATCH_SIZE (extra underscore is missing)

https://www.astronomer.io/docs/learn/airflow-scaling-workers/
AIRFLOW__KUBERNETES__WORKER_PODS_CREATION_BATCH_SIZE

can you please suggest appropriate documentation for 2.9.3 version. ?

[nitinmahawadiwar]

@potiuk can you please help me here. or redirect to someone else ?

[potiuk]

Searched and found https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/configurations-ref.html#worker-pods-creation-batch-size - but I do know too well how/whether it works.

Please remember this forum is a free forum of people who are helping others in their free time.

Please do not ping people to "find" someone else. People here help when they feel like it and have time and feel they can help - we have no structure of hierarchy here - everyone contributes individually - If you need to ping someone specifically, you can look at the source code and see if they were involved in related code and you may - in justified cases ping them directly - but pinging me in this case is just abusing the fact that I am more active than others in helping (but it does not mean I have an infinite time to help everyone with everything or find another person for them).

In this case I just searched for the name you mentioned and found it. If you did a bit of search yourself more, you could have probably saved a week. I do not have deep knowledge about how it works - you need to find out on your own and maybe ping people who are related.

Generic "ping for help" is way better if you do not get answer (and sometimes you will not get it even if you ping several times - there is no guarantee). But also by pinging individuals, you lower your chances to get help from others - because even if someone could help, the others pinged will help - but they might be busy or on holidays or simply not feel like they can help because they were not - like in this case involved in that part.

[nitinmahawadiwar]

Sorry to mention/ping you directly. I respect your time and acknowledge that I should not have done this.
Thanks