Should you be able to override pools in backfill?

[dstandish]

Note: I originally posted this on slack but I'm reposting here for posterity

So i was going to implement the "pool override" behavior on backfill.

But it strikes me that, the option to provide one "pool override" that overrides pool for all tasks in the backfill, that just doesn't make a lot of sense.

Obviously different tasks may use different pools. so all-or-nothing override doesn't make a lot of sense.

And note, in the old backfill, it just straight up ignored pools unless you provided an override --- by default just no pools were used. this was surprising to me.

But ok, i don't really like that option.

so what's the alternative? we could allow user to provide a dict of tasks or something where you optionally override the pool for any specific task.

and then if no override is provided, then use the normal pool. this would work and make more sense.

but it seems like a rather kludgy interface.

WDYT?

Separately though, does anyone feel that backfill should still completely ignore pools by default? Anyone think that behavior should be preserved?

[dstandish]

Responses from slack:

raphaelauv
Yesterday at 1:22 PM
Backfill was ignoring pools , that's a bug
Backfill is a trick on time , not on the code of the dag , airflow should respect the code of the dag no matter what

Daniel Standish
Yesterday at 1:24 PM
And your opinion concerning a possible interface for overriding pools?
1:24
Maybe it’s not really necessary, or not as important, now that we will have separate concurrency controls for backfill?

Jarek Potiuk
Yesterday at 2:54 PM
Yeah. Backfill ignoring pools was a bug.
I think we should not override pools at all with backfill - see the discusion on backfill vs. deferrable. There is a reason a task needs a pool, and backfill ignoring that is a bad idea. (edited)
➕
2

2:56
There are probably a couple of “override” scenarios, but - paraphrasing your request
@dstandish
from the backfill/deferrable discussion - we should only allow overridability if someone comes up with - maybe a made up but realistic - case where it makes sense (edited)

raphaelauv
Yesterday at 5:05 PM
a possible interface for overriding pools
maybe some people would expect this feature ( and I don't see what are the purpose of pools in that case in this scenario ) , but they can always
copy paste the dag
sufix the name
edit the pool usage in the tasks
trigger the backill of the dag

Daniel Standish
6 minutes ago
and I don't see what are the purpose of pools in that case in this scenario
12:44
pools are a bit of an enigma
12:45
alright, well, if people say no override, then no override there shall be
12:45
i will make a lazy consensus vote

raphaelauv
4 minutes ago
pools are a bit of an enigma
I only use them as cross-dag semaphore

[shubham22]

To some extent, I am kind of surprised that no user screamed about pools being ignored during backfill, at least I didn't find an open issue on a quick search. I feel either users never fully grasped what to expect from the backfills or just were happy with the behavior - I suspect it's the former.
Overall, I agree with the direction to treat pools as an integral part of DAG that should be respected during backfills. Backfills should only be modifying the data interval, and the rest of the behavior should mimic the regular DAG run. This option also keeps the interface less complex and more predictable. If there is a need for override, which I doubt, it's much easier to add later when we have concrete use cases.

[dstandish]

I feel either users never fully grasped what to expect from the backfills or just were happy with the behavior - I suspect it's the former.

yeah or maybe folks did not use backfill as much or did not use pools as much as we might think 🤷

backfill only makes sense when your dag is driven by execution_date. when i was airflow user, i almost never used execution date.

[shubham22]

Probably, I do see "backfill" mentioned often in internal slack channels, and most ask about the CLI command or try to understand best practices.