Return non-direct descendants from dags/{dag_id}/tasks/{task_id}

[ed-sparkes]

Apache Airflow version

2.3.4

What happened

We are trying to find a solution to clear a task and all its downstream tasks via the REST api. We tested a solution that included the following

• use dags/{dag_id}/tasks/{task_id} to get downstream_task_ids
• pass the failed task and the list of downstream_task_ids to dags/{dag_id}/clearTaskInstances

However it appears that the first call only returns a subset of downstream tasks, probably just direct descendants? Is there a way to get the full list of descendants. This seems to be very simple from the UI but unclear how to achieve the same via the api.

What you think should happen instead

Should return all downstream task ids. Or if not is there a way to achieve the same without recursively calling the get task endpoint? Aim is to be able clear task and all its downstreams. This is simple from the UI but seemingly difficult via the api?

How to reproduce

No response

Operating System

Debian GNU/Linux 11 (bullseye)

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[uranusjr]

This is expected since “downstream” in Airflow is defined as direct descendants. A flag on the API to find all descendants can be added. There’s already code for this (get_flat_relatives) but it’s comparatively expensive and likely shouldn’t be called by default.

[chenglongyan]

Use ”include_downstream" as parameter of clearTaskInstances ?

[potiuk]

I think there is no clarity on how exactly this can be solved - there are a few ways, I think no need to have "Feature" request for that - likely creating a PR that adds this functionality is the best way to proceed @ed-sparkes.