DAG is not being retried on failure

[nitra-1]

i have put below code to retry the dag after failure
default_args = {
'owner': 'content',
'retries': 2,
'retry_delay': timedelta(minutes=5)
}

But it does not retry and below logs are shown
Falling back to local log
*** Log file does not exist: mssqltopg_photofound_dataupdate/task_load_records/2022-11-15T02:30:00+00:00/1.log
*** Fetching from: http://:8793/log/mssqltopg_photofound_dataupdate/task_load_records/2022-11-15T02:30:00+00:00/1.log
*** Failed to fetch log file from worker. Invalid URL 'http://:8793/log/mssqltopg_photofound_dataupdate/task_load_records/2022-11-15T02:30:00+00:00/1.log': No host supplied

[potiuk]

I think you have airflow misconfigured. There are two reasons you can see no log:

• for some reason your task did not start at all - but you need to look in other logs (including logs of your deployment (systemd/kubernetes, stdout/stderr for all components if you run Airflow manually)

• your DNS or host is misconfigured and host name for your worker cannot be retrieved - look at this configuration options for ways how you can handle it - but most likely you need to fix the environment in which Airflow is run: https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#hostname-callable

Good luck with further investigations

[nitra-1]

this error is for a particular job, all other working fine.
Even this particular jobs works fine and fails on some occasion like 5-8 times in a day.
And when it fails it is not retried.

[potiuk]

Unfortunately, there is nothing I can do to help with your problem without you providing more details. It's the first time you mention some jobs are fine, other jobs are not. I have no idea what deployment you have, I have no idea what kind of jobs you have, I have no idea what your deployment says when the job fails.

Your deployment (I am not sure what it is - K8S? whatever) decided likely to kill the job.. Why it has done that? I have no idea - maybe memory? Maybe some other resources? It usually is about some resources. But In order to find out, you need to look at your deployment logs as suggested before and try to see why it happened. It - apparently - happend before Airflow had a chance to log anythong (unless you find something in Airlfow logs) - so you need to look why your deployment did it, I am afraid.

Airflow (and help here) can only go as far as airflow components are responsible, but if something happens outside of it, you are manging the deployment (or someone in your company) and they should be able to investigate further. This is the 1) st case from the options I mentioned above.