Scheduler stopped with (so it seems) Exiting gracefully upon receiving signal 15

[waardd]

In the GUI it states scheduler is not running altough the service is up.
After restarting the service for the airflow-scheduler it runs again.
The dag_processor_manager.log states:
Exiting gracefully upon receiving signal 15

[2023-02-15T21:45:31.575+0100] {manager.py:465} INFO - Exiting gracefully upon receiving signal 15
[2023-02-16T07:09:22.804+0100] {manager.py:483} INFO - Processing files using up to 2 processes at a time
[2023-02-16T07:09:22.814+0100] {manager.py:484} INFO - Process each file at most once every 30 seconds
[2023-02-16T07:09:22.814+0100] {manager.py:485} INFO - Checking for new files in /appl/mca/airflow/dags every 300 seconds
[2023-02-16T07:09:22.814+0100] {manager.py:726} INFO - Searching for files in /appl/mca/airflow/dags
[2023-02-16T07:09:22.824+0100] {manager.py:729} INFO - There are 8 files in /appl/mca/airflow/dags
[2023-02-16T07:09:22.956+0100] {manager.py:854} INFO -
================================================================================
DAG File Processing Stats

File Path                                              PID  Runtime      # DAGs    # Errors  Last Runtime    Last Run
-------------------------------------------------  -------  ---------  --------  ----------  --------------  ----------
/appl/mca/airflow/dags/kwargs.py                                              0           0
/appl/mca/airflow/dags/agentscanner_monitoring.py  1278457  0.07s             0           0
/appl/mca/airflow/dags/OrderwatchImport.py         1278458  0.06s             0           0
/appl/mca/airflow/dags/SophisImport.py                                        0           0
/appl/mca/airflow/dags/base_monitoring.py                                     0           0
/appl/mca/airflow/dags/TradebookImport.py                                     0           0
/appl/mca/airflow/dags/OrderblotterImport.py                                  0           0
/appl/mca/airflow/dags/xcom_test.py                                           0           0
================================================================================
[2023-02-16T07:09:53.019+0100] {manager.py:854} INFO -
================================================================================
DAG File Processing Stats

File Path                                          PID    Runtime      # DAGs    # Errors  Last Runtime    Last Run
-------------------------------------------------  -----  ---------  --------  ----------  --------------  -------------------
/appl/mca/airflow/dags/kwargs.py                                            1           0  0.16s           2023-02-16T06:09:23

[Taragolis]

Exiting gracefully upon receiving signal 15

Signal 15 it is refers to SIGTERM. It might happen due to OOM Manager (os level) send SIGTERM to airflow scheduler due to insufficient RAM.

Could you provide a bit more detail about your Airflow deployment?

[waardd]

Here some system info.
Memory:
total used free shared buff/cache available
Mem: 15Gi 7.8Gi 367Mi 1.0Mi 7.3Gi 7.3Gi
Swap: 5.0Gi 1.4Gi 3.6Gi

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 2

Running on RHEL8 with Python 3.6.8

Overview of process resources for web and scheduler

I also see these settings:
default_owner = airflow
default_cpus = 1
default_ram = 512
default_disk = 512

[potiuk]

It must be somewhere outside of Airflow. I do not know how you are running your aiflow - but if you are using systemd - there used to be (the flag was removed in October 2017) this line in scheduler.service:

ExecStart=/bin/airflow scheduler -n ${SCHEDULER_RUNS}

So possibly this is what you still have there.

[waardd]

Yes the -n ${SCHEDULER_RUNS} is in there.
Can i take that piece out or is changing the value for SCHEDULER_RUNS to 0 the same?

[potiuk]

Take them out

[mohittalele]

I am also experiencing same issue. @waardd what changes solved issue for you?

[vadipo]

same question here...