Task exited with return code Negsignal.SIGSEGV

[samsudinde]

Apache Airflow version

2.4.2

What happened

I tried to load data .csv into bigquery. but I got this errors.

[2022-10-27T11:09:58.907+0700] {taskinstance.py:1165} INFO - Dependencies all met for <TaskInstance: replication_invoice_item_new.upload_test_and_debug_area_invoice_item scheduled__2022-10-27T00:00:00+00:00 [queued]>
[2022-10-27T11:09:58.910+0700] {taskinstance.py:1165} INFO - Dependencies all met for <TaskInstance: replication_invoice_item_new.upload_test_and_debug_area_invoice_item scheduled__2022-10-27T00:00:00+00:00 [queued]>
[2022-10-27T11:09:58.911+0700] {taskinstance.py:1362} INFO - 
--------------------------------------------------------------------------------
[2022-10-27T11:09:58.911+0700] {taskinstance.py:1363} INFO - Starting attempt 2 of 7
[2022-10-27T11:09:58.911+0700] {taskinstance.py:1364} INFO - 
--------------------------------------------------------------------------------
[2022-10-27T11:09:58.915+0700] {taskinstance.py:1383} INFO - Executing <Task(PythonOperator): upload_test_and_debug_area_invoice_item> on 2022-10-27 00:00:00+00:00
[2022-10-27T11:09:58.919+0700] {standard_task_runner.py:55} INFO - Started process 28260 to run task
[2022-10-27T11:09:58.925+0700] {standard_task_runner.py:82} INFO - Running: ['airflow', 'tasks', 'run', 'replication_invoice_item_new', 'upload_test_and_debug_area_invoice_item', 'scheduled__2022-10-27T00:00:00+00:00', '--job-id', '64', '--raw', '--subdir', 'DAGS_FOLDER/db_replication/invoice_item.py', '--cfg-path', '/var/folders/fk/vjp80qzx7ms45v1jv4vz56980000gn/T/tmpiwnk5v73']
[2022-10-27T11:09:58.927+0700] {standard_task_runner.py:83} INFO - Job 64: Subtask upload_test_and_debug_area_invoice_item
[2022-10-27T11:09:58.955+0700] {task_command.py:376} INFO - Running <TaskInstance: replication_invoice_item_new.upload_test_and_debug_area_invoice_item scheduled__2022-10-27T00:00:00+00:00 [running]> on host Samsudins-MacBook-Pro.local
[2022-10-27T11:09:58.974+0700] {taskinstance.py:1590} INFO - Exporting the following env vars:

AIRFLOW_CTX_DAG_ID=replication_invoice_item_new
AIRFLOW_CTX_TASK_ID=upload_test_and_debug_area_invoice_item
AIRFLOW_CTX_EXECUTION_DATE=2022-10-27T00:00:00+00:00
AIRFLOW_CTX_TRY_NUMBER=2
AIRFLOW_CTX_DAG_RUN_ID=scheduled__2022-10-27T00:00:00+00:00
[2022-10-27T11:09:59.201+0700] {local_task_job.py:164} INFO - Task exited with return code Negsignal.SIGSEGV
[2022-10-27T11:09:59.210+0700] {local_task_job.py:273} INFO - 0 downstream tasks scheduled from follow-on schedule check

What you think should happen instead

I was migrate my own laptop from os linux to mac m1. the code production should be fine on my previouse laptop, but I tried to test on my new laptop its doesnt work well to process load data.
do have any idea about this?

How to reproduce

No response

Operating System

mac m1

Versions of Apache Airflow Providers

No response

Deployment

Other

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[Taragolis]

@samsudinde this is not first time when issue with Negsignal.SIGSEGV appear:

• When triggering a job via airfow UI, job fails abruptly with Negsignal.SIGSEGV  #16243
• Task exited with return code Negsignal.SIGSEGV #20309
• Task exited with return code Negsignal.SIGSEGV when dag triggered from UI #24463

It is very hard to debug segmentation violation error.

You might try to debug what actually cause SIGSEGV error in your case.

One thing you can try. Can you set the PYTHONFAULTHANDLER variable before running airlfow ?

export PYTHONFAULTHANDLER="true"

Setting this variable before running your airlfow might provide more information on what caused the problem - it will not make it possible to handle it better, but at least might save you the guesswork.

Or you could try to run Airflow on macOs in docker in this case this kind of of error should disappear.
Be aware that if you choose this path you need to have some knowledge about docker I can add some just simple stuff which might help:

1. M1 it is ARM64 architecture (aka armv8 aka AArch64) but ARM support in Airflow is Experimental (some of the packages not built for this architecture)
2. If you build docker image on mac with M1/M2 by default it build for linux/arm64 architecture. As I mention previously not everything might work on ARM in this case you could also build and run image for linux/amd64 architecture by adding flag --platform linux/amd64 unfortunately it will greatly affect performance

[potiuk]

Converted it into discussion - this is at most this - not an Airflow issue.

[potiuk]

In the future @samsudinde -> please try to search first in the issues/discussions and if refrain from opening issue/bug if you need to troubleshoot (it's quite clear in the template when you open issue that you should open discussions if you need help in troubleshooting your problems).

[samsudinde]

okey @potiuk , thanks for advance