Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CloudBatchSubmitJobOperator incorrectly reports success when defer mode is off #43744

Open
1 of 2 tasks
adamszustak opened this issue Nov 6, 2024 · 2 comments
Open
1 of 2 tasks
Labels
area:providers kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet provider:google Google (including GCP) related issues

Comments

@adamszustak
Copy link

adamszustak commented Nov 6, 2024

Apache Airflow Provider(s)

google

Versions of Apache Airflow Providers

apache-airflow-providers-apache-beam==5.6.2
apache-airflow-providers-celery==3.6.0
apache-airflow-providers-cncf-kubernetes==8.0.1
apache-airflow-providers-common-sql==1.11.1
apache-airflow-providers-dbt-cloud==3.7.0
apache-airflow-providers-ftp==3.7.0
apache-airflow-providers-google==10.16.0
apache-airflow-providers-hashicorp==3.6.4
apache-airflow-providers-http==4.10.0
apache-airflow-providers-imap==3.5.0
apache-airflow-providers-mysql==5.5.4
apache-airflow-providers-postgres==5.10.2
apache-airflow-providers-sendgrid==3.4.0
apache-airflow-providers-sqlite==3.7.1
apache-airflow-providers-ssh==3.10.1

Apache Airflow version

airflow 2.7.3

Operating System

composer-2.6.5-airflow-2.7.3

Deployment

Google Cloud Composer

Deployment details

No response

What happened

When defer mode is off, the CloudBatchSubmitJobOperator operator always reports success regardless of the job's actual execution status.
When defer mode is on, the CloudBatchSubmitJobOperator operator correctly reports success or failure based on the job's exit code.

What you think should happen instead

The operator should always report the correct success or failure status based on the job's execution, regardless of whether defer mode is on or off.

How to reproduce

Create a CloudBatchSubmitJobOperator that always fails (like exit 1) and run it in sync mode.

([
            {
                "taskSpec": {
                    "runnables": [
                        {
                            "script": {
                                "text": "echo This is task ${BATCH_TASK_INDEX}. This job has a total of ${BATCH_TASK_COUNT} tasks.; exit 1"
                            }
                        }
                    ],
                    "computeResource": {"cpuMilli": 2000, "memoryMib": 16},
                    "maxRetryCount": 2,
                    "maxRunDuration": "3600s",
                },
                "taskCount": 1,
                "parallelism": 3,
            }
        ],
        deferrable=False,
)

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@adamszustak adamszustak added area:providers kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet labels Nov 6, 2024
Copy link

boring-cyborg bot commented Nov 6, 2024

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

@dosubot dosubot bot added the provider:google Google (including GCP) related issues label Nov 6, 2024
@potiuk
Copy link
Member

potiuk commented Nov 6, 2024

cc: @VladaZakharova

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:providers kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet provider:google Google (including GCP) related issues
Projects
None yet
Development

No branches or pull requests

2 participants