[Regression][SPIKE] Understand DataProc Batch Jobs Failure Scenarios

[colin-rogers-dbt]

Is this a regression in a recent version of dbt-bigquery?

• I believe this is a regression in dbt-bigquery functionality
• I have searched the existing issues, and I could not find an existing issue for this regression

Current Behavior

Users have noted (see: #1157 ) inconsistent performance in production with python models on dataproc.

We need to map out when/why jobs on dataproc are failing and how dbt-bigquery should handle those scenarios (i.e. retry, raise a warning etc.)

Expected/Previous Behavior

Python model execution should be stable.

Steps To Reproduce

N/A

Relevant log output

No response

Environment

- OS:
- Python:
- dbt-core (working version):
- dbt-bigquery (working version):
- dbt-core (regression version):
- dbt-bigquery (regression version):

Additional Context

No response