Memory leak when using AWS Glue Job Console in Airflow #45845
Assignees
Labels
area:performance
good first issue
kind:feature
Feature Requests
provider:amazon-aws
AWS/Amazon - related issues
Comments
hazemAmr0
added
kind:feature
dosubot
bot
added
area:performance
provider:amazon-aws
potiuk
added
good first issue
and removed
needs-triage
|
Feel free to look at this |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
When running an AWS Glue job via Airflow, there appears to be a memory leak in the task rate monitoring component. After the job runs for a few hours, memory usage steadily increases, leading to crashes.
Use case/motivation
Improved memory management would help ensure long-running jobs, such as large data processing tasks, can run smoothly without crashes.
Related issues
No specific stack trace, just system logs indicating out-of-memory (OOM) errors.
Files and functions to be edited (if known):
Potentially related to the AWSGlueJobOperator or task rate monitoring functions in Airflow.
Are you willing to submit a PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: