Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix slow partition adjustment issue with multiple streams and dynamic partitions detection #572

Open
wants to merge 45 commits into
base: master
Choose a base branch
from

Conversation

nyaghma
Copy link
Contributor

@nyaghma nyaghma commented Dec 8, 2020

1- Using both slow partition adjustment and dynamic partition detection features, or
2- Using both slow partition adjustment with multiple streaming queries in one job
led to an issue once a new partition was added (and detected) since the partition count was not updated properly in the PartitionStatusTracker. The issue has been detected in #571.
This PR fixes this problem and also several typos in the PartitionStatusTracker.

@nyaghma nyaghma requested a review from sjkwak December 8, 2020 23:02
@nyaghma nyaghma changed the title Fix slow partition adjustment issue when a new partition being detected by dynamic partitions feature Fix slow partition adjustment issue with multiple streams and dynamic partitions detection Dec 11, 2020
@ganeshchand
Copy link

@nyaghma is there any ETA when this PR will get merged? We recently ran into #571 and we are using azure-eventhubs-spark_2.12:2.3.18

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants