Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multithreaded replication WIP #1454

Draft
wants to merge 47 commits into
base: master
Choose a base branch
from
Draft

Multithreaded replication WIP #1454

wants to merge 47 commits into from

Conversation

meiji163
Copy link
Contributor

@meiji163 meiji163 commented Oct 4, 2024

A Pull Request should be associated with an Issue.

We wish to have discussions in Issues. A single issue may be targeted by multiple PRs.
If you're offering a new feature or fixing anything, we'd like to know beforehand in Issues,
and potentially we'll be able to point development in a particular direction.

Related issue: https://github.com/github/gh-ost/issues/0123456789

Further notes in https://github.com/github/gh-ost/blob/master/.github/CONTRIBUTING.md
Thank you! We are open to PRs, but please understand if for technical reasons we are unable to accept each and any PR

Description

This PR [briefly explain what it does]

In case this PR introduced Go code changes:

  • contributed code is using same conventions as original code
  • script/cibuild returns with no formatting errors, build errors or unit test errors.

arthurschreiber and others added 30 commits September 30, 2024 09:15
@arthurschreiber
WIP.
37c1abd
@arthurschreiber
Fixes.
5d69fd9
@arthurschreiber
Fixups.
df29aa8
@arthurschreiber
Special handling for first job.
81a6e6e
@arthurschreiber
Fix deadlocks.
b44c6df
@arthurschreiber
Update docs and simulate changes.
55ce5a2
@meiji163
start StreamTransaction implementation
0d8f39b
@meiji163
add job Coordinator
22a4b8c
@meiji163
fix job complete logic
3fd6e71
@meiji163
add docker compose
f209592
@meiji163
fix application of applyEvent.writeFunc
c6c877e
@arthurschreiber
Send off transaction events as soon as we see a GTID event.
b031166 
Multiple `TableMap` events can happen inside of a single transaction, so we can't really use them to filter out events early or anything like that. Instead, we'll just skip them.

Also adds some basic test cases for the transaction streaming.
@arthurschreiber @danieljoos
WIP.
f81a790 
Co-authored-by: Daniel Joos <[email protected]>
@arthurschreiber
WIP.
e3b2cda
@meiji163
fix MarkTransactionCompleted
3ba9058
@meiji163
go mod tidy
74f6c9c
@meiji163
configure max idle connections
eddc1c9
@meiji163
vendor packages
6321e73
@meiji163
track binlog coords
5600b91
@meiji163
binlog streamer reconnect
3f47ebd
@meiji163
fix TestMigrate
e81aabf
@meiji163
worker-stats command
e4da5f8
@meiji163
setup mysql in CI
c5e239c
@meiji163
set transaction_write_set_extraction in test
2e78f6f
@meiji163
add ci mysql opt flags
875d00d
@meiji163
Revert "set transaction_write_set_extraction in test"
9a022e2 
This reverts commit 2e78f6f.
@meiji163
change ci mysql opts
18dcee1
@meiji163
try custom docker run
299df37
@arthurschreiber
Make the linter happy.
941689f
@arthurschreiber
Use testcontainers.
9e3bc1c
danieljoos
danieljoos previously approved these changes Oct 22, 2024
View reviewed changes
@Kiran14330 Kiran14330 mentioned this pull request Nov 2, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@arthurschreiber arthurschreiber arthurschreiber approved these changes

@danieljoos danieljoos danieljoos left review comments

@rashiq rashiq Awaiting requested review from rashiq rashiq will be requested when the pull request is marked ready for review rashiq is a code owner

@timvaillancourt timvaillancourt Awaiting requested review from timvaillancourt timvaillancourt will be requested when the pull request is marked ready for review timvaillancourt is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@meiji163 @arthurschreiber @danieljoos