Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

VReplication VPlayer: support statement and transaction batching #14502

Merged
merged 28 commits into from
Dec 7, 2023
Merged

VReplication VPlayer: support statement and transaction batching #14502

merged 28 commits into from
Dec 7, 2023

Conversation

mattlord
Copy link
Contributor

@mattlord mattlord commented Nov 9, 2023 •
edited
Loading

Description

When there is a bulk DELETE on the source, the target sees a stream of row changes, each of which is a DELETE. Currently we run each of these DELETEs as an individual query. So if there are a thousand (say) rows deleted because of a single DELETE statement, we will run a thousand individual DELETEs on the target. This is wasteful in performance and can result in a large vreplication lag for clusters with a high QPS and frequent bulk DELETEs (in particular when using external tablets or tablets with otherwise high latency connections to the mysqld). This PR identifies such transactions and coalesces these statements using IN resulting in a single DELETE statement for the case mentioned above (currently limited to tables with single column Primary Keys).

Similarly, when there is a bulk INSERT on the source we now batch those into a single statement with multiple values tuples on the target.

Lastly, we batch all of the final transaction's statements together in a single multi-statement MySQL protocol message when we want to commit.

This functionality is opt-in using the --vreplication_experimental_flags approach. A new bitmask VReplicationExperimentalFlagVPlayerBatching = 4 (0x100) needs to be set to enable it. It is disabled by default.

Related Issue(s)

Checklist

@vitess-bot vitess-bot
Copy link
Contributor

vitess-bot bot commented Nov 9, 2023

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

  • Ensure that the Pull Request has a descriptive title.
  • Ensure there is a link to an issue (except for internal cleanup and flaky test fixes), new features should have an RFC that documents use cases and test cases.

Tests

  • Bug fixes should have at least one unit or end-to-end test, enhancement and new features should have a sufficient number of tests.

Documentation

  • Apply the release notes (needs details) label if users need to know about this change.
  • New features should be documented.
  • There should be some code comments as to why things are implemented the way they are.
  • There should be a comment at the top of each new or modified test to explain what the test does.

New flags

  • Is this flag really necessary?
  • Flag names must be clear and intuitive, use dashes (-), and have a clear help text.

If a workflow is added or modified:

  • Each item in Jobs should be named in order to mark it as required.
  • If the workflow needs to be marked as required, the maintainer team must be notified.

Backward compatibility

  • Protobuf changes should be wire-compatible.
  • Changes to _vt tables and RPCs need to be backward compatible.
  • RPC changes should be compatible with vitess-operator
  • If a flag is removed, then it should also be removed from vitess-operator and arewefastyet, if used there.
  • vtctl command output order should be stable and awk-able.

@vitess-bot vitess-bot bot added NeedsDescriptionUpdate The description is not clear or comprehensive enough, and needs work NeedsIssue A linked issue is missing for this Pull Request NeedsWebsiteDocsUpdate What it says labels Nov 9, 2023
@github-actions github-actions bot added this to the v19.0.0 milestone Nov 9, 2023
@mattlord mattlord force-pushed the vplayer_trx_batching branch 3 times, most recently from e72a899 to 45239f9 Compare November 11, 2023 05:28
@mattlord mattlord removed NeedsIssue A linked issue is missing for this Pull Request NeedsWebsiteDocsUpdate What it says labels Nov 16, 2023
@mattlord mattlord force-pushed the vplayer_trx_batching branch 2 times, most recently from c858006 to 9af3f4a Compare November 16, 2023 15:47
@rohit-nayak-ps
Fix goimports
a4b62da 
Signed-off-by: Rohit Nayak <[email protected]>
@rohit-nayak-ps rohit-nayak-ps added Type: Enhancement Logical improvement (somewhere between a bug and feature) Component: VReplication labels Nov 17, 2023
@rohit-nayak-ps rohit-nayak-ps removed the NeedsDescriptionUpdate The description is not clear or comprehensive enough, and needs work label Nov 17, 2023
@rohit-nayak-ps rohit-nayak-ps changed the title Add batching support to vplayer VReplication VPlayer: batch a multi-delete statement while applying on the target Nov 18, 2023
@mattlord mattlord changed the title VReplication VPlayer: batch a multi-delete statement while applying on the target VReplication VPlayer: support statement batching while applying on the target Nov 30, 2023
@mattlord mattlord changed the title VReplication VPlayer: support statement batching while applying on the target VReplication VPlayer: support statement batching Nov 30, 2023
@mattlord mattlord requested review from dbussink and removed request for GuptaManan100 December 4, 2023 21:26
dbussink
dbussink approved these changes Dec 4, 2023
View reviewed changes
go/vt/binlog/binlogplayer/mock_dbclient.go Outdated Show resolved Hide resolved
go/vt/vttablet/tabletmanager/vreplication/replicator_plan.go
return nil, nil
}
if (len(tp.TablePlanBuilder.pkCols) + len(tp.TablePlanBuilder.extraSourcePkCols)) != 1 {
return nil, vterrors.Errorf(vtrpcpb.Code_INVALID_ARGUMENT, "bulk delete is only supported for tables with a single primary key column")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't want to extend this here for composite primary keys? Or is that for a separate PR?

Copy link
Contributor Author

@mattlord mattlord Dec 4, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I planned to punt on that until later, but I can take a stab at it. I think I'd prefer using OR groups. Maybe it will be easy....

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a TODO for now. That would require much more testing as the performance of those DELETEs can be unexpected based on past experience. That concern may not be warranted anymore with 8.0+, but for a related recent example with MariaDB:
https://jira.mariadb.org/browse/MDEV-32878

@mattlord
Properly incorporate pos update in batch mgmt
706d7dd 
Signed-off-by: Matt Lord <[email protected]>
@mattlord
Merge remote-tracking branch 'origin/main' into vplayer_trx_batching
0cb8d47 
Signed-off-by: Matt Lord <[email protected]>
@mattlord
Import fixup after merge
434f7c0 
Signed-off-by: Matt Lord <[email protected]>
@mattlord
Correct log messages
d55ca70 
Signed-off-by: Matt Lord <[email protected]>
@mattlord
Changes after self review
18636e0 
Signed-off-by: Matt Lord <[email protected]>
@mattlord
Minor change after 2nd self review
0721e4b 
Signed-off-by: Matt Lord <[email protected]>
@mattlord
Fix bug that caused us to skip non-trx queries
ead398a 
Signed-off-by: Matt Lord <[email protected]>
@mattlord
Improve func names
5747049 
Signed-off-by: Matt Lord <[email protected]>
@mattlord
Can't stop nitting...
d15fb0b 
Signed-off-by: Matt Lord <[email protected]>
rohit-nayak-ps
rohit-nayak-ps approved these changes Dec 6, 2023
View reviewed changes
Copy link
Contributor

@rohit-nayak-ps rohit-nayak-ps left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, 💯

The VReplication FK Stress test is failing, most likely not related to the PR functionality.

go/vt/vttablet/tabletmanager/vreplication/vplayer_flaky_test.go Outdated Show resolved Hide resolved
mattlord added a commit to vitessio/website that referenced this pull request Dec 6, 2023
@mattlord
Update vreplication_experimental_flags docs after vitessio/vitess#14502
b1f959f 
Signed-off-by: Matt Lord <[email protected]>
@mattlord mattlord mentioned this pull request Dec 6, 2023
Merged
@mattlord mattlord merged commit c6a6bfe into vitessio:main Dec 7, 2023
116 checks passed
@mattlord mattlord deleted the vplayer_trx_batching branch December 7, 2023 00:39
mattlord added a commit to vitessio/website that referenced this pull request Dec 9, 2023
@mattlord
Update vreplication_experimental_flags docs after vitessio/vitess#14502…
b84d282 
… (#1647)

Signed-off-by: Matt Lord <[email protected]>
ejortegau pushed a commit to slackhq/vitess that referenced this pull request Dec 13, 2023
@mattlord @rohit-nayak-ps @ejortegau
VReplication VPlayer: support statement and transaction batching (vit…
361a43b 
…essio#14502)

Signed-off-by: Matt Lord <[email protected]>
Signed-off-by: Rohit Nayak <[email protected]>
Co-authored-by: Rohit Nayak <[email protected]>
@mattlord mattlord mentioned this pull request Nov 7, 2024
Merged
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dbussink dbussink dbussink approved these changes

@rohit-nayak-ps rohit-nayak-ps rohit-nayak-ps approved these changes

@deepthi deepthi Awaiting requested review from deepthi deepthi is a code owner

@shlomi-noach shlomi-noach Awaiting requested review from shlomi-noach shlomi-noach is a code owner

Assignees
No one assigned
Labels
Component: VReplication Type: Enhancement Logical improvement (somewhere between a bug and feature)
Projects
VReplication
Status: Done
Milestone
v19.0.0
Development

Successfully merging this pull request may close these issues.

Feature Request: Optimize VReplication Binlog Replay (VPlayer) Query Execution
3 participants
@mattlord @dbussink @rohit-nayak-ps