Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[EXPERIMENTAL] Parallel VPlayer #17628

Draft
wants to merge 13 commits into
base: main
Choose a base branch
from
+1,640 −241
Draft

[EXPERIMENTAL] Parallel VPlayer #17628

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
18e5d4c
go/mysql: GTID event contains and returns lastCommitted and sequenceN…
shlomi-noach Jan 26, 2025
427349c
proto: VEventType has last_committed, sequence_number, must_save
shlomi-noach Jan 26, 2025
3242e14
adapt binlog_streamer
shlomi-noach Jan 26, 2025
78760b5
vstreamer reads and populates vevent.SequenceNumber, vevent.LastCommi…
shlomi-noach Jan 26, 2025
b5d19ce
benchmark CI job
shlomi-noach Jan 26, 2025
68506b6
VReplicationExperimentalFlagVPlayerParallel
shlomi-noach Jan 26, 2025
445571b
benchmark endtoend test
shlomi-noach Jan 26, 2025
37edb42
decorate events with EventGTID
shlomi-noach Jan 27, 2025
0d38020
decorate events with GTID
shlomi-noach Jan 27, 2025
d86850a
Enable writeset based binlog dependency tracking
mattlord Jan 28, 2025
b743481
worker-pos table
shlomi-noach Jan 28, 2025
715cda5
Merge branch 'parallel-vplayer' of github.com:planetscale/vitess into…
shlomi-noach Jan 28, 2025
6305ee1
make proto
shlomi-noach Feb 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions config/mycnf/mysql8026.cnf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,3 +40,9 @@ super-read-only
# Replication parameters to ensure reparents are fast.
replica_net_timeout = 8


binlog_transaction_dependency_tracking=WRITESET
slave_preserve_commit_order=ON
slave_parallel_type=LOGICAL_CLOCK
transaction_write_set_extraction=XXHASH64

428 changes: 214 additions & 214 deletions go/vt/proto/binlogdata/binlogdata.pb.go
View file
Open in desktop

Large diffs are not rendered by default.

16 changes: 8 additions & 8 deletions go/vt/proto/binlogdata/binlogdata_vtproto.pb.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

11 changes: 6 additions & 5 deletions go/vt/vttablet/tabletserver/vstreamer/vstreamer.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ type vstreamer struct {
format mysql.BinlogFormat
pos replication.Position
stopPos string
lastCommitted int64
commitParent int64
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While I prefer the terminology of "commit parent", the reason for calling it "last committed" is because that's the term used by MySQL, and so I did not want to introduce a new term

sequenceNumber int64
eventGTID replication.GTID

Expand Down Expand Up @@ -470,21 +470,22 @@ func (vs *vstreamer) parseEvent(ev mysql.BinlogEvent, bufferAndTransmit func(vev
})
vs.eventGTID = nil
case ev.IsGTID():
gtid, hasBegin, lastCommitted, sequenceNumber, err := ev.GTID(vs.format)
gtid, hasBegin, commitParent, sequenceNumber, err := ev.GTID(vs.format)
if err != nil {
return nil, fmt.Errorf("can't get GTID from binlog event: %v, event data: %#v", err, ev)
}
if hasBegin {
vevents = append(vevents, &binlogdatapb.VEvent{
Type: binlogdatapb.VEventType_BEGIN,
LastCommitted: lastCommitted,
CommitParent: commitParent,
SequenceNumber: sequenceNumber,
})
}
vs.pos = replication.AppendGTID(vs.pos, gtid)
vs.lastCommitted = lastCommitted
vs.commitParent = commitParent
vs.sequenceNumber = sequenceNumber
vs.eventGTID = gtid
log.Errorf("DEBUG: GTID: %v, seq_no: %d, commit_parent: %d", gtid, sequenceNumber, commitParent)
case ev.IsXID():
vevents = append(vevents, &binlogdatapb.VEvent{
Type: binlogdatapb.VEventType_GTID,
Expand Down Expand Up @@ -725,7 +726,7 @@ func (vs *vstreamer) parseEvent(ev mysql.BinlogEvent, bufferAndTransmit func(vev
vevent.Timestamp = int64(ev.Timestamp())
vevent.CurrentTime = time.Now().UnixNano()
vevent.SequenceNumber = vs.sequenceNumber
vevent.LastCommitted = vs.lastCommitted
vevent.CommitParent = vs.commitParent
if vs.eventGTID != nil {
vevent.EventGtid = vs.eventGTID.String()
}
Expand Down
6 changes: 3 additions & 3 deletions proto/binlogdata.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -480,9 +480,9 @@ message VEvent {
bool throttled = 24;
// ThrottledReason is a human readable string that explains why the stream is throttled
string throttled_reason = 25;
// For GTID events, the sequence number of the most recent transaction this event depends on / conflicts with.
int64 last_committed = 26;
// For GTID events, the sequence number of this transaction.
// For GTID events, the sequence number of the most recent transaction this event depends on.
int64 commit_parent = 26;
// For GTID events, the sequence number (logical clock) value of this transaction.
int64 sequence_number = 27;
// EventGTID is decorated by VPlayer. It is the specific GTID (not the GTID set) for this event.
string event_gtid = 28;
Expand Down
27 changes: 26 additions & 1 deletion web/vtadmin/src/proto/vtadmin.d.ts
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

127 changes: 127 additions & 0 deletions web/vtadmin/src/proto/vtadmin.js
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading