Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

codec: add simple protocol #1139

Open
wants to merge 11 commits into
base: master
Choose a base branch
from
Open

codec: add simple protocol #1139

wants to merge 11 commits into from

Conversation

wk989898
Copy link
Collaborator

@wk989898 wk989898 commented Mar 19, 2025

What problem does this PR solve?

Issue Number: ref #1189

What is changed and how it works?

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • kafka_simple_basic
  • kafka_simple_basic_avro
  • kafka_simple_claim_check
  • kafka_simple_claim_check_avro
  • kafka_simple_handle_key_only
  • kafka_simple_handle_key_only_avro

Questions

Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?

Release note

`None`.

Sorry, something went wrong.

Copy link

ti-chi-bot bot commented Mar 19, 2025

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

Copy link

ti-chi-bot bot commented Mar 19, 2025

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from wk989898, ensuring that each of them provides their approval before proceeding. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Mar 19, 2025
@wk989898 wk989898 marked this pull request as ready for review March 20, 2025 10:39
@ti-chi-bot ti-chi-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 20, 2025
@ti-chi-bot ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. and removed release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Mar 21, 2025
@wk989898
Copy link
Collaborator Author

/retest

@asddongmen
Copy link
Collaborator

Have you support sending bootstrap event when changefeed starting in this PR?

This reverts commit 3f455a7.
Copy link

ti-chi-bot bot commented Mar 28, 2025

@wk989898: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-cdc-mysql-integration-light 298c55f link true /test pull-cdc-mysql-integration-light

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@3AceShowHand 3AceShowHand requested a review from Copilot March 28, 2025 13:57
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces and integrates a simple protocol for the codec component, updating event type usage and variable references to align with the new protocol.

  • Refactored marshaller and encoder implementations to use commonEvent types instead of model types.
  • Updated table ID and schema version handling across components.
  • Removed the legacy simple protocol decoder and adapted bootstrap and encoder_group logic accordingly.

Reviewed Changes

Copilot reviewed 14 out of 15 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
pkg/sink/codec/simple/mock/marshaller.go New autogenerated mock implementation for the marshaller interface.
pkg/sink/codec/simple/marshaller.go Updates to interface signatures and config usage for simple protocol.
pkg/sink/codec/simple/encoder.go Replaced commented legacy code with active implementation using common.
pkg/sink/codec/simple/decoder.go Removed the decoder, reflecting protocol deprecation.
pkg/sink/codec/simple/avro.go Adjusted Avro schema mappings and DML message generation.
pkg/sink/codec/encoder_group.go Updated bootstrap event handling using the new RowEvent type.
pkg/sink/codec/encoder_builder.go Modified protocol selection to instantiate simple protocol encoder.
pkg/sink/codec/common/utils.go Added test utilities for large event generation.
pkg/sink/codec/bootstraper.go Updated bootstrap logic to use the new table ID property from RowEvent.
pkg/common/event/row_change.go Added PhysicalTableID and its accessor to support simple protocol changes.
pkg/common/event/ddl_event.go Added an IsBootstrap flag to the DDL event for protocol distinction.
downstreamadapter/worker/mq_dml_worker.go Propagated PhysicalTableID from events to downstream adapter processing.
Files not reviewed (1)
  • Makefile: Language not supported
Comments suppressed due to low confidence (3)

pkg/common/event/row_change.go:88

  • [nitpick] Consider adding a comment to explain the purpose and usage of the 'PhysicalTableID' field in the RowEvent struct.
PhysicalTableID int64

pkg/sink/codec/simple/avro.go:194

  • Directly accessing ddl.MultipleTableInfos[1] could lead to an index out-of-range error; consider adding an explicit check to ensure the index exists or use a safer approach to retrieve the previous table info.
preTableInfo := ddl.MultipleTableInfos[1] // previous table info, TODO: need check it

pkg/sink/codec/bootstraper.go:232

  • Ensure that 'row.TableInfo' is non-nil before accessing 'row.TableInfo.TableName.TableID' to prevent potential runtime panics.
id: row.TableInfo.TableName.TableID,

// return errors.Trace(err)
// }
// }
if g.bootstrapWorker != nil {
Copy link
Preview

Copilot AI Mar 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Validate that the 'events' slice is non-empty before accessing events[0] in bootstrapWorker.addEvent() to avoid potential out-of-range panics.

Suggested change
if g.bootstrapWorker != nil {
if g.bootstrapWorker != nil {
if len(events) == 0 {
return errors.New("no events to add")
}

Copilot is powered by AI, so mistakes are possible. Review output carefully before use.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants