Refactor kafka log directory setup #84

ryan-gang · 2025-07-25T19:57:46Z

This is part 2 of 3 in a stack made with GitButler:

3 Add Produce extension stages #88
2 Refactor kafka log directory setup #84 👈
1 Add Produce assertions #80

coderabbitai · 2025-07-25T19:57:55Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing Touches

🧪 Generate unit tests

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch r-branch-produce-stages

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Explain this complex logic.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai explain this code block.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and explain its main purpose.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai generate docstrings to generate docstrings for this PR.
@coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
@coderabbitai generate unit tests to generate unit tests for this PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

internal/test_helpers/course_definition.yml

internal/test_helpers/fixtures/produce/pass

internal/stage_p6.go

internal/test_helpers/fixtures/fetch/pass

protocol/serializer/generate_log_dir.go

ryan-gang · 2025-07-28T11:50:33Z

@rohitpaulk

Renamed TopicRecord and PartitionRecord to ClusterMetadataTopicRecord and ClusterMetadataPartitionRecord.
Added cluster_metadata_payload_builder
Added cluster_metadata_partition_record_builder
Added cluster_metadata_topic_record_builder
Refactor manually created records to use builders across entire generation stack
Individual stages now need to mention which topics they want to generate setup logs for
By default only minimal files are created (4 files).

2 each for Fetch, Produce Of which 1 has single partition, and the other has multiple partitions.

…ndex

…values

…itions

…records

…rn to improve code clarity and reduce duplication

…record batches and timestamps

…rder

… of booleans

protocol/serializer/cluster_metadata.go

rohitpaulk · 2025-07-28T19:13:01Z

protocol/common/constants.go

 	TOPIC1_NAME = topic_names[0]
 	TOPIC2_NAME = topic_names[1]
 	TOPIC3_NAME = topic_names[2]
-	random_topic_uuids := random.RandomInts(10, 100, 3)
+	TOPIC4_NAME = topic_names[3]
+	random_topic_uuids := random.RandomInts(10, 100, 4)


Let's avoid "pre-creating" these randomly here and stick to our usual pattern of creating random values on the fly in each stage. You can expose something like getRandomTopicConfig() / getRandomTopicConfigs() if you'd like and use that.

From a user's perspective, let's have the stage functions read more cleanly and make it clear that values are generated randomly on the fly.

rohitpaulk

Note added

ryan-gang mentioned this pull request Jul 25, 2025

Add Produce assertions #80

Open

cursor bot reviewed Jul 25, 2025

View reviewed changes

internal/test_helpers/course_definition.yml Outdated Show resolved Hide resolved

ryan-gang requested a review from rohitpaulk July 25, 2025 21:32

ryan-gang self-assigned this Jul 25, 2025

rohitpaulk requested changes Jul 28, 2025

View reviewed changes

internal/test_helpers/fixtures/produce/pass Outdated Show resolved Hide resolved

ryan-gang force-pushed the r-branch-produce-fw-3 branch from b4fe97c to cd90e90 Compare July 28, 2025 08:07

ryan-gang force-pushed the r-branch-produce-stages branch from ff576c4 to bf72df4 Compare July 28, 2025 08:07

cursor bot reviewed Jul 28, 2025

View reviewed changes

internal/stage_p6.go Outdated Show resolved Hide resolved

internal/test_helpers/fixtures/fetch/pass Outdated Show resolved Hide resolved

ryan-gang force-pushed the r-branch-produce-stages branch 3 times, most recently from f103d15 to 66a1c32 Compare July 28, 2025 08:21

ryan-gang force-pushed the r-branch-produce-fw-3 branch from cd90e90 to a8ef41f Compare July 28, 2025 08:22

ryan-gang force-pushed the r-branch-produce-stages branch 2 times, most recently from 0869f6a to 3ee5c83 Compare July 28, 2025 11:33

cursor bot reviewed Jul 28, 2025

View reviewed changes

protocol/serializer/generate_log_dir.go Outdated Show resolved Hide resolved

ryan-gang changed the title ~~Implement Produce stages~~ Refactor kafka log directory setup Jul 28, 2025

ryan-gang force-pushed the r-branch-produce-stages branch 3 times, most recently from 079b212 to 4931bba Compare July 28, 2025 11:49

ryan-gang force-pushed the r-branch-produce-stages branch from 4931bba to 5738f1b Compare July 28, 2025 15:14

ryan-gang force-pushed the r-branch-produce-fw-3 branch 3 times, most recently from 55642db to c89f8cc Compare July 28, 2025 15:29

ryan-gang force-pushed the r-branch-produce-stages branch from 5738f1b to 48e7511 Compare July 28, 2025 15:30

ryan-gang added 4 commits July 28, 2025 21:03

feat: add 4th topic to support Produce tests

0e5e550

2 each for Fetch, Produce Of which 1 has single partition, and the other has multiple partitions.

fix: increase topic count from 4 to 5 and update unknown topic name i…

4d2e505

…ndex

fix: update record batch timestamps and IDs to current time and zero …

af4e00d

…values

refactor: generalize log directory generation for all topics and part…

392d774

…itions

ryan-gang added 12 commits July 28, 2025 21:03

feat: add predefined topics with partitions and messages to constants

4449f13

feat: add ClusterMetadataPayloadBuilder to simplify payload creation

f484af1

refactor: rename TopicRecord and PartitionRecord for clarity

bd4401a

feat(builder): add builders for cluster metadata topic and partition …

0545e85

…records

refactor: replace manual cluster metadata payloads with builder patte…

931fae6

…rn to improve code clarity and reduce duplication

refactor: simplify cluster metadata serialization by removing unused …

b35428c

…record batches and timestamps

refactor: replace types.Topic with TopicConfig for topic data structures

2daf999

refactor: simplify cluster metadata generation using topic list

6dd6b28

refactor: update GenerateLogDirs to accept topic list and fix write o…

48e89d5

…rder

fix: update GenerateLogDirs calls to use explicit topic lists instead…

29aaa33

… of booleans

fix(stage_p1): update GenerateLogDirs call to use empty slice argument

e0ccb15

fix(serializer): handle missing topics and correct variable naming

5285a69

ryan-gang force-pushed the r-branch-produce-stages branch from 48e7511 to 5285a69 Compare July 28, 2025 15:34

cursor bot reviewed Jul 28, 2025

View reviewed changes

protocol/serializer/cluster_metadata.go Show resolved Hide resolved

ryan-gang requested a review from rohitpaulk July 28, 2025 15:37

ryan-gang mentioned this pull request Jul 28, 2025

Add Produce extension stages #88

Closed

rohitpaulk reviewed Jul 28, 2025

View reviewed changes

rohitpaulk requested changes Jul 28, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Refactor kafka log directory setup #84

Refactor kafka log directory setup #84

Uh oh!

ryan-gang commented Jul 25, 2025 •

edited

Loading

Uh oh!

coderabbitai bot commented Jul 25, 2025

Review skipped

Chat

Support

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (`.coderabbit.yaml`)

Documentation and Community

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ryan-gang commented Jul 28, 2025

Uh oh!

Uh oh!

rohitpaulk Jul 28, 2025

Uh oh!

rohitpaulk left a comment

Uh oh!

Uh oh!

Refactor kafka log directory setup #84

Are you sure you want to change the base?

Refactor kafka log directory setup #84

Uh oh!

Conversation

ryan-gang commented Jul 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai bot commented Jul 25, 2025

Review skipped

Chat

Support

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (.coderabbit.yaml)

Documentation and Community

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ryan-gang commented Jul 28, 2025

Uh oh!

Uh oh!

rohitpaulk Jul 28, 2025

Choose a reason for hiding this comment

Uh oh!

rohitpaulk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ryan-gang commented Jul 25, 2025 •

edited

Loading

CodeRabbit Configuration File (`.coderabbit.yaml`)