Full sync: Set chunk size of Woo modules dynamically #41433

darssen · 2025-01-30T15:14:16Z

Part of https://github.com/Automattic/vulcan/issues/661

#34390 created the ability to have a dynamic number of items sent for posts. The final number of items sent will be determined by reaching the MAX_SIZE for a number of items or if we reach the chunk_size in full_sync_limits.
Same work as the one done in #41350 now for the Woo modules living in the Sync package. ( woocommerce and woocommerce-hpos-orders)

Proposed changes:

Re-use the filter_objects_and_metadata_by_size method written in the Module parent class.
Abstract some of the constants and functions to the parent class to keep things tidy.

Other information:

Have you written new tests for your changes, if applicable?
Have you checked the E2E test CI results, and verified that your changes do not break them?
Have you tested your changes on WordPress.com, if applicable (if so, you'll see a generated comment below with a script to run)?

Jetpack product discussion

Does this pull request change what data or activity we track or use?

Testing instructions:

Repeat the instructions in Full sync: Set chunk size of comments dynamically #41350 but do them for woocommerce and woocommerce-hpos-orders modules instead.

github-actions · 2025-01-30T15:15:05Z

Are you an Automattician? Please test your changes on all WordPress.com environments to help mitigate accidental explosions.

To test on WoA, go to the Plugins menu on a WordPress.com Simple site. Click on the "Upload" button and follow the upgrade flow to be able to upload, install, and activate the Jetpack Beta plugin. Once the plugin is active, go to Jetpack > Jetpack Beta, select your plugin, and enable the update/full-sync-woo-dynamic branch.
- For jetpack-mu-wpcom changes, also add define( 'JETPACK_MU_WPCOM_LOAD_VIA_BETA_PLUGIN', true ); to your wp-config.php file.

To test on Simple, run the following command on your sandbox:

bin/jetpack-downloader test jetpack update/full-sync-woo-dynamic

bin/jetpack-downloader test jetpack-mu-wpcom-plugin update/full-sync-woo-dynamic

Interested in more tips and information?

In your local development environment, use the jetpack rsync command to sync your changes to a WoA dev blog.
Read more about our development workflow here: PCYsg-eg0-p2
Figure out when your changes will be shipped to customers here: PCYsg-eg5-p2

github-actions · 2025-01-30T15:15:32Z

Thank you for your PR!

When contributing to Jetpack, we have a few suggestions that can help us test and review your patch:

✅ Include a description of your PR changes.
✅ Add a "[Status]" label (In Progress, Needs Team Review, ...).
✅ Add a "[Type]" label (Bug, Enhancement, Janitorial, Task).
✅ Add testing instructions.
✅ Specify whether this PR includes any changes to data or privacy.
✅ Add changelog entries to affected projects

This comment will be updated as you work on your PR and make changes. If you think that some of those checks are not needed for your PR, please explain why you think so. Thanks for cooperation 🤖

The e2e test report can be found here. Please note that it can take a few minutes after the e2e tests checks are complete for the report to be available.

Follow this PR Review Process:

Ensure all required checks appearing at the bottom of this PR are passing.
Choose a review path based on your changes:
- A. Team Review: add the "[Status] Needs Team Review" label
  - For most changes, including minor cross-team impacts.
  - Example: Updating a team-specific component or a small change to a shared library.
- B. Crew Review: add the "[Status] Needs Review" label
  - For significant changes to core functionality.
  - Example: Major updates to a shared library or complex features.
- C. Both: Start with Team, then request Crew
  - For complex changes or when you need extra confidence.
  - Example: Refactor affecting multiple systems.
Get at least one approval before merging.

Still unsure? Reach out in #jetpack-developers for guidance!

Jetpack plugin:

The Jetpack plugin has different release cadences depending on the platform:

WordPress.com Simple releases happen semi-continuously (PCYsg-Jjm-p2).
WoA releases happen weekly.
Releases to self-hosted sites happen monthly. The next release is scheduled for none scheduled (scheduled code freeze on undefined).

If you have any questions about the release process, please ask in the #jetpack-releases channel on Slack.

github-actions · 2025-01-30T15:27:24Z

Code Coverage Summary

Coverage changed in 5 files.

File	Coverage	Δ%	Δ Uncovered
projects/packages/sync/src/modules/class-woocommerce.php	0/124 (0.00%)	0.00%	32 💔
projects/packages/sync/src/modules/class-woocommerce-hpos-orders.php	0/183 (0.00%)	0.00%	26 💔
projects/packages/sync/src/modules/class-module.php	45/210 (21.43%)	0.66%	1 💔
projects/packages/sync/src/modules/class-comments.php	44/169 (26.04%)	0.60%	-4 💚
projects/packages/sync/src/modules/class-posts.php	23/247 (9.31%)	0.15%	-4 💚

Full summary · PHP report · JS report

…_objects_and_metadata_by_size.

…oper term

… which is necessary for the chunk logic

fgiannar

Great work, Juanma!

Tested again after all the fixes and chunking works as expected 👍

Left a couple inline comments :)

fgiannar · 2025-02-05T09:39:12Z

projects/packages/sync/src/modules/class-woocommerce.php

@@ -668,8 +641,77 @@ public function get_order_item_by_ids( $ids ) {
 		$placeholders = implode( ',', array_fill( 0, count( $ids ), '%d' ) );

 		$query = "SELECT * FROM {$this->order_item_table_name} WHERE order_item_id IN ( $placeholders )";
+		if ( ! empty( $order ) ) {


Is there any reason to not fetch the items in DESC order in all cases? So that we don't need to add the extra $order param.
In any case, if we decide to keep it like this lets at least check and sanitize it before using it, eg $order = in_array( $order, [ 'ASC', 'DESC' ] ) ? $order : 'DESC';

Thanks for the comment, get_order_item_by_ids is mainly used in checksum logic. We might be able to just use DESC order in all cases, but wanted to play it safe.
What do you think is better?

I'd keep it simple assuming this doesn't affect any consumers of this method, which I guess it shouldn't. I'm good with w/e you prefer as long as we sanitize it if we keep it like this.

Let's keep it safe, I added the sanitization in 17e43c2

projects/packages/sync/src/modules/class-woocommerce.php

darssen added 2 commits January 30, 2025 15:51

Add woo modules dynamic

5d7ff3c

changelog

2eb7f71

darssen added [Type] Enhancement Changes to an existing feature — removing, adding, or changing parts of it [Focus] Performance [Status] In Progress [Package] Sync labels Jan 30, 2025

darssen requested a review from fgiannar January 30, 2025 15:14

darssen self-assigned this Jan 30, 2025

Typo from merge

a36b7c1

github-actions bot added [Plugin] Jetpack Issues about the Jetpack plugin. https://wordpress.org/plugins/jetpack/ [Tests] Includes Tests labels Jan 30, 2025

Typos regarding MAX_META_LENGTH and phan issues

954e4b8

darssen added [Status] Needs Review To request a review from fellow Jetpack developers. Label will be renamed soon. and removed [Status] In Progress labels Feb 3, 2025

darssen added 4 commits February 4, 2025 16:52

Bug fixingsince objects can actually be arrays :(

bdf71f5

Added tests so that objects coming as arrays are also added in filter…

c19c14c

…_objects_and_metadata_by_size.

Woocommerce module handles order items, let's make sure we use the pr…

86e2326

…oper term

Changes so that order items when expanded, are returned in desc order…

58db634

… which is necessary for the chunk logic

fgiannar requested changes Feb 5, 2025

View reviewed changes

darssen added 3 commits February 5, 2025 11:23

Deprecate removed methods

2c84768

Sanitize order in get_order_item_by_ids

17e43c2

Renamed function to build_full_sync_action_array

d22aa13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Full sync: Set chunk size of Woo modules dynamically #41433

Full sync: Set chunk size of Woo modules dynamically #41433

darssen commented Jan 30, 2025 •

edited

Loading

github-actions bot commented Jan 30, 2025 •

edited

Loading

github-actions bot commented Jan 30, 2025 •

edited

Loading

github-actions bot commented Jan 30, 2025 •

edited

Loading

fgiannar left a comment

fgiannar Feb 5, 2025

darssen Feb 5, 2025

fgiannar Feb 5, 2025

darssen Feb 5, 2025

Full sync: Set chunk size of Woo modules dynamically #41433

Are you sure you want to change the base?

Full sync: Set chunk size of Woo modules dynamically #41433

Conversation

darssen commented Jan 30, 2025 • edited Loading

Proposed changes:

Other information:

Jetpack product discussion

Does this pull request change what data or activity we track or use?

Testing instructions:

github-actions bot commented Jan 30, 2025 • edited Loading

github-actions bot commented Jan 30, 2025 • edited Loading

github-actions bot commented Jan 30, 2025 • edited Loading

Code Coverage Summary

fgiannar left a comment

Choose a reason for hiding this comment

fgiannar Feb 5, 2025

Choose a reason for hiding this comment

darssen Feb 5, 2025

Choose a reason for hiding this comment

fgiannar Feb 5, 2025

Choose a reason for hiding this comment

darssen Feb 5, 2025

Choose a reason for hiding this comment

darssen commented Jan 30, 2025 •

edited

Loading

github-actions bot commented Jan 30, 2025 •

edited

Loading

github-actions bot commented Jan 30, 2025 •

edited

Loading

github-actions bot commented Jan 30, 2025 •

edited

Loading