chore: disabling merge before processor.Store(...) - default true #5338
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #5338 +/- ##
==========================================
- Coverage 74.78% 74.73% -0.05%
==========================================
Files 433 437 +4
Lines 60979 61236 +257
==========================================
+ Hits 45601 45766 +165
- Misses 12848 12928 +80
- Partials 2530 2542 +12 ☔ View full report in Codecov by Sentry. |
| if w.handle.config().disableStoreMerge.Load() { | ||
| if firstSubJob { | ||
| w.handle.Store(w.partition, subJob) | ||
| continue | ||
| } | ||
| mergedJob.merge(subJob) | ||
| w.handle.Store(w.partition, mergedJob) | ||
| firstSubJob = true | ||
| continue | ||
| } |
There was a problem hiding this comment.
@Sidddddarth I'm a bit confused by this. Shouldn't feature flags be backwards compatible? Meaning, given the default of the new setting, nothing should change in terms of behaviour.
I see that the default for the new setting is true. In that case we get into the new if on line 108 but we're not doing what we have on main. The logic is different now.
There was a problem hiding this comment.
You're right
I wanted to make this the new behaviour. But it might not make sense in all cases
Will change the default
Description
Disable merging
subJobs before storing them in processor.In cases where there's heavy multiplexing, it could be better to save each subJob's transformed jobs.
Also helpful in cases where transformation is the limiting step in processor pipeline.
Router sees a continuous input of jobs instead of once every few seconds.
Put behind a flag(default allows writing each subJob right away).
Linear Ticket
Resolves PIPE-1785
Security