Skip to content

Conversation

@seut
Copy link
Member

@seut seut commented Dec 3, 2025

Adds a new flag --report-indexing which causes the compare_run to show shards and shards segments statistical information.
Mostly useful when running index(write) benchmarks, reporting information about the resulting segments and shard flush, refresh and merge statistics.

Example output:

Shard indexing stats across all primary shards
 Segments
     |  cnt      avg_size      min_size      max_size 
  V1 |   44       7457.84       6064.00      19478.00
  V2 |   17       9926.06       6392.00      20300.00
    
 Flush                   
     |      Counts      |                   Times (ms)                
     | total   periodic |        sum        avg        min        max 
  V1 |     0          0 |       0.00       0.00       0.00       0.00
  V2 |     0          0 |       0.00       0.00       0.00       0.00 
    
 Refresh 
     |      Counts      |                   Times (ms)                
     | total    pending |        sum        avg        min        max 
  V1 |    88          0 | 9867992.00 2466998.22 2230420.00 2917457.00
  V2 |    61          0 | 15808759.00 3952189.95 3732926.00 4354724.00 
    
 Merge 
     |     Counts     |                   Times (ms)                |             Throttle Times (ms)             |      Docs       |     Bytes       | Throttle
     | total  current |        sum        avg        min        max |        sum        avg        min        max | total   current | total   current |    in MB
  V1 |     8        0 |     585.00     146.25     126.00     167.00 |       0.00       0.00       0.00       0.00 |  2028         0 | 504856         0 |    83.89
  V2 |     4        0 |     639.00     159.75     107.00     187.00 |       0.00       0.00       0.00       0.00 |  1418         0 | 265736         0 |    83.89 

seut added 2 commits December 3, 2025 12:07
Condition to run the perf report it was using the wrong var.
Adds a new flag `--report-indexing` which causes the compare_run
to show shards and shards segments statistical information.
Mostly useful when running index(write) benchmarks, reporting
informations about the resulting segments and shard flush, refresh
and merge statistics.
@seut
Copy link
Member Author

seut commented Dec 3, 2025

Still in draft as I recognized a huge variance when using the same branch for v1 and v2. Need to investigate/play around more to see how the variance can be decreased.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants