Profile-Guided Optimization (PGO) benchmark report

[zamazan4ik]

Hi! Thanks a lot for the library!

It's not exactly the question - just a benchmark report. Since no other categories are available in Discussions, I decided to report it here. Feel free to move it to another place if you want ;)

As I have done many times before, I decided to test the Profile-Guided Optimization (PGO) technique to optimize the library performance. For reference, results for other projects are available at https://github.com/zamazan4ik/awesome-pgo . Since PGO helped a lot for many other libraries, I decided to apply it on jiff to see if the performance win (or lose) can be achieved. chrono and time libraries are also included. Here are my benchmark results.

Test environment

• Fedora 40
• Linux kernel 6.9.9
• AMD Ryzen 9 5900x
• 48 Gib RAM
• SSD Samsung 980 Pro 2 Tib
• Compiler - Rustc 1.79.0
• jiff version: master branch on commit 31c766a9f35b3ba37e1b7253c3bf93f0281a3127
• Disabled Turbo boost

Benchmark

For benchmark purposes, I use built-in into the project benchmarks. For PGO optimization I use cargo-pgo tool. Release bench results I got with taskset -c 0 cargo bench command. The PGO training phase is done with taskset -c 0 cargo pgo bench, PGO optimization phase - with taskset -c 0 cargo pgo optimize bench.

taskset -c 0 is used for reducing the OS scheduler's influence on the results. All measurements are done on the same machine, with the same background "noise" (as much as I can guarantee).

Results

I got the following results:

• Release: https://gist.github.com/zamazan4ik/d815f545a5d4ebe5662c1b5941a00450
• PGO optimized compared to Release: https://gist.github.com/zamazan4ik/bfd2a64201c2fb0bbff264459922d9b8
• (just for reference) PGO instrumented compared to Release: https://gist.github.com/zamazan4ik/d995f2bf2d4151b6d84873618cf38024

According to the results, PGO measurably improves the libraries' performance in many cases.

Further steps

I understand that the steps above can be time-consuming and hard to implement in practice. At the very least, the library's users can find this performance report and decide to enable PGO for their applications if they care about jiff performance in their workloads. Maybe a small note somewhere in the documentation will be enough to raise awareness about this work. Also, it can be helpful for chrono and time users since in some scenarios performance for them was also improved.

Thank you.

[BurntSushi]

Thanks! Can you share a comparison of the benchmarking runs using critcmp? I find the default Criterion comparison output almost impossible to read quickly.

[zamazan4ik]

Sure. Since I didn't use baselines functionality before (just moved around criterion directories - what a shame!) I reran the benchmarks. Here we go ("before" - Release, "after" - PGO optimized):

critcmp before after
group                                                       after                                  before
-----                                                       -----                                  ------
chrono-amortize/parse_strptime                              1.00     90.6±0.16ns        ? ?/sec    1.44    130.7±0.28ns        ? ?/sec
chrono-tzfile/civil_datetime_to_instant_static              1.00     25.4±0.11ns        ? ?/sec    1.32     33.5±0.09ns        ? ?/sec
chrono-tzfile/civil_datetime_to_instant_with_tzdb_lookup    1.00      7.4±0.24µs        ? ?/sec    1.00      7.4±0.02µs        ? ?/sec
chrono-tzfile/instant_to_civil_datetime_static              1.00     34.2±0.21ns        ? ?/sec    1.18     40.3±0.26ns        ? ?/sec
chrono/civil_datetime_to_instant_static                     1.00     30.3±0.13ns        ? ?/sec    1.23     37.4±0.11ns        ? ?/sec
chrono/instant_to_civil_datetime_offset                     1.02     11.4±0.08ns        ? ?/sec    1.00     11.2±0.03ns        ? ?/sec
chrono/instant_to_civil_datetime_static                     1.03     39.7±0.16ns        ? ?/sec    1.00     38.7±0.04ns        ? ?/sec
chrono/offset_to_civil_datetime                             1.00      8.8±0.07ns        ? ?/sec    1.02      8.9±0.01ns        ? ?/sec
chrono/offset_to_instant                                    1.00      2.7±0.00ns        ? ?/sec    1.00      2.7±0.00ns        ? ?/sec
chrono/parse_civil_datetime                                 1.00     82.0±0.41ns        ? ?/sec    1.86    152.8±0.73ns        ? ?/sec
chrono/parse_rfc2822                                        1.00     63.3±0.06ns        ? ?/sec    1.74    110.3±1.41ns        ? ?/sec
chrono/parse_strptime                                       1.10    292.2±1.67ns        ? ?/sec    1.00    265.5±0.50ns        ? ?/sec
chrono/zoned_add_time_duration                              1.00      8.7±0.05ns        ? ?/sec    1.06      9.2±0.01ns        ? ?/sec
jiff/civil_datetime_to_instant_static                       1.00     31.9±0.08ns        ? ?/sec    1.06     33.8±0.03ns        ? ?/sec
jiff/civil_datetime_to_instant_with_tzdb_lookup             1.00    141.6±0.30ns        ? ?/sec    1.06    150.6±0.46ns        ? ?/sec
jiff/instant_to_civil_datetime_offset                       1.00     32.3±0.06ns        ? ?/sec    1.22     39.2±0.17ns        ? ?/sec
jiff/instant_to_civil_datetime_static                       1.00     48.3±0.26ns        ? ?/sec    1.29     62.2±0.55ns        ? ?/sec
jiff/offset_to_civil_datetime                               1.00      1.8±0.00ns        ? ?/sec    1.00      1.8±0.00ns        ? ?/sec
jiff/offset_to_instant                                      1.00      0.5±0.00ns        ? ?/sec    1.00      0.5±0.00ns        ? ?/sec
jiff/parse_civil_datetime                                   1.00     52.6±0.15ns        ? ?/sec    1.09     57.3±0.35ns        ? ?/sec
jiff/parse_rfc2822                                          1.00     53.6±0.08ns        ? ?/sec    1.46     78.1±0.53ns        ? ?/sec
jiff/parse_strptime                                         1.00     95.1±0.20ns        ? ?/sec    1.07    102.1±0.55ns        ? ?/sec
jiff/zoned_add_time_duration                                1.00     48.8±0.29ns        ? ?/sec    1.23     60.0±0.21ns        ? ?/sec
time/instant_to_civil_datetime_offset                       1.00     26.0±0.15ns        ? ?/sec    1.12     29.1±0.06ns        ? ?/sec
time/offset_to_civil_datetime                               1.20      1.6±0.00ns        ? ?/sec    1.00      1.4±0.00ns        ? ?/sec
time/offset_to_instant                                      1.01      6.0±0.01ns        ? ?/sec    1.00      5.9±0.02ns        ? ?/sec
time/parse_civil_datetime                                   1.00    109.8±0.40ns        ? ?/sec    1.04    114.1±0.58ns        ? ?/sec
time/parse_rfc2822                                          1.00     95.8±1.26ns        ? ?/sec    1.30    124.7±0.65ns        ? ?/sec
time/parse_strptime                                         1.00    177.9±0.36ns        ? ?/sec    1.16    206.4±3.34ns        ? ?/sec
time/zoned_add_time_duration                                1.00     36.4±0.32ns        ? ?/sec    1.11     40.3±0.16ns        ? ?/sec

[BurntSushi]

Very interesting, thank you!