perf(profiling): cache code-provenance.json#16578
Conversation
Codeowners resolved as |
This comment has been minimized.
This comment has been minimized.
Performance SLOsComparing candidate taegyunkim/code-provenance-cache (b4eb14a) with baseline main (8c9f2b5) 📈 Performance Regressions (2 suites)📈 iastaspects - 118/118✅ add_aspectTime: ✅ 103.677µs (SLO: <130.000µs 📉 -20.2%) vs baseline: +0.7% Memory: ✅ 42.998MB (SLO: <46.000MB -6.5%) vs baseline: +4.5% ✅ add_inplace_aspectTime: ✅ 101.444µs (SLO: <130.000µs 📉 -22.0%) vs baseline: -1.1% Memory: ✅ 42.920MB (SLO: <46.000MB -6.7%) vs baseline: +4.3% ✅ add_inplace_noaspectTime: ✅ 28.659µs (SLO: <40.000µs 📉 -28.4%) vs baseline: +1.9% Memory: ✅ 43.018MB (SLO: <46.000MB -6.5%) vs baseline: +4.3% ✅ add_noaspectTime: ✅ 48.556µs (SLO: <70.000µs 📉 -30.6%) vs baseline: -2.0% Memory: ✅ 42.900MB (SLO: <46.000MB -6.7%) vs baseline: +4.3% ✅ bytearray_aspectTime: ✅ 249.886µs (SLO: <400.000µs 📉 -37.5%) vs baseline: +0.2% Memory: ✅ 42.900MB (SLO: <46.000MB -6.7%) vs baseline: +4.6% ✅ bytearray_extend_aspectTime: ✅ 632.418µs (SLO: <800.000µs 📉 -20.9%) vs baseline: -3.5% Memory: ✅ 42.939MB (SLO: <46.000MB -6.7%) vs baseline: +4.2% ✅ bytearray_extend_noaspectTime: ✅ 263.152µs (SLO: <400.000µs 📉 -34.2%) vs baseline: -4.8% Memory: ✅ 42.920MB (SLO: <46.000MB -6.7%) vs baseline: +4.2% ✅ bytearray_noaspectTime: ✅ 133.952µs (SLO: <300.000µs 📉 -55.3%) vs baseline: -4.9% Memory: ✅ 42.959MB (SLO: <46.000MB -6.6%) vs baseline: +4.4% ✅ bytes_aspectTime: ✅ 218.672µs (SLO: <300.000µs 📉 -27.1%) vs baseline: -1.2% Memory: ✅ 42.939MB (SLO: <46.000MB -6.7%) vs baseline: +4.2% ✅ bytes_noaspectTime: ✅ 134.014µs (SLO: <200.000µs 📉 -33.0%) vs baseline: -1.7% Memory: ✅ 42.959MB (SLO: <46.000MB -6.6%) vs baseline: +4.7% ✅ bytesio_aspectTime: ✅ 3.794ms (SLO: <5.000ms 📉 -24.1%) vs baseline: -0.4% Memory: ✅ 42.939MB (SLO: <46.000MB -6.7%) vs baseline: +4.5% ✅ bytesio_noaspectTime: ✅ 312.773µs (SLO: <420.000µs 📉 -25.5%) vs baseline: -1.8% Memory: ✅ 42.939MB (SLO: <46.000MB -6.7%) vs baseline: +4.6% ✅ capitalize_aspectTime: ✅ 89.443µs (SLO: <300.000µs 📉 -70.2%) vs baseline: +0.4% Memory: ✅ 42.920MB (SLO: <46.000MB -6.7%) vs baseline: +4.6% ✅ capitalize_noaspectTime: ✅ 248.470µs (SLO: <300.000µs 📉 -17.2%) vs baseline: -1.9% Memory: ✅ 42.959MB (SLO: <46.000MB -6.6%) vs baseline: +4.1% ✅ casefold_aspectTime: ✅ 92.777µs (SLO: <500.000µs 📉 -81.4%) vs baseline: +5.0% Memory: ✅ 42.939MB (SLO: <46.000MB -6.7%) vs baseline: +4.4% ✅ casefold_noaspectTime: ✅ 306.254µs (SLO: <500.000µs 📉 -38.7%) vs baseline: -1.5% Memory: ✅ 42.900MB (SLO: <46.000MB -6.7%) vs baseline: +4.2% ✅ decode_aspectTime: ✅ 86.410µs (SLO: <100.000µs 📉 -13.6%) vs baseline: -0.9% Memory: ✅ 42.900MB (SLO: <46.000MB -6.7%) vs baseline: +4.4% ✅ decode_noaspectTime: ✅ 152.071µs (SLO: <210.000µs 📉 -27.6%) vs baseline: -2.6% Memory: ✅ 42.939MB (SLO: <46.000MB -6.7%) vs baseline: +4.4% ✅ encode_aspectTime: ✅ 84.489µs (SLO: <200.000µs 📉 -57.8%) vs baseline: -0.5% Memory: ✅ 42.959MB (SLO: <46.000MB -6.6%) vs baseline: +4.4% ✅ encode_noaspectTime: ✅ 140.439µs (SLO: <200.000µs 📉 -29.8%) vs baseline: -1.7% Memory: ✅ 42.880MB (SLO: <46.000MB -6.8%) vs baseline: +4.4% ✅ format_aspectTime: ✅ 14.601ms (SLO: <19.200ms 📉 -24.0%) vs baseline: -0.3% Memory: ✅ 43.014MB (SLO: <46.000MB -6.5%) vs baseline: +4.4% ✅ format_map_aspectTime: ✅ 16.469ms (SLO: <21.500ms 📉 -23.4%) vs baseline: ~same Memory: ✅ 42.977MB (SLO: <46.000MB -6.6%) vs baseline: +4.3% ✅ format_map_noaspectTime: ✅ 371.790µs (SLO: <500.000µs 📉 -25.6%) vs baseline: -0.8% Memory: ✅ 42.841MB (SLO: <46.000MB -6.9%) vs baseline: +4.1% ✅ format_noaspectTime: ✅ 303.306µs (SLO: <500.000µs 📉 -39.3%) vs baseline: -1.5% Memory: ✅ 42.920MB (SLO: <46.000MB -6.7%) vs baseline: +4.4% ✅ index_aspectTime: ✅ 121.357µs (SLO: <300.000µs 📉 -59.5%) vs baseline: -0.8% Memory: ✅ 42.939MB (SLO: <46.000MB -6.7%) vs baseline: +4.5% ✅ index_noaspectTime: ✅ 39.914µs (SLO: <300.000µs 📉 -86.7%) vs baseline: -0.4% Memory: ✅ 42.959MB (SLO: <46.000MB -6.6%) vs baseline: +4.5% ✅ join_aspectTime: ✅ 209.871µs (SLO: <300.000µs 📉 -30.0%) vs baseline: -2.6% Memory: ✅ 42.920MB (SLO: <46.000MB -6.7%) vs baseline: +4.3% ✅ join_noaspectTime: ✅ 141.477µs (SLO: <300.000µs 📉 -52.8%) vs baseline: -1.9% Memory: ✅ 42.959MB (SLO: <46.000MB -6.6%) vs baseline: +4.6% ✅ ljust_aspectTime: ✅ 493.628µs (SLO: <700.000µs 📉 -29.5%) vs baseline: -2.7% Memory: ✅ 42.959MB (SLO: <46.000MB -6.6%) vs baseline: +4.5% ✅ ljust_noaspectTime: ✅ 259.664µs (SLO: <300.000µs 📉 -13.4%) vs baseline: -1.0% Memory: ✅ 42.979MB (SLO: <46.000MB -6.6%) vs baseline: +4.5% ✅ lower_aspectTime: ✅ 293.296µs (SLO: <500.000µs 📉 -41.3%) vs baseline: -1.7% Memory: ✅ 42.880MB (SLO: <46.000MB -6.8%) vs baseline: +4.3% ✅ lower_noaspectTime: ✅ 235.811µs (SLO: <300.000µs 📉 -21.4%) vs baseline: -1.7% Memory: ✅ 42.900MB (SLO: <46.000MB -6.7%) vs baseline: +4.3% ✅ lstrip_aspectTime: ✅ 0.270ms (SLO: <3.000ms 📉 -91.0%) vs baseline: -2.8% Memory: ✅ 42.880MB (SLO: <46.000MB -6.8%) vs baseline: +4.4% ✅ lstrip_noaspectTime: ✅ 0.211ms (SLO: <3.000ms 📉 -93.0%) vs baseline: 📈 +18.5% Memory: ✅ 42.939MB (SLO: <46.000MB -6.7%) vs baseline: +4.6% ✅ modulo_aspectTime: ✅ 14.483ms (SLO: <18.750ms 📉 -22.8%) vs baseline: +1.4% Memory: ✅ 42.977MB (SLO: <46.000MB -6.6%) vs baseline: +4.2% ✅ modulo_aspect_for_bytearray_bytearrayTime: ✅ 14.759ms (SLO: <19.350ms 📉 -23.7%) vs baseline: -0.1% Memory: ✅ 43.057MB (SLO: <46.000MB -6.4%) vs baseline: +4.4% ✅ modulo_aspect_for_bytesTime: ✅ 14.420ms (SLO: <18.900ms 📉 -23.7%) vs baseline: -0.1% Memory: ✅ 43.077MB (SLO: <46.000MB -6.4%) vs baseline: +4.4% ✅ modulo_aspect_for_bytes_bytearrayTime: ✅ 14.569ms (SLO: <19.150ms 📉 -23.9%) vs baseline: -0.1% Memory: ✅ 43.018MB (SLO: <46.000MB -6.5%) vs baseline: +4.3% ✅ modulo_noaspectTime: ✅ 0.362ms (SLO: <3.000ms 📉 -87.9%) vs baseline: -0.9% Memory: ✅ 42.880MB (SLO: <46.000MB -6.8%) vs baseline: +4.5% ✅ replace_aspectTime: ✅ 18.516ms (SLO: <24.000ms 📉 -22.8%) vs baseline: +0.3% Memory: ✅ 42.998MB (SLO: <46.000MB -6.5%) vs baseline: +4.2% ✅ replace_noaspectTime: ✅ 282.639µs (SLO: <300.000µs -5.8%) vs baseline: ~same Memory: ✅ 42.959MB (SLO: <46.000MB -6.6%) vs baseline: +4.5% ✅ repr_aspectTime: ✅ 310.261µs (SLO: <420.000µs 📉 -26.1%) vs baseline: -3.1% Memory: ✅ 42.959MB (SLO: <46.000MB -6.6%) vs baseline: +4.3% ✅ repr_noaspectTime: ✅ 46.699µs (SLO: <90.000µs 📉 -48.1%) vs baseline: +0.8% Memory: ✅ 42.959MB (SLO: <46.000MB -6.6%) vs baseline: +4.6% ✅ rstrip_aspectTime: ✅ 383.778µs (SLO: <500.000µs 📉 -23.2%) vs baseline: ~same Memory: ✅ 42.919MB (SLO: <46.000MB -6.7%) vs baseline: +4.2% ✅ rstrip_noaspectTime: ✅ 185.981µs (SLO: <300.000µs 📉 -38.0%) vs baseline: +1.0% Memory: ✅ 42.900MB (SLO: <46.000MB -6.7%) vs baseline: +4.5% ✅ slice_aspectTime: ✅ 185.683µs (SLO: <300.000µs 📉 -38.1%) vs baseline: +2.6% Memory: ✅ 42.979MB (SLO: <46.000MB -6.6%) vs baseline: +4.4% ✅ slice_noaspectTime: ✅ 54.439µs (SLO: <90.000µs 📉 -39.5%) vs baseline: +0.8% Memory: ✅ 42.959MB (SLO: <46.000MB -6.6%) vs baseline: +4.3% ✅ stringio_aspectTime: ✅ 3.829ms (SLO: <5.000ms 📉 -23.4%) vs baseline: -0.9% Memory: ✅ 42.900MB (SLO: <46.000MB -6.7%) vs baseline: +4.4% ✅ stringio_noaspectTime: ✅ 378.007µs (SLO: <500.000µs 📉 -24.4%) vs baseline: +5.4% Memory: ✅ 42.900MB (SLO: <46.000MB -6.7%) vs baseline: +4.5% ✅ strip_aspectTime: ✅ 269.271µs (SLO: <350.000µs 📉 -23.1%) vs baseline: -2.2% Memory: ✅ 42.900MB (SLO: <46.000MB -6.7%) vs baseline: +4.3% ✅ strip_noaspectTime: ✅ 177.869µs (SLO: <240.000µs 📉 -25.9%) vs baseline: +0.7% Memory: ✅ 42.939MB (SLO: <46.000MB -6.7%) vs baseline: +4.4% ✅ swapcase_aspectTime: ✅ 331.219µs (SLO: <500.000µs 📉 -33.8%) vs baseline: -2.4% Memory: ✅ 42.920MB (SLO: <46.000MB -6.7%) vs baseline: +4.2% ✅ swapcase_noaspectTime: ✅ 272.175µs (SLO: <400.000µs 📉 -32.0%) vs baseline: -1.2% Memory: ✅ 42.900MB (SLO: <46.000MB -6.7%) vs baseline: +4.6% ✅ title_aspectTime: ✅ 321.388µs (SLO: <500.000µs 📉 -35.7%) vs baseline: -2.5% Memory: ✅ 42.880MB (SLO: <46.000MB -6.8%) vs baseline: +4.3% ✅ title_noaspectTime: ✅ 257.813µs (SLO: <400.000µs 📉 -35.5%) vs baseline: -1.0% Memory: ✅ 42.920MB (SLO: <46.000MB -6.7%) vs baseline: +4.4% ✅ translate_aspectTime: ✅ 489.785µs (SLO: <700.000µs 📉 -30.0%) vs baseline: -2.9% Memory: ✅ 42.920MB (SLO: <46.000MB -6.7%) vs baseline: +4.2% ✅ translate_noaspectTime: ✅ 423.170µs (SLO: <500.000µs 📉 -15.4%) vs baseline: -2.4% Memory: ✅ 42.998MB (SLO: <46.000MB -6.5%) vs baseline: +4.4% ✅ upper_aspectTime: ✅ 295.770µs (SLO: <500.000µs 📉 -40.8%) vs baseline: -1.4% Memory: ✅ 42.900MB (SLO: <46.000MB -6.7%) vs baseline: +4.3% ✅ upper_noaspectTime: ✅ 236.656µs (SLO: <400.000µs 📉 -40.8%) vs baseline: -0.8% Memory: ✅ 42.939MB (SLO: <46.000MB -6.7%) vs baseline: +4.4% 📈 iastaspectsospath - 24/24✅ ospathbasename_aspectTime: ✅ 511.572µs (SLO: <700.000µs 📉 -26.9%) vs baseline: 📈 +20.1% Memory: ✅ 42.585MB (SLO: <46.000MB -7.4%) vs baseline: +4.0% ✅ ospathbasename_noaspectTime: ✅ 429.920µs (SLO: <700.000µs 📉 -38.6%) vs baseline: -0.6% Memory: ✅ 42.585MB (SLO: <46.000MB -7.4%) vs baseline: +4.4% ✅ ospathjoin_aspectTime: ✅ 628.018µs (SLO: <700.000µs 📉 -10.3%) vs baseline: +0.7% Memory: ✅ 42.526MB (SLO: <46.000MB -7.6%) vs baseline: +3.8% ✅ ospathjoin_noaspectTime: ✅ 629.276µs (SLO: <700.000µs 📉 -10.1%) vs baseline: -1.6% Memory: ✅ 42.664MB (SLO: <46.000MB -7.3%) vs baseline: +4.3% ✅ ospathnormcase_aspectTime: ✅ 352.193µs (SLO: <700.000µs 📉 -49.7%) vs baseline: -0.4% Memory: ✅ 42.507MB (SLO: <46.000MB -7.6%) vs baseline: +4.1% ✅ ospathnormcase_noaspectTime: ✅ 360.252µs (SLO: <700.000µs 📉 -48.5%) vs baseline: +0.9% Memory: ✅ 42.664MB (SLO: <46.000MB -7.3%) vs baseline: +4.4% ✅ ospathsplit_aspectTime: ✅ 492.443µs (SLO: <700.000µs 📉 -29.7%) vs baseline: +0.7% Memory: ✅ 42.585MB (SLO: <46.000MB -7.4%) vs baseline: +4.2% ✅ ospathsplit_noaspectTime: ✅ 498.799µs (SLO: <700.000µs 📉 -28.7%) vs baseline: -0.7% Memory: ✅ 42.526MB (SLO: <46.000MB -7.6%) vs baseline: +3.6% ✅ ospathsplitdrive_aspectTime: ✅ 376.842µs (SLO: <700.000µs 📉 -46.2%) vs baseline: +0.9% Memory: ✅ 42.546MB (SLO: <46.000MB -7.5%) vs baseline: +4.2% ✅ ospathsplitdrive_noaspectTime: ✅ 72.839µs (SLO: <700.000µs 📉 -89.6%) vs baseline: -1.0% Memory: ✅ 42.448MB (SLO: <46.000MB -7.7%) vs baseline: +3.5% ✅ ospathsplitext_aspectTime: ✅ 457.949µs (SLO: <700.000µs 📉 -34.6%) vs baseline: -0.8% Memory: ✅ 42.605MB (SLO: <46.000MB -7.4%) vs baseline: +4.1% ✅ ospathsplitext_noaspectTime: ✅ 466.542µs (SLO: <700.000µs 📉 -33.4%) vs baseline: -0.3% Memory: ✅ 42.644MB (SLO: <46.000MB -7.3%) vs baseline: +4.2% 🟡 Near SLO Breach (1 suite)🟡 tracer - 6/6✅ largeTime: ✅ 31.362ms (SLO: <32.950ms -4.8%) vs baseline: -0.1% Memory: ✅ 36.746MB (SLO: <39.250MB -6.4%) vs baseline: +4.6% ✅ mediumTime: ✅ 3.126ms (SLO: <3.200ms -2.3%) vs baseline: +0.7% Memory: ✅ 35.586MB (SLO: <38.750MB -8.2%) vs baseline: +4.3% ✅ smallTime: ✅ 362.836µs (SLO: <370.000µs 🟡 -1.9%) vs baseline: +4.6% Memory: ✅ 35.586MB (SLO: <38.750MB -8.2%) vs baseline: +4.5%
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 96c33df13a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
KowalskiThomas
left a comment
There was a problem hiding this comment.
This makes sense to me overall and it seems like it handles all cases where we could e.g. not be able to write to/read from disk.
Description
This PR reduces code provenance overhead by:
ddtraceversino, and installed dependency environment.Testing
Added a microbenchmark where we fork 10 workers sequentially each calling
CodeProvenance.json_str_to_export(). Prior to this PR, each worker would build the json file reading all package files. With this PR, only the first worker would do so and the following workers would read from /tmp file cache.Benchmark results on macOS with Apple M3 MAX
Benchmark results on GitLab
Risks
Additional Notes