Investigate source of build_report_from_commit Slowness

[ajay-sentry]

Example Trace:

https://codecov.sentry.io/performance/trace/d31835da6f49465bb199bd5c867bcc5f?fov=0%2C9833.00011062622&node=span-b89fb8f4f4a52139&node=txn-69c34a8b5e39456f8d5f1b4c2d79a32a&node=txn-e7f6eef673764e6e99895937c9414616

Related comment with a bit extra context:

#1921 (comment)

[trent-codecov]

@Swatinem please take a look at this next sprint and provide recommendations for improvements

[Swatinem]

@ajay-sentry I can’t access the trace you linked above, possibly it has already expired.

Can you remember whether the trace had a ton of SQL calls in it, or whether it was purely download related?

The build_report_from_commit function fundamentally does 2 file downloads from GCS: the chunks file, and either the report_json file or the files_array file.

These are IO bound, and we are at the mercy of GCS here, which is notoriously slow and has inconsistent latency. Sentry built its own filestore service specifically to improve on those GCS limitations, but comes with its own set of problems and challenges.

Apart from optimizing GCS usage in itself, there is #2257 which should help both with CPU usage for de/compression, as well as improve the size we have to store.

I spun out #2553 as another issue to optimize the GCS bucket configuration.

---

If you have indeed seen tons of SQL queries, it might be related to #2554, though that feature flag is limited to only 4 repos at this time.

[ajay-sentry]

@Swatinem Super weird the link didn't work, was able to access it on my end still. Attached a screenshot above, but the trace looks to have a 4 second gap where nothing is happening; not sure if you knew what that could indicate

Amazing investigation, massive kudos. Just from reading all those issues I feel like I've learned so much 😂

[Swatinem]

the trace looks to have a 4 second gap where nothing is happening; not sure if you knew what that could indicate

This looks like api code (indicated by the shared.api_archive span), which I haven’t touched directly.
Its quite possible that it does some heavy work within the "fetch files/sessions/totals" span.

Its weird though that I can’t load that trace myself. I will ask internally about that.