fix(pageserver): drain upload queue before offloading timeline

[skyzh]

Problem

It is possible (is it?) at the point we shutdown the timeline, there are still layer files we did not upload.

Summary of changes

• If the queue is not empty, avoid offloading.
  • Hmm... is it correct? We don't explicitly wait for files to be uploaded in the compaction code, and we offload the timeline right after compaction, which means that it will always fail if a compaction gets scheduled and triggers timeline offload?
• Otherwise, shutdown the timeline gracefully using the flush mode to ensure all local files are uploaded before deleting the timeline directory.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

5328 tests run: 5106 passed, 0 failed, 222 skipped (full report)

---

Code coverage* (full report)

• functions: 31.6% (7801 of 24714 functions)
• lines: 49.0% (61173 of 124811 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
bc85b03 at 2024-11-07T20:15:51.064Z :recycle:}