Garbage collect shards in SQS Filesource

[rdettai]

Description

Trigger the prune_shards endpoint from the SQS file sources.

How was this PR tested?

Added integration test.

[github-actions]

On SSD:

Average search latency is 1.0x that of the reference (lower is better).
Ref run id: 3537, ref commit: c9dfb6d
Link

On GCS:

Average search latency is 1.3x that of the reference (lower is better).
Ref run id: 3538, ref commit: c9dfb6d
Link

[rdettai]

There is one footgun here that I would like to avoid: it might happen that a shard is garbage collected right before it's published (e.g deduplication_window_max_messages is small compared to processed message rate). Possible solutions:

• GC only EOF shards, the drawback is that if a lot of junk shards that couldn't be published accumulate, we might end up overwhelming the metastore with a huge shard table
• set a grace period of x*commit_timeout, reducing the situations where the shard could be garbage collected while it was still being worked on. This is not bulletproof though as the indexing might be interrupted for some reason and resume after the grace period, but failing the commit in those rare cases should be supportable