refactor: Make SimpleRetriever thread-safe so that different partitions can share the same SimpleRetriever

[brianjlai]

What

Right now, because the SimpleRetriever has an internally managed and modified state, we cannot have multiple partitions running at the same time share the same SimpleRetriever. Otherwise we might run into data loss because the state of the SimpleRetriever (things like page number, etc) are modified by different partitions. We've seen this problem arise in some connections (link issue) and we have temporarily solved this by having every Partition instantiate it's own SimpleRetriever.

This however is very inefficient for a couple reasons like needing to perform auth for every partition (link issue). And this is a hard blocker on AsyncRetriever which must be shared across partitions in order to manage the shared job repository.

This PR replaces an internal state for SimpleRetriever and all of its dependencies which are manged via a token field and makes all methods stateless by relying on passing parameterized values for the next_page_token instead.

How

Refactor retrievers, paginators, and pagination strategy to be stateless