GraphQL Search vs Issues Graph

[Chocrates]

As we see in #42, if we are utilizing the search graphQL api then we are limited to returning the first 1000 results.
If we instead were to use the Issue's graph then we would lose the search functionality and have to potentially reimplement it locally after pulling all the data locally.

What do we think are good options to fix this moving forward?
I can think of a couple options:

• Implement a new command that dumps the Issues without the ability to search, but giving the ability to utilize the filters and leave search how it is while giving the users a warning if the result set is larger than 1k
• Pull all objects down locally and search through it on disk/in memory.

Thoughts?

CC: @jasonmacgowan @zkoppert

[zkoppert]

Any idea how much space the local option takes up? I think Actions runners only have 14gb total to play with.

[Chocrates]

no idea, a handful of bytes/kb per object (issue or PR) times the number of objects in the repo, maybe 10 or 20k at the highest
We could write a short script to grab the JSON object of every issue in a large repo and see.

[zkoppert]

That sounds well within the amount given. Let's go for that!

[RyanCavanaugh]

Pulling all the data (issues, PRs, and their full timeline items) from TypeScript resulted in about 300 MB across the 43,000 items when stored as naive JSON

[Chocrates]

Thanks @RyanCavanaugh! That is a ton of data but not an unworkable amount imo.
We could explore chunking/streaming the data so the memory cost at any one time wouldn't be that bad.
@zkoppert what are you concerned about RE the Actions runners?

My main concern is trying to keep the Search syntax in sync with GitHub.com

[jasonmacgowan]

Since search:issues is tightly coupled to the search api (and name thusly), I agree we should expose exporting all issues from a repository using the other issues endpoints. These other endpoints have their own filters.

https://docs.github.com/en/rest/reference/issues

I'm thinking:

• issues listing issues for auth'd user
• org:issues listing org issues
• repo:issues listing org issues

We'll want to update the way we're writing to disk as @Chocrates where we stream out the results instead of storing all in memory.