Copying very large repo (30k issues, 33k PRs)

[velle]

I want to keep a local copy of metadata for qgis/QGIS, which is quite large with 30k issues and 33k pull requests. Has anyone used github-to-sqlite for anything like that?

The GitHub API has a rate limit of 5,000 API calls per hours, for authenticated users. And I think it takes one call for each issue - am I right? And github-to-sqlite does not offer only fetching only some subset of the issues, and furthermore does not commit to sqlite before all issues are fetched (so in case a call is interrupted, e.g. by the rate limit, the fetched data is simply discarded).

So I don't see any way that I can actually use github-to-sqlite for this (unless I pay for such privileges to github). Can you confirm?

Sincerely, Thomas

[velle]

And I think it takes one call for each issue - am I right?

I was wrong about that. As far as I can tell from looking at the code, all data is fetched via the REST API (as opposed to the GraphQL API) using pagination, with 100 rows per call. So 33000 issues is only 330 API calls.

Fetching all issues from qgis/QGIS took roughly two hours:

$ time github-to-sqlite issues github.db qgis/QGIS
real    129m16.692s

But to my surprise around 62k rows were inserted into the issues table, as opposed to the expected 33. I found out that pull requests are also included. If I don't count those, then the numbers match.

sqlite> select count(*)
from issues
where repo = 1690480 and pull_request is null
;
33148