fix(github): paginate owned repos beyond the first 100#46
Open
ChristianUtria wants to merge 1 commit into
Open
fix(github): paginate owned repos beyond the first 100#46ChristianUtria wants to merge 1 commit into
ChristianUtria wants to merge 1 commit into
Conversation
GitHub's repositories field caps at 100 per page. Accounts with more owned repos (e.g. prolific maintainers with 500-1000+ repos) had their languages, total stars, and active-years signals computed only from their top 100 most-starred repos, silently undercounting devs whose less-starred repos use different languages or predate their popular ones. Follows up to 2 extra pages (300 repos total) via cursor pagination, mirroring the existing best-effort batching used for lifetime contributions - a failed extra page degrades gracefully instead of failing the whole scout.
|
A friend of mine has over 110 repositories and his information was incomplete; I think expanding the mapping to 300 repositories is a great move. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
repositories()in the profile query caps at 100 (GraphQL's page limit,orderBy: STARGAZERS DESC). Any account with more than 100 owned,non-fork repos has its languages, total stars, and active-years signals
computed from only those top-100-by-stars repos — the rest are silently
ignored.
Example
@sindresorhushas ~1,100 owned repos. Before this fix, his card showedonly 7 languages despite that repo count — because his most-starred repos
happen to be mostly JS/TS, and everything else (which may use other
languages) falls outside the top 100.
Fix
Follows the first repo page up to 2 extra pages (300 repos total) via
cursor pagination, using the same best-effort pattern already used for
fetchLifetime— a failed extra page degrades gracefully (fewer reposcounted) instead of failing the whole scout. Accounts under 100 repos
(the common case) do zero extra requests.
Why cap at 3 pages (300 repos)?
Repo pages are fetched sequentially (each needs the previous page's cursor),
so they can't run in parallel like the lifetime-contribution batches do.
Given the existing 8s request timeout (Vercel's ~10s serverless cap), 3
pages keeps the worst case comfortably inside that budget while covering
the overwhelming majority of profiles. Accounts under 100 repos pay zero
extra requests. This is a tunable constant (
REPO_MAX_PAGES), not a hardarchitectural limit — easy to revisit if needed.
Verified
npx vitest run— 109/109 passing (2 new tests covering the paginationfollow-through and graceful degradation on failure).
npx tsc --noEmit— clean.Compared
gitfut.com/sindresorhus(prod, unpatched) vslocalhost:3000/sindresorhus(this branch) side by side: