Summary
When the SPARQL construct executor retries a transient error (packages/pipeline/src/sparql/executor.ts), it uses its own backoff schedule and ignores any Retry-After header on the response. For rate-limited endpoints (HTTP 429, and 503 with a Retry-After), the server tells us exactly how long to wait — we should honour that instead of guessing.
Why
Retry-After is the servers explicit instruction for how long to back off, as a delay in seconds or an HTTP date. Honouring it is both more polite (we stop hammering an endpoint that is already struggling) and more effective (we wait the right amount rather than retrying too soon and getting blocked again, or too late and wasting the runs time budget).
This surfaced while harvesting an endpoint that rate-limits a burst of heavy analysis queries. That particular endpoint returned a bare Apache 403 with no Retry-After, so this would not have helped that case — but any well-behaved endpoint that returns 429/503 with a Retry-After should have it respected.
Proposed change
When an attempt fails with a retryable status that carries a Retry-After header:
- Parse
Retry-After (both the delta-seconds and the HTTP-date forms).
- Use it as the next backoff delay, ideally clamped to a sane maximum and with a little jitter, overriding the default exponential schedule.
- Fall back to the existing backoff when the header is absent or unparseable.
This depends on the retry path being reachable for the relevant statuses (e.g. 429 — tracked in the sibling issue) and may require surfacing response headers through the fetch error, which fetch-sparql-endpoint does not currently expose on the thrown error.
Scope / non-goals
- Does not change which statuses are considered transient (see the sibling 429 issue).
- Does not add proactive request throttling to avoid tripping limiters in the first place.
Summary
When the SPARQL construct executor retries a transient error (
packages/pipeline/src/sparql/executor.ts), it uses its own backoff schedule and ignores anyRetry-Afterheader on the response. For rate-limited endpoints (HTTP 429, and 503 with aRetry-After), the server tells us exactly how long to wait — we should honour that instead of guessing.Why
Retry-Afteris the servers explicit instruction for how long to back off, as a delay in seconds or an HTTP date. Honouring it is both more polite (we stop hammering an endpoint that is already struggling) and more effective (we wait the right amount rather than retrying too soon and getting blocked again, or too late and wasting the runs time budget).This surfaced while harvesting an endpoint that rate-limits a burst of heavy analysis queries. That particular endpoint returned a bare Apache 403 with no
Retry-After, so this would not have helped that case — but any well-behaved endpoint that returns 429/503 with aRetry-Aftershould have it respected.Proposed change
When an attempt fails with a retryable status that carries a
Retry-Afterheader:Retry-After(both the delta-seconds and the HTTP-date forms).This depends on the retry path being reachable for the relevant statuses (e.g. 429 — tracked in the sibling issue) and may require surfacing response headers through the fetch error, which
fetch-sparql-endpointdoes not currently expose on the thrown error.Scope / non-goals