Skip to content

Use flightCache for index cache implementation#2175

Open
markusthoemmes wants to merge 1 commit intochainguard-dev:mainfrom
markusthoemmes:index-cache-flight-cache
Open

Use flightCache for index cache implementation#2175
markusthoemmes wants to merge 1 commit intochainguard-dev:mainfrom
markusthoemmes:index-cache-flight-cache

Conversation

@markusthoemmes
Copy link
Copy Markdown
Member

Currently, there's a race between store, load and forget if indexes get looked up concurrently and change mid-way through the process. The r ace goes like this:

  1. A: HEAD request → gets etag1, key = "url@etag1"
  2. A: Once.Do runs: fetches, parses, calls store("url@etag1", idx), then sets urlToEtag[url] = etag1
  3. B: HEAD request → gets etag2 (index was updated server-side), key = "url@etag2"
  4. B: Once.Do runs: sees prev = etag1, calls forget("url@etag1") — which deletes indexes["url@etag1"]
  5. A: calls load("url@etag1") → key not found → consistency error

Using the inFlight cache fixes that as values are not stored and loaded separately.

Separately, there's another race where we'd leak indexes with outdated etags if two HEAD requests would race outside of the critical section of adjusting the etag pointers. That's fixed through forgetting all "old" etag values for a given index URL as well.

Currently, there's a race between store, load and forget if indexes get looked up concurrently and change mid-way through the process. The r
ace goes like this:

  1. A: HEAD request → gets etag1, key = "url@etag1"
  2. A: Once.Do runs: fetches, parses, calls store("url@etag1", idx), then sets urlToEtag[url] = etag1
  3. B: HEAD request → gets etag2 (index was updated server-side), key = "url@etag2"
  4. B: Once.Do runs: sees prev = etag1, calls forget("url@etag1") — which deletes indexes["url@etag1"]
  5. A: calls load("url@etag1") → key not found → consistency error

Using the inFlight cache fixes that as values are not stored and loaded separately.

Separately, there's another race where we'd leak indexes with outdated etags if two HEAD requests would race outside of the critical section of adjusting the etag pointers. That's fixed through forgetting all "old" etag values for a given index URL as well.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant