Skip to content

feat(filegen): deterministic replay mode and seeded random mode (BP-1303) - #343

Open
Dylan-M wants to merge 4 commits into
mainfrom
dylanmyers/bp-1303-filegen-replay-mode
Open

Dylan-M wants to merge 4 commits into
mainfrom
dylanmyers/bp-1303-filegen-replay-mode

Conversation

@Dylan-M

@Dylan-M Dylan-M commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Proposed Change

Add deterministic replay mode in addition to the normal random read mode. In this mode (generator.filegen.mode: replay), Blitz sends every line of a capture file once, in order, then ends the run. Lines without timestamp directives replay byte for byte. There is also a new generator.filegen.seed that makes random-mode line picks repeatable.

count.Tracker.Acquire now returns a Permit that the generator settles with Commit or Release. Embedders calling count.Tracker directly need the new API.

Checklist
  • Changes are tested
  • CI has passed

@Dylan-M
Dylan-M requested review from a team as code owners October 30, 2026 14:29
@Dylan-M
Dylan-M added this pull request to stack #347 October 30, 2026 14:29
@Dylan-M Dylan-M changed the title dylanmyers/bp 1303 filegen replay mode feat(filegen): deterministic replay mode and seeded random mode (BP-1303) Oct 30, 2026
@Caleb-Hurshman Caleb-Hurshman self-assigned this Oct 30, 2026
@Caleb-Hurshman

Copy link
Copy Markdown

A couple things that Claude flagged:

  1. Replay isn't byte-for-byte. Every replay goes through processTimestamps, which means percent-encoded URLs will get interpreted like GET /caf%C3%A9 -> /caf%C3Friday9
  2. A failed record is retried without a cap, so a finite run may never end.
    • Hostmetrics is especially risky. If one scraper fails, the whole scrape is released
  3. Seeding doesn't make random mode repeatable. A retry after a failed write makes a new rng.Intn, shifting every later pick

@Caleb-Hurshman Caleb-Hurshman left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pretty cool, see my comment above for stuff that needs tweaking

…es up a seeded filegen pick

Assisted-By: Claude Opus 5.5
@Dylan-M

Dylan-M commented Oct 30, 2026

Copy link
Copy Markdown
Contributor Author

The byte-for-byte limit is documented in filegen.md. Replay's retry cap is in #345, which drops a line after three failed retries. A run whose destination keeps rejecting one hostmetrics scraper still waits, since the count is of delivered records. That rejection now logs at error, so the stall shows up. The seed docs now say a failed send uses up a pick, so delivered lines only repeat when every send succeeds.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants