Add Healthline mirror + task verifiers (site by @JeremyJC67, verifiers by reviewer)#59
Open
DEM1TASSE wants to merge 4 commits into
Open
Add Healthline mirror + task verifiers (site by @JeremyJC67, verifiers by reviewer)#59DEM1TASSE wants to merge 4 commits into
DEM1TASSE wants to merge 4 commits into
Conversation
Addresses review on PR aiming-lab#53: - Rebase onto current main; register healthline as index 16 -> port 40016 (append after merriam_webster in websyn_start.sh + control_server.py; Dockerfile EXPOSE -> 40000-40016). merriam_webster preserved. - websyn_start.sh site-count comments reconciled to 17. - .assets-revision pinned to HF PR aiming-lab#40 (clean tarball, no macOS AppleDouble junk, based on current main -> all 17 tarballs). Bump to merged SHA once HF PR aiming-lab#40 lands. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reviewer deliverable for the Healthline mirror (site by @JeremyJC67, PR aiming-lab#53): one deterministic verifier per task under sites/healthline/verify/, plus verifier_path + judge_rubric recorded in every tasks.jsonl row. No answer key in tasks.jsonl — ground truth lives only inside the verifiers. Deterministic-first: (1) trajectory navigation gate (anti knowledge-shortcut, important here since several answers are medically recallable), (2) SQLite DB after-state for the stateful tasks (save / register / password-change, plus DB cross-checks for saved-count and reading-history), (3) answer vs frozen ground truth, with the LLM only as an anchored consistency check. Validated against the official react agent (agent_demo/agent.py): a no-op run fails all 20 verifiers; a human answer-check confirms every frozen value matches what the page renders. On the full 20-task run, after fixing one too-strict nav gate the run itself surfaced, verifier and LLM judge agree 15/20, and all 5 remaining divergences are the deterministic verifier being correct while the LLM judge false-positives (blank answer / knowledge-shortcut) or false-negatives (DB-confirmed save / password change). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
5 tasks
Search result cards showed drug_class (e.g. 'ACE inhibitor'), letting T16 be answered from the results page without opening the drug pages. Show the broader category (e.g. 'Heart Medications') instead — same card layout, but the ACE-inhibitor-vs-statin distinction now requires opening each detail page. drug_class still shown on the drug detail page; search backend still matches on drug_class. Addresses DEM1TASSE review note on PR aiming-lab#53. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- web: all 20 tasks pointed at :40015 (Merriam-Webster) after the site was rebased to index 16 / :40016; corrected to http://localhost:40016/. - judge_rubric for Healthline--3 and Healthline--16: spell out the full ground-truth answer the page states (T3 symptom list, T16 what each drug treats) so the LLM judge grades against complete facts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the Healthline mirror + its task verifiers into
mainas one PR, following the samefork-of-fork pattern as CarMax (#57) and NVIDIA (#58). Companion to the site PR #53.
Authorship is preserved — both contributors stay credited. This branch is @JeremyJC67's
Healthline site commits with the reviewer's verifier commit on top; each commit keeps its
original author (site = @JeremyJC67, verifiers = the reviewer). Please merge with a regular
merge (not squash) so both signatures are kept.
Contents
sites/healthline/site (Flask app, templates, seed, 20 tasks) — @JeremyJC67 (from feat: add Healthline site mirror (20 tasks, hardened) #53)sites/healthline/verify/verify_<0..19>.py+verify_lib.py, andverifier_path+judge_rubricin everytasks.jsonlrow — reviewer deliverable. Noanswerkey; groundtruth lives only inside the verifiers.
Why this is close to mergeable
Unlike the other verifier PRs, #53 was already rebased onto current
mainby the author:Healthline is registered at index 16 / port 40016 with Merriam-Webster preserved at 40015, so
this PR does not conflict. I independently re-verified: 17/17 sites return 200, byte-identical
reset (
a8a9b54…) matches the seed and the author's reported hash, and reset-all runs in ~1 s.Verifier design & validation
Deterministic-first per task: navigation gate (anti knowledge-shortcut — important since several
medical facts are LLM-recallable) + SQLite DB after-state for the stateful tasks (save / register /
password-change) + answer vs frozen ground truth, LLM only as an anchored check. Validated with
the official react agent: no-op fails all 20; a human answer-check confirms every frozen value
matches the rendered page. On the full 20-task run, after fixing one too-strict nav gate the run
surfaced, verifier and LLM judge agree 15/20, and all 5 remaining divergences are the verifier
being correct while the LLM judge false-positives (blank answer / knowledge-shortcut) or
false-negatives (DB-confirmed writes).
One dependency
.assets-revisionpoints at HF discussion #40 (the author's clean asset PR). Merge that HF PR,then repin
.assets-revisionto its merged SHA. A full review is posted as a comment on #53.