feat: wire layer-3 hooks + CI doctor check + test coverage - #9
Merged
Conversation
…mory The adaptive throttling and selector memory modules were added in PR #8 (v1.3) as standalone, tested modules — but the scraper never called them. This commit wires them into the real scrape path. Adaptive throttling (DomainRateLimiter.record_outcome): - After a successful scrape: record_outcome(domain, success=True) so the rate multiplier can recover from past failures (+0.05 per success, ceiling 1.0). - After a failed scrape: record_outcome(domain, success=False) so the multiplier backs off (×0.7, floor 0.1). - The scraper already had rl = self._rl or get_domain_rate_limiter() so the limiter is the same instance the rest of the code uses. Selector memory (SelectorMemory): - When include_tags is provided and the scrape succeeds, each selector is recorded via record_success(url, selector). - When include_tags is empty, suggestions from memory are surfaced in result.metadata['selector_hints'] as {selector, score} pairs. - Lazy singleton via get_selector_memory(); tests can override scraper._selector_memory directly. Also fixes api.py dt.fromisoformat → datetime.fromisoformat (the import was 'from datetime import datetime, timezone' but the code used the 'dt' alias which was never imported in that scope).
Runs the deep health check after unit tests. Uses '|| true' because the check may WARN on missing LLM credentials in CI (expected — the server can run scrapes without an LLM key). The step is informational for now; it surfaces broken Playwright or webhook signing issues in the CI log without failing the build.
… mock 8 new tests verifying adaptive throttling (record_outcome called with success=True/False) and selector memory (records successful include_tags, surfaces hints when include_tags empty, skips on error status, no hints when memory empty). Fix: test_lightweight_scrape_also_includes_language didn't set mock_browser.last_status_code, causing TypeError when layer-3 hooks compared status_code < 400. Added explicit last_status_code=200.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
v1.4 — Layer-3 hooks wiring + cleanup
What changed
Layer-3 hooks wired into
Scraper.scrape()— the main "NOT done" item from v1.3:record_outcome(domain, success=True/False)called after every scrape, feeding the per-domain rate multiplier so it backs off on failures and recovers on successes.include_tagsare recorded per-URL; when future scrapes of the same URL omitinclude_tags,selector_hintsappear inmetadatawith scored suggestions.CI:
huginn doctor --checkadded as an informational step in the test workflow.Bug fix:
api.py—dt.fromisoformat()→datetime.fromisoformat()(undefineddtalias).Tests: 8 new tests in
tests/test_layer3_hooks.pycovering both hooks. Fixedtest_lightweight_scrape_also_includes_languagemock to setlast_status_code.Test count
Commits
feat(scraper): wire layer-3 hooks — adaptive throttling + selector memoryci: add huginn doctor --check step to test workflowtest(scraper): add layer-3 hook wiring tests + fix language detection mock