WIP: Huginn v1.3 — PR 3/3: Final (layer 3 primitives + SDK refactor) - #8
Merged
Conversation
Null-Phnix
marked this pull request as draft
June 15, 2026 21:06
…/ recover
Adds per-event adaptive throttling to DomainRateLimiter. The static
limiter is now also reactive: when the upstream server returns 429/5xx
or times out, the rate is automatically reduced; when things go well,
the rate gradually recovers.
New API:
- record_outcome(domain, success, status_code=None) — caller reports
a request outcome. 4xx/5xx and timeouts are caller-classified as
failure (success=False).
- get_current_rate(domain) — returns the current rate multiplier
(1.0 = full, 0.1 = heavily backed off)
- get_outcome_history(domain) — recent outcome log (capped at 100)
- DomainLimitConfig.with_rate_multiplier(m) — new helper to build a
scaled config (burst_size clamped to >= 1)
Algorithm: per-event adjustment (not windowed)
- failure → multiplier *= 0.7, floor 0.1
- success → multiplier += 0.05, ceiling 1.0
Why per-event: simpler, more predictable, no 'stuck in backoff' state
from old failures lingering in a sliding window. 3 consecutive
failures → 0.7³ ≈ 0.34 (clearly backed off). 14 consecutive successes
after that → 0.34 + 14×0.05 = 1.04, capped at 1.0 (fully recovered).
Bonus bug fix: when window_seconds=0, the sliding window fallback was
incorrectly allowing requests (timestamp cleaning would empty the list,
then 0 < rpm is True). Now window_seconds=0 means 'no window fallback'
— bucket is the only constraint, which is what the user opted in to.
Tests: 9 new tests in tests/test_adaptive_throttling.py (all pass).
Full suite: 680 passed, 6 deselected.
… per URL
When a user scrapes a URL with include_tags=['.article'] and gets good
results, SelectorMemory records that pattern. Next time they scrape
the same URL, get_suggestions() returns the selectors that worked,
sorted by score (success count × recency decay).
Public API (huginn.selector_memory.SelectorMemory):
- record_success(url, selector) — store a successful selector use
- get_suggestions(url, limit=5) — best selectors for a URL, sorted
- get_stats(url) — Counter of {selector: success_count} for a URL
- get_all_stats() — Counter for all remembered URLs
- get_last_used(url, selector) — timestamp of last success
- forget(url) / forget(url, selector) — cleanup
- clear() — wipe all memory
Algorithm:
- Score = count * recency_factor. recency is 1.0 (last hour) /
0.5 (last day) / 0.1 (older). Sort by score desc, then by name.
- Bounded: max_selectors_per_url=20 (LRU-evict by lowest score),
max_urls=1000 (LRU-evict by last-used).
- Empty/whitespace selectors are ignored.
Why per-URL, in-memory: this is a heuristic, not authoritative.
Stays out of the way of cache invalidation. Persistence would be
a layer 3 enhancement once we know which selectors actually get
reused in practice.
Tests: 16 new tests in tests/test_selector_memory.py (all pass).
Full suite: 696 passed, 6 deselected.
…now canonical
The 'two-SDK smell' is resolved. sdk/python/huginn_client/ is the
single source of truth for HuginnClient, HuginnSync, and the error
hierarchy. huginn/sdk.py (the internal duplicate) is deleted.
Changes:
- sdk/python/huginn_client/__init__.py:
- HuginnError: added error_code= param (matches internal SDK shape)
- New: CircuitOpenError, RateLimitError (backported from internal)
- New: HuginnSync class (sync wrapper, ~100 lines)
- HuginnClient: added public .timeout attribute + async close() method
- HuginnClient.__aexit__: now calls self.close() (single source of truth)
- HuginnSync._run: enters/exits the async context automatically
(caller no longer needs to async with the client first)
- HuginnSync.scrape: now an alias for probe() (matches internal SDK
naming, back-compat for downstream users)
- Bumped __version__ 1.0.0 → 1.2.0 (mark internal SDK removal)
- huginn/__init__.py:
- sys.path manipulation: adds sdk/python/ in dev mode (no pip
install needed)
- from huginn_client import HuginnClient, HuginnError, HuginnSync,
CircuitOpenError, RateLimitError (no more fallback to internal)
- Bumped Huginn __version__ 1.2.0 → 1.3.0
- tests/test_sdk.py:
- Rewrote to test the public SDK API (was testing internal)
- Added 17 new tests:
- TestCircuitAndRateLimitErrors (7 tests): message preservation,
status_code/error_code/response attachment, raise+catch
- TestHuginnSyncMethods (10 tests): health() / scrape() /
probe() with httpx.MockTransport (no real server needed),
close-without-context safety, sync method existence
- Tests now use sys.path manipulation to find the published SDK
in the source tree
Impact for users:
- pip install huginn-client users: no change — they already had the
public SDK. New classes (CircuitOpenError, RateLimitError,
HuginnSync) are now available.
- Huginn self-hosters: no change. The internal SDK was never
public-facing, and Huginn's own code now imports from the
public package which has more features.
- One canonical SDK, no drift risk. Drift prevention achieved.
Tests: 723 passed, 6 deselected, 1 warning (StarletteDeprecation).
Up from 696 (pre-SDK refactor).
Null-Phnix
force-pushed
the
nuwa/huginn-pr3-final
branch
from
June 17, 2026 19:29
e07b092 to
0cf7e76
Compare
Null-Phnix
marked this pull request as ready for review
June 17, 2026 19:30
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR 3 of 3: Huginn v1.3 — Final (layer 3 primitives + SDK refactor)
Series: 3 stacked PRs.
Status: draft, do not merge until the v1.3 series is reviewed.
What this PR is
The last 3 commits of the v1.3 series. Two parts:
huginn/sdk.py(812 lines), promotes the publishedhuginn-client(PyPI package) to canonical. The two-SDK smell is resolved.Stats (PR 2 → PR 3)
huginn/selector_memory.py(155 lines)huginn/sdk.py, 771 lines)Features (3 commits)
feat(rate-limiter):adaptive throttling — observe 429/5xx outcomes, back off per-event (21738c6)feat(memory):SelectorMemory — remembers successful CSS selectors per URL (de1a579)refactor(sdk):kill internalhuginn/sdk.py— publishedhuginn-clientis canonical (5259306)SDK refactor impact
pip install huginn-clientusers get 3 new classes:CircuitOpenError,RateLimitError,HuginnSyncHuginnClientgot public.timeoutattribute and properasync close()methodHuginnSync._runauto-enters the async context — callers no longer needasync with sync:HuginnSync.scrapeis an alias forprobe()(back-compat for users expecting the Firecrawl-style name)sys.pathshim in dev modeAdaptive throttling algorithm
Per-event adjustment (not windowed):
multiplier *= 0.7, floor 0.1multiplier += 0.05, ceiling 1.03 consecutive failures → 0.7³ ≈ 0.34 (clearly backed off). 14 consecutive successes after that → 0.34 + 14×0.05 = 1.04, capped at 1.0 (fully recovered).
Caller decides success/failure classification (4xx/5xx + timeouts = failure is the typical pattern).
How to test