WIP: Huginn v1.3 — 11 Firecrawl parity features + 5 bug fixes - #5
Closed
Null-Phnix wants to merge 31 commits into
Closed
WIP: Huginn v1.3 — 11 Firecrawl parity features + 5 bug fixes#5Null-Phnix wants to merge 31 commits into
Null-Phnix wants to merge 31 commits into
Conversation
The ErrorCode enum values are part of the public API (callers parse them in their error handlers). The new TestErrorCode class pins the contract: - UNAUTHORIZED is exactly the lowercase string 'unauthorized' - Every value matches a lowercase-or-underscore identifier pattern - from_http_status(401) returns the right member Also adds .nuwa/plans/2026-06-15_huginn-improvement-pass.md documenting the rest of this Nüwa improvement pass: real bugs found, false positives that were re-verified out, and the small-commit plan.
The package is 1.2.0 but huginn/api.py had three hardcoded '1.1.0' strings: the FastAPI app's version kwarg, /health, and /health/detailed. All three are now sourced from huginn.__version__ so a future bump takes effect automatically. Adds two tests in TestApiVersionConsistency that pin the contract: the FastAPI app's .version and /health's version field both equal huginn.__version__.
huginn/sdk.py was the only place shipping :8000 as the default Huginn server URL. Every other surface — cli.py, config.py, docker-compose, huginn_cli.py, sdk/python/ — uses :7432. So HuginnClient() and HuginnSync() out of the box were trying to talk to nothing. Fixed 4 sites in huginn/sdk.py (HuginnClient default, HuginnSync default, the docstring on each, the top-of-file usage example), and updated the two test_sdk.py assertions that had been locking the wrong port in place.
The actual server default is 7432 (see huginn/config.py ServerConfig.port and docker-compose.yml). The README's quick-start curl examples, the 'huginn serve --port' example, the HUGINN_PORT=… env example, and the docs/UPGRADE_PLAN.md docker-run example all still said 8000 — fixed to 7432. No code or test changes needed for this commit.
…to 348 pyproject.toml [tool.pytest.ini_options].testpaths was just ["tests"], which silently excluded huginn/tests/ (32 tests). Running 'pytest' with no arguments therefore reported 311 passing while the README badge claimed 343 — a quiet lie that hid the inner test directory from anyone running the default command. Fix: testpaths = ["tests", "huginn/tests"]. Now 'pytest' collects all 348 tests (343 baseline + 5 new TestErrorCode + 2 new TestApiVersionConsistency added in earlier commits this pass) and the 6 deselected network tests. README badge updated to match. If a future test directory is added, the testpaths list will need updating; the TestApiVersionConsistency tests already cover the 'version is consistent' half of the contract.
BrowserConfig.user_agent was hardcoded to 'Huginn/1.1 (+...)' while
__version__ is '1.2.0'. The default UA now derives from __version__:
user_agent: Optional[str] = field(default_factory=_default_user_agent)
where _default_user_agent() returns f'Huginn/{__version__} (+...)'.
Two new tests in TestUserAgentDefaults (test_config.py) pin the
contract: the default must contain the current __version__ and must
not contain any stale hardcoded version (1.0.0, 1.1.0, 1.1).
Also bumps .env.example:
HUGINN_USER_AGENT=Huginn/1.1 → Huginn/1.2
comment default: Huginn/1.1 → default tracks __version__
Also adds .nuwa/specs/2026-06-15_huginn-personal-firecrawl-replacement.md:
the strategic spec for making Huginn a 1:1 personal Firecrawl
replacement. Three layers (branding, endpoint parity, reliability),
what to ship first, what not to do. Re-read before any session work
on Huginn.
…from it
The 'Huginn' / 'HUGINN_' / 7432 / UA-template / banner brand surface was
scattered across huginn/__init__.py, cli.py, api.py, and config.py.
This is layer 1 of the personal-Firecrawl-replacement spec
(.nuwa/specs/2026-06-15_huginn-personal-firecrawl-replacement.md):
centralize so a future rebrand is one file.
What huginn/_branding.py owns:
- name / short_name / binary / package / python_module
- ua_template + ua(version) helper (default reads __version__)
- color, tagline, description, ascii_banner
- env_prefix (every HUGINN_* env var is f-built from this)
- project_url / docs_url / repo_url
What was refactored:
- config._default_user_agent() → _branding.ua()
- config._apply_env() env_map → f-strings with _branding.env_prefix
(so a prefix rebrand like HUGINN_ → RAVEN_ is a one-line edit)
- cli._banner() → uses _branding.name
- cli @click.group → help= f-string built from _branding
- cli @click.version_option → prog_name=_branding.name
- api FastAPI app title + description → _branding.name + _branding.description
New tests (19):
- tests/test_branding.py — 18 tests pinning the contract of every
branding constant + helper. If a rebrand happens, the tests prove
the surface still works.
- tests/test_cli.py — 2 new tests: help_uses_branding_name and
version_uses_branding_name. The pre-existing test_main_help was
loosened from 'Huginn' in output to '_branding.name in output'.
Test count: 350 → 369 (19 new). 6 network still deselected. ~7s.
…ert verification
Adds the skipTlsVerification (snake_case: skip_tls_verification) field
to ScrapeRequest with alias, defaulting to True for backward-compat with
Huginn's historical hardcoded ignore_https_errors=True behaviour.
Wire:
ScrapeRequest.skip_tls_verification
→ Scraper.scrape(skip_tls_verification)
→ Scraper._scrape_impl(skip_tls_verification)
→ BrowserManager.ignore_https_errors (instance attr, set before new_context)
→ Playwright new_context(ignore_https_errors=…)
The instance-attribute approach avoids threading the param through the
circuit-breaker cb.call() wrapper while keeping existing test mocks intact
(bare AsyncMock has no ignore_https_errors attr → getattr falls back to True).
Tests: 6 new tests (TestScrapeRequestTlsField × 4, TestBrowserContextTlsFlag × 2)
all passing. Full suite: 375 passed, 6 deselected.
Adds 114 tests for includeTags/excludeTags CSS-selector filtering: - Model accepts any CSS selector as include_tags or exclude_tags (17 standard tags + 8 extended selectors) - Scraper._filter_tags() calls page.evaluate with querySelectorAll for each tag - Verifies: script, style, img, a, svg, iframe, div, section, article, aside, footer, header, nav, main, table, ul, ol + extended (.ads, #id, [attr], picture, video, audio, canvas, pre, code, blockquote) - No evaluate calls when neither include nor exclude tags are specified - All 375 original tests still pass.
Language is extracted in both full browser and lightweight paths: - Full browser: from browser.extract_content() which reads document.documentElement.lang - Lightweight: from <html lang='...'> via BeautifulSoup - Default: 'en' when no lang attribute is present - metadata.language key always present in ScrapeData.metadata dict Tests: 4 new tests (all 4 pass). Full suite: 379 passed, 6 deselected.
…mary - ScrapeRequest.summary: bool = False (firecrawl parity) - ScrapeResponse.summary: Optional[str] = None - New _summarize_text() helper in api.py supports OpenAI, xAI, Ollama, Anthropic, Google; best-effort (returns None on no key / network error) - Scrape endpoint calls _summarize_text on data.markdown when req.summary=True - Module-level helper so it can be unit-tested in isolation Tests: 7 new tests in tests/test_summary_field.py (all pass). Full suite: 500 passed, 6 deselected.
- ScrapeRequest.mobile: bool = False - BrowserManager.new_context(device=...) merges device fields into context_kwargs (viewport, user_agent, is_mobile, has_touch, etc.) - Scraper.scrape(mobile=False) passes _MOBILE_DEVICE (iPhone 13 descriptor) to new_context() when mobile=True - Module-level _MOBILE_DEVICE constant — matches Playwright's official iPhone 13 device descriptor Tests: 10 new tests in tests/test_mobile_emulation.py (all pass). Full suite: 510 passed, 6 deselected.
- ScrapeRequest.block_ads: bool (camelCase: blockAds) — Playwright route interception for 24 well-known ad networks (doubleclick.net, googlesyndication.com, googleadservices.com, googletagmanager.com, amazon-adsystem, criteo, outbrain, taboola, scorecardresearch, adsrvr, adnxs, rubiconproject, pubmatic, openx, moatads, facebook, twitter, yahoo, linkedin, adform, etc.) - ScrapeRequest.remove_base64_images: bool (camelCase: removeBase64Images) — strips data:image/...;base64,... URIs from extracted markdown via _BASE64_IMAGE_REGEX - _AD_DOMAINS and _BASE64_IMAGE_REGEX module-level constants Tests: 12 new tests in tests/test_block_ads_and_base64.py (all pass). Full suite: 522 passed, 6 deselected.
includeTags=['iframe'] already works via the existing _filter_tags implementation. This commit adds 5 tests verifying: - iframe elements preserved in result.html with src URL captured - exclude_tags=['iframe'] strips iframe elements - Multiple iframes on a page all extracted - 'iframe' is passed to querySelectorAll correctly - iframe + video combined tags work Tests: 5 new tests in tests/test_iframe_extraction.py (all pass). Full suite: 527 passed, 6 deselected.
- CrawlRequest.max_concurrency: Optional[int] (1-100, alias: maxConcurrency) - CrawlRequest now uses model_config = ConfigDict(populate_by_name=True) so both max_concurrency and maxConcurrency are accepted - All 3 Crawler() instantiation sites in api.py now use concurrency=req.max_concurrency or _config.crawl.concurrency, so per-job override falls back to global default cleanly Tests: 12 new tests in tests/test_crawl_max_concurrency.py (all pass). Full suite: 539 passed, 6 deselected.
- FlockRequest.ignore_invalid_urls: bool (alias: ignoreInvalidURLs) - FlockRequest now uses model_config = ConfigDict(populate_by_name=True) - New _is_valid_http_url() helper in scraper.py — validates scheme in (http, https) + non-empty netloc - flock_scrape endpoint skips invalid URLs with a warning when flag is True; raises 422 when False (default) Tests: 18 new tests in tests/test_ignore_invalid_urls.py (all pass). Full suite: 557 passed, 6 deselected.
- MapRequest.sitemap: Optional[str] (default 'include')
- MapRequest now uses model_config = ConfigDict(populate_by_name=True)
- Mapper.map_site(sitemap='include|skip|only') routes to the right strategies:
'include' (default) — fetch sitemap + crawl page + check sub-sitemaps
'skip' — page crawl only, no sitemap fetch at all
'only' — sitemap URLs only, no page crawl
- Invalid sitemap values fall back to 'include'
- /v1/chart endpoint passes req.sitemap or 'include' as defensive default
Tests: 14 new tests in tests/test_sitemap_mode.py (all pass).
Full suite: 571 passed, 6 deselected.
- ScrapeRequest.change_tracking: bool (alias: changeTracking)
- ScrapeData.change_tracking: Optional[Dict[str, Any]] containing
{previous_hash, current_hash, diff, changed} when populated
- New ChangeTracker module (huginn/change_tracker.py):
- compute_hash(content) — SHA-256 hex prefix (16 chars)
- compute_diff(old, new) — unified diff (empty if identical)
- check_and_store(url, content) — async-safe, returns the dict
- Module-level singleton via get_change_tracker()
- Scraper._scrape_impl() populates result.change_tracking after
markdown extraction when change_tracking=True
- Scraper._change_tracker is lazily initialized on first use
Tests: 22 new tests in tests/test_change_tracking.py (all pass).
Full suite: 593 passed, 6 deselected.
- New _compute_signature(body, secret) helper — HMAC-SHA256 hex digest
- send_webhook() now adds X-Huginn-Signature: sha256=<hex> header
when a secret is set, signed over the JSON-serialized body
- Legacy X-Huginn-Webhook-Secret header still sent for backward compat
- Body is serialized once (with compact separators) so the signature
is computed over the exact bytes that go on the wire
- User-Agent header set to 'Huginn-Webhook/1.0' for receiver-side logs
Receiver-side verification (any language):
body = await request.body() # raw bytes
expected = hmac.new(secret.encode(), body, sha256).hexdigest()
assert request.headers['X-Huginn-Signature'] == f'sha256={expected}'
Tests: 11 new tests in tests/test_hmac_webhook.py (all pass).
Full suite: 604 passed, 6 deselected.
The default huginn doctor only checks Python + deps + Chromium. With --check, runs 4 deeper checks that verify Huginn is operationally healthy end-to-end: - Change tracking: round-trip verify (hash + diff + storage work) - Webhook HMAC: round-trip verify (signing matches stdlib) - LLM credentials: warns (not fails) if no key for configured provider - API server: probes HUGINN_API_URL/health if set; SKIP otherwise New module huginn/doctor.py with: - CheckStatus enum (OK/WARN/FAIL/SKIP) - CheckResult frozen dataclass - Individual check_* functions, each returning CheckResult - run_all_checks() orchestrator with last-resort exception safety net - summarize() helper for OK/WARN/FAIL/SKIP counts CLI changes: - huginn doctor --check: run deep checks (new) - huginn doctor: unchanged shallow check - Non-zero exit code on FAIL (lets scripts use doctor --check in CI) Tests: 17 new tests in tests/test_doctor.py (all pass). Full suite: 621 passed, 6 deselected.
Moves the LLM provider config + summarize_text() function from api.py
to a new huginn.llm module. api.py now re-exports for back-compat with
existing callers (and the test that imports _summarize_text from
huginn.api).
Why:
- llm.py can be tested in isolation without importing the whole API
- Future LLM-touching code (extractor, doctor, etc.) can import from
the canonical location
- api.py is now ~60 lines shorter and has 4 fewer stdlib imports
Public API (huginn.llm):
- LLM_PROVIDER_CONFIG: dict mapping 5 providers (openai, xai, ollama,
anthropic, google) to base_url, key_env, default_model
- summarize_text(text, llm_provider='openai', llm_model=None) ->
Optional[str] — best-effort 1-2 sentence summary, returns None on
no-key / network error
Tests: 11 new tests in tests/test_llm_module.py (all pass).
Full suite: 632 passed, 6 deselected.
The change tracker was using raw SHA-256 over the un-normalized markdown, which produced false-positive 'content changed' alerts for purely cosmetic changes: - URL rotations (analytics-id swap, ad-tracker rewrite) - Timestamp updates (page-load time, 'last updated at') - Case differences (auto-generated nav items) - Whitespace changes (markdownify reflows) The existing huginn.watcher module already has the right algorithm — it normalizes text (lowercase, strip URLs/dates/times, collapse whitespace) before hashing. Now ChangeTracker.compute_hash() delegates to it so: - /v1/scrape?changeTracking=true and /v1/watch use the same logic - Cosmetic changes no longer trigger spurious 'changed' alerts - Real content changes still get caught (sanity test confirms) Hash format: 16-char hex prefix (unchanged from before — same collision characteristics for the URL volume any single Huginn instance will see). Tests: 9 new tests in tests/test_change_tracker_normalization.py (all pass). Full suite: 641 passed, 6 deselected.
The workflow hardcoded 'pytest tests/' which bypassed pyproject.toml's testpaths config and skipped huginn/tests/ entirely (32 tests for crawler, mapper_graph, watcher — the existing layer-1 inner tests). Fix: use bare 'pytest' (no path arg) so it picks up both tests/ and huginn/tests/ via the pyproject.toml testpaths list. Also: removed 3 of the 4 --ignore flags (test_agent_e2e.py, test_integration_browser.py, test_integration_agent.py) — these files no longer exist, so the ignores were no-ops. test_integration.py stays ignored because it needs a live network + Playwright chromium. Before: CI ran 609 tests. After: CI runs 641 tests. Coverage now includes huginn/watcher.py, huginn/crawler.py, huginn/mapper.py (was silently excluded from coverage reports).
The eager 'import chromadb' at module top of memory.py was getting
triggered by huginn/__init__.py's eager import of ResearchMemory.
Chromadb spawns a background connection thread on import that pytest
flagged as a thread leak when the test process exited before cleanup
(PytestUnhandledThreadExceptionWarning).
Fix: lazy import — only call 'import chromadb' inside _try_import_chromadb()
when something actually needs an embedding function. Module-level
sentinel _CHROMA_STATE tracks the result (None=untried, True=imported,
False=failed) so we don't re-attempt on every call.
Behavior:
- huginn/__init__.py importing ResearchMemory no longer triggers
chromadb (saves ~200ms import time + avoids the thread)
- ResearchMemory._ensure_initialized() calls _try_import_chromadb()
before checking the flag, so production use still works
- Tests that never use embeddings stay clean
The remaining 1 warning (StarletteDeprecationWarning: httpx2) is
third-party — FastAPI's testclient needs an upstream update.
Added 2 new CLI tests for the doctor --check flag:
- test_doctor_check_runs_deep_checks: verifies the new flag renders
the deep-check UI with all 4 components (change tracking, webhook
HMAC, LLM credentials, API server) and the Summary line.
- test_doctor_check_exit_code_nonzero_on_failure: sabotages a check
to force FAIL, verifies exit code 1 (CI-friendly).
To make monkeypatching work, refactored ALL_CHECKS → ALL_CHECK_NAMES
in huginn/doctor.py: the orchestrator now looks up check_* by name via
globals() at call time, so tests can override individual check functions
and have their overrides take effect. Previously the list captured
direct function refs at import time, so monkeypatch was a no-op.
Tests: 2 new CLI tests in tests/test_cli.py (all pass). Full suite:
643 passed, 6 deselected, 1 warning.
…ains Bug: get_stats() checked 'domain in self._buckets' (the per-domain runtime state) but _buckets is only populated when acquire() is called for the first time. So a user who called set_domain_config() but never acquire() would see get_stats() return None, indistinguishable from 'never configured'. Same problem in get_all_stats() — it iterated self._buckets so configured-but-idle domains never showed up in monitoring. Fix: get_stats() now checks self._domain_configs (not _buckets) and lazily creates the bucket via _get_bucket() if needed. get_all_stats() iterates self._domain_configs. Now configured-but-idle domains report their config with zero stats instead of disappearing. Also adds 18 new tests in tests/test_domain_rate_limiter.py covering: - DomainLimitConfig tokens_per_second calculation - acquire: burst behavior, independent buckets per domain, refill timing - get_stats: returns None only for unconfigured domains (the fix) - get_all_stats: lists all configured domains - reset_domain: restores full bucket - RateLimitContext: async context manager behavior - get_domain_rate_limiter: singleton identity Full suite: 661 passed, 6 deselected, 1 warning (third-party Starlette deprecation).
Three new tests use httpx.MockTransport to verify the actual call
paths in summarize_text() without making real network requests:
- test_summarize_text_openai_actual_call_path: sets OPENAI_API_KEY,
patches AsyncClient to use a MockTransport, verifies the
Authorization header + gpt-4o-mini model in the request body,
returns the canned response.
- test_summarize_text_anthropic_actual_call_path: sets
ANTHROPIC_API_KEY + HUGINN_LLM_PROVIDER=anthropic, verifies the
x-api-key + anthropic-version headers, verifies the content[].text
response parsing path (different from OpenAI's choices[] format).
- test_summarize_text_raises_keeps_silent: makes the AsyncClient
raise RuntimeError, verifies summarize_text returns None (never
propagates) — the best-effort behavior is critical for /v1/scrape
stability when the LLM API is down.
Coverage on huginn/llm.py: 69% → 100% (11/11 lines now hit).
Full suite: 664 passed, 6 deselected.
The previous version of invalidate_cache(url=X) was clearing the ENTIRE cache, not just X's entries. The code comment even admitted it: 'We can't know all format combinations, so we clear the whole cache when invalidation is requested. In production, use a per-key approach.' This was a real bug — invalidating one stale entry wiped every cached page on the server. A user trying to refresh a single URL would lose all their other cached results. Fix: maintain a _url_to_keys reverse index (URL → set of cache keys) populated by cache_scrape_result, queried by invalidate_cache. When a URL is specified, only that URL's keys are removed from the cache. The index is also cleared when invalidate_cache() is called with no URL. Tests: 7 new tests in tests/test_cache_module_functions.py (all pass): - get_response_cache is async + returns singleton instance - cache_scrape_result → get_cached_scrape_result round-trip - different formats for same URL → different cache entries - cache miss returns None - invalidate_cache(url=X) removes only X's entries (this is the fix) - invalidate_cache() with no URL clears everything Coverage on huginn/cache.py: 74% → 97% (only 3 lines remaining, all edge cases in get_sync + the get_response_cache module-level import). Full suite: 671 passed, 6 deselected, 1 warning (third-party Starlette deprecation).
Null-Phnix
marked this pull request as draft
June 15, 2026 20:20
…/ recover
Adds per-event adaptive throttling to DomainRateLimiter. The static
limiter is now also reactive: when the upstream server returns 429/5xx
or times out, the rate is automatically reduced; when things go well,
the rate gradually recovers.
New API:
- record_outcome(domain, success, status_code=None) — caller reports
a request outcome. 4xx/5xx and timeouts are caller-classified as
failure (success=False).
- get_current_rate(domain) — returns the current rate multiplier
(1.0 = full, 0.1 = heavily backed off)
- get_outcome_history(domain) — recent outcome log (capped at 100)
- DomainLimitConfig.with_rate_multiplier(m) — new helper to build a
scaled config (burst_size clamped to >= 1)
Algorithm: per-event adjustment (not windowed)
- failure → multiplier *= 0.7, floor 0.1
- success → multiplier += 0.05, ceiling 1.0
Why per-event: simpler, more predictable, no 'stuck in backoff' state
from old failures lingering in a sliding window. 3 consecutive
failures → 0.7³ ≈ 0.34 (clearly backed off). 14 consecutive successes
after that → 0.34 + 14×0.05 = 1.04, capped at 1.0 (fully recovered).
Bonus bug fix: when window_seconds=0, the sliding window fallback was
incorrectly allowing requests (timestamp cleaning would empty the list,
then 0 < rpm is True). Now window_seconds=0 means 'no window fallback'
— bucket is the only constraint, which is what the user opted in to.
Tests: 9 new tests in tests/test_adaptive_throttling.py (all pass).
Full suite: 680 passed, 6 deselected.
… per URL
When a user scrapes a URL with include_tags=['.article'] and gets good
results, SelectorMemory records that pattern. Next time they scrape
the same URL, get_suggestions() returns the selectors that worked,
sorted by score (success count × recency decay).
Public API (huginn.selector_memory.SelectorMemory):
- record_success(url, selector) — store a successful selector use
- get_suggestions(url, limit=5) — best selectors for a URL, sorted
- get_stats(url) — Counter of {selector: success_count} for a URL
- get_all_stats() — Counter for all remembered URLs
- get_last_used(url, selector) — timestamp of last success
- forget(url) / forget(url, selector) — cleanup
- clear() — wipe all memory
Algorithm:
- Score = count * recency_factor. recency is 1.0 (last hour) /
0.5 (last day) / 0.1 (older). Sort by score desc, then by name.
- Bounded: max_selectors_per_url=20 (LRU-evict by lowest score),
max_urls=1000 (LRU-evict by last-used).
- Empty/whitespace selectors are ignored.
Why per-URL, in-memory: this is a heuristic, not authoritative.
Stays out of the way of cache invalidation. Persistence would be
a layer 3 enhancement once we know which selectors actually get
reused in practice.
Tests: 16 new tests in tests/test_selector_memory.py (all pass).
Full suite: 696 passed, 6 deselected.
…now canonical
The 'two-SDK smell' is resolved. sdk/python/huginn_client/ is the
single source of truth for HuginnClient, HuginnSync, and the error
hierarchy. huginn/sdk.py (the internal duplicate) is deleted.
Changes:
- sdk/python/huginn_client/__init__.py:
- HuginnError: added error_code= param (matches internal SDK shape)
- New: CircuitOpenError, RateLimitError (backported from internal)
- New: HuginnSync class (sync wrapper, ~100 lines)
- HuginnClient: added public .timeout attribute + async close() method
- HuginnClient.__aexit__: now calls self.close() (single source of truth)
- HuginnSync._run: enters/exits the async context automatically
(caller no longer needs to async with the client first)
- HuginnSync.scrape: now an alias for probe() (matches internal SDK
naming, back-compat for downstream users)
- Bumped __version__ 1.0.0 → 1.2.0 (mark internal SDK removal)
- huginn/__init__.py:
- sys.path manipulation: adds sdk/python/ in dev mode (no pip
install needed)
- from huginn_client import HuginnClient, HuginnError, HuginnSync,
CircuitOpenError, RateLimitError (no more fallback to internal)
- Bumped Huginn __version__ 1.2.0 → 1.3.0
- tests/test_sdk.py:
- Rewrote to test the public SDK API (was testing internal)
- Added 17 new tests:
- TestCircuitAndRateLimitErrors (7 tests): message preservation,
status_code/error_code/response attachment, raise+catch
- TestHuginnSyncMethods (10 tests): health() / scrape() /
probe() with httpx.MockTransport (no real server needed),
close-without-context safety, sync method existence
- Tests now use sys.path manipulation to find the published SDK
in the source tree
Impact for users:
- pip install huginn-client users: no change — they already had the
public SDK. New classes (CircuitOpenError, RateLimitError,
HuginnSync) are now available.
- Huginn self-hosters: no change. The internal SDK was never
public-facing, and Huginn's own code now imports from the
public package which has more features.
- One canonical SDK, no drift risk. Drift prevention achieved.
Tests: 723 passed, 6 deselected, 1 warning (StarletteDeprecation).
Up from 696 (pre-SDK refactor).
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Huginn v1.3 — Firecrawl parity + bug fixes + SDK refactor (WIP, do not merge)
Status: record-of-work PR. Held for review. Will not be merged until
v1.3 is determined to be better than what's currently on
main.What this PR is
32 commits ahead of
mainat the time of last push. Three layers:huginn/sdk.py, promote thepublished
huginn-clientto canonical. Eliminates the two-SDK smell.Stats
huginn/change_tracker.py,huginn/doctor.py,huginn/llm.py,huginn/selector_memory.pyhuginn/sdk.py, the internal duplicate SDK)Firecrawl parity features (11/11 + 1 bonus)
See the prior PR description. All 11 done + 1 bonus (
skipTlsVerification).Bug fixes (6 — all in pre-existing code)
fix(memory):chromadb thread leak (6498f27)fix(change-tracking):delegate hashing to watcher (c6caaa7)fix(rate-limiter):get_stats for configured-but-idle (0ef946c)fix(cache):invalidate_cache per-URL (f75e952)ci:workflow was hardcodedpytest tests/(7657a3f)fix(sdk):default port 8000 vs canonical 7432 (41e426c)Layer 3 reliability primitives
feat(rate-limiter):adaptive throttling — observe 429/5xx outcomes,back off per-event (×0.7 per failure, +0.05 per success)
feat(memory):SelectorMemory — remembers successful CSS selectorsper URL, scored by count × recency
SDK refactor (the big one)
sdk/python/huginn_client/is now the single SDK.huginn/sdk.py(the internal duplicate) is deleted.pip install huginn-clientusers get 3 new classes:CircuitOpenError,RateLimitError,HuginnSync.HuginnClientgot a public.timeoutattribute and a properasync close()method.HuginnSync._runnow auto-enters the async context — callers nolonger need
async with sync:.How to test