WIP: Huginn v1.3 — PR 1/3: Parity (branding + 11 features + early bug fixes) - #6
Merged
Conversation
The ErrorCode enum values are part of the public API (callers parse them in their error handlers). The new TestErrorCode class pins the contract: - UNAUTHORIZED is exactly the lowercase string 'unauthorized' - Every value matches a lowercase-or-underscore identifier pattern - from_http_status(401) returns the right member Also adds .nuwa/plans/2026-06-15_huginn-improvement-pass.md documenting the rest of this Nüwa improvement pass: real bugs found, false positives that were re-verified out, and the small-commit plan.
The package is 1.2.0 but huginn/api.py had three hardcoded '1.1.0' strings: the FastAPI app's version kwarg, /health, and /health/detailed. All three are now sourced from huginn.__version__ so a future bump takes effect automatically. Adds two tests in TestApiVersionConsistency that pin the contract: the FastAPI app's .version and /health's version field both equal huginn.__version__.
huginn/sdk.py was the only place shipping :8000 as the default Huginn server URL. Every other surface — cli.py, config.py, docker-compose, huginn_cli.py, sdk/python/ — uses :7432. So HuginnClient() and HuginnSync() out of the box were trying to talk to nothing. Fixed 4 sites in huginn/sdk.py (HuginnClient default, HuginnSync default, the docstring on each, the top-of-file usage example), and updated the two test_sdk.py assertions that had been locking the wrong port in place.
The actual server default is 7432 (see huginn/config.py ServerConfig.port and docker-compose.yml). The README's quick-start curl examples, the 'huginn serve --port' example, the HUGINN_PORT=… env example, and the docs/UPGRADE_PLAN.md docker-run example all still said 8000 — fixed to 7432. No code or test changes needed for this commit.
…to 348 pyproject.toml [tool.pytest.ini_options].testpaths was just ["tests"], which silently excluded huginn/tests/ (32 tests). Running 'pytest' with no arguments therefore reported 311 passing while the README badge claimed 343 — a quiet lie that hid the inner test directory from anyone running the default command. Fix: testpaths = ["tests", "huginn/tests"]. Now 'pytest' collects all 348 tests (343 baseline + 5 new TestErrorCode + 2 new TestApiVersionConsistency added in earlier commits this pass) and the 6 deselected network tests. README badge updated to match. If a future test directory is added, the testpaths list will need updating; the TestApiVersionConsistency tests already cover the 'version is consistent' half of the contract.
BrowserConfig.user_agent was hardcoded to 'Huginn/1.1 (+...)' while
__version__ is '1.2.0'. The default UA now derives from __version__:
user_agent: Optional[str] = field(default_factory=_default_user_agent)
where _default_user_agent() returns f'Huginn/{__version__} (+...)'.
Two new tests in TestUserAgentDefaults (test_config.py) pin the
contract: the default must contain the current __version__ and must
not contain any stale hardcoded version (1.0.0, 1.1.0, 1.1).
Also bumps .env.example:
HUGINN_USER_AGENT=Huginn/1.1 → Huginn/1.2
comment default: Huginn/1.1 → default tracks __version__
Also adds .nuwa/specs/2026-06-15_huginn-personal-firecrawl-replacement.md:
the strategic spec for making Huginn a 1:1 personal Firecrawl
replacement. Three layers (branding, endpoint parity, reliability),
what to ship first, what not to do. Re-read before any session work
on Huginn.
…from it
The 'Huginn' / 'HUGINN_' / 7432 / UA-template / banner brand surface was
scattered across huginn/__init__.py, cli.py, api.py, and config.py.
This is layer 1 of the personal-Firecrawl-replacement spec
(.nuwa/specs/2026-06-15_huginn-personal-firecrawl-replacement.md):
centralize so a future rebrand is one file.
What huginn/_branding.py owns:
- name / short_name / binary / package / python_module
- ua_template + ua(version) helper (default reads __version__)
- color, tagline, description, ascii_banner
- env_prefix (every HUGINN_* env var is f-built from this)
- project_url / docs_url / repo_url
What was refactored:
- config._default_user_agent() → _branding.ua()
- config._apply_env() env_map → f-strings with _branding.env_prefix
(so a prefix rebrand like HUGINN_ → RAVEN_ is a one-line edit)
- cli._banner() → uses _branding.name
- cli @click.group → help= f-string built from _branding
- cli @click.version_option → prog_name=_branding.name
- api FastAPI app title + description → _branding.name + _branding.description
New tests (19):
- tests/test_branding.py — 18 tests pinning the contract of every
branding constant + helper. If a rebrand happens, the tests prove
the surface still works.
- tests/test_cli.py — 2 new tests: help_uses_branding_name and
version_uses_branding_name. The pre-existing test_main_help was
loosened from 'Huginn' in output to '_branding.name in output'.
Test count: 350 → 369 (19 new). 6 network still deselected. ~7s.
…ert verification
Adds the skipTlsVerification (snake_case: skip_tls_verification) field
to ScrapeRequest with alias, defaulting to True for backward-compat with
Huginn's historical hardcoded ignore_https_errors=True behaviour.
Wire:
ScrapeRequest.skip_tls_verification
→ Scraper.scrape(skip_tls_verification)
→ Scraper._scrape_impl(skip_tls_verification)
→ BrowserManager.ignore_https_errors (instance attr, set before new_context)
→ Playwright new_context(ignore_https_errors=…)
The instance-attribute approach avoids threading the param through the
circuit-breaker cb.call() wrapper while keeping existing test mocks intact
(bare AsyncMock has no ignore_https_errors attr → getattr falls back to True).
Tests: 6 new tests (TestScrapeRequestTlsField × 4, TestBrowserContextTlsFlag × 2)
all passing. Full suite: 375 passed, 6 deselected.
Adds 114 tests for includeTags/excludeTags CSS-selector filtering: - Model accepts any CSS selector as include_tags or exclude_tags (17 standard tags + 8 extended selectors) - Scraper._filter_tags() calls page.evaluate with querySelectorAll for each tag - Verifies: script, style, img, a, svg, iframe, div, section, article, aside, footer, header, nav, main, table, ul, ol + extended (.ads, #id, [attr], picture, video, audio, canvas, pre, code, blockquote) - No evaluate calls when neither include nor exclude tags are specified - All 375 original tests still pass.
Language is extracted in both full browser and lightweight paths: - Full browser: from browser.extract_content() which reads document.documentElement.lang - Lightweight: from <html lang='...'> via BeautifulSoup - Default: 'en' when no lang attribute is present - metadata.language key always present in ScrapeData.metadata dict Tests: 4 new tests (all 4 pass). Full suite: 379 passed, 6 deselected.
…mary - ScrapeRequest.summary: bool = False (firecrawl parity) - ScrapeResponse.summary: Optional[str] = None - New _summarize_text() helper in api.py supports OpenAI, xAI, Ollama, Anthropic, Google; best-effort (returns None on no key / network error) - Scrape endpoint calls _summarize_text on data.markdown when req.summary=True - Module-level helper so it can be unit-tested in isolation Tests: 7 new tests in tests/test_summary_field.py (all pass). Full suite: 500 passed, 6 deselected.
- ScrapeRequest.mobile: bool = False - BrowserManager.new_context(device=...) merges device fields into context_kwargs (viewport, user_agent, is_mobile, has_touch, etc.) - Scraper.scrape(mobile=False) passes _MOBILE_DEVICE (iPhone 13 descriptor) to new_context() when mobile=True - Module-level _MOBILE_DEVICE constant — matches Playwright's official iPhone 13 device descriptor Tests: 10 new tests in tests/test_mobile_emulation.py (all pass). Full suite: 510 passed, 6 deselected.
- ScrapeRequest.block_ads: bool (camelCase: blockAds) — Playwright route interception for 24 well-known ad networks (doubleclick.net, googlesyndication.com, googleadservices.com, googletagmanager.com, amazon-adsystem, criteo, outbrain, taboola, scorecardresearch, adsrvr, adnxs, rubiconproject, pubmatic, openx, moatads, facebook, twitter, yahoo, linkedin, adform, etc.) - ScrapeRequest.remove_base64_images: bool (camelCase: removeBase64Images) — strips data:image/...;base64,... URIs from extracted markdown via _BASE64_IMAGE_REGEX - _AD_DOMAINS and _BASE64_IMAGE_REGEX module-level constants Tests: 12 new tests in tests/test_block_ads_and_base64.py (all pass). Full suite: 522 passed, 6 deselected.
includeTags=['iframe'] already works via the existing _filter_tags implementation. This commit adds 5 tests verifying: - iframe elements preserved in result.html with src URL captured - exclude_tags=['iframe'] strips iframe elements - Multiple iframes on a page all extracted - 'iframe' is passed to querySelectorAll correctly - iframe + video combined tags work Tests: 5 new tests in tests/test_iframe_extraction.py (all pass). Full suite: 527 passed, 6 deselected.
- CrawlRequest.max_concurrency: Optional[int] (1-100, alias: maxConcurrency) - CrawlRequest now uses model_config = ConfigDict(populate_by_name=True) so both max_concurrency and maxConcurrency are accepted - All 3 Crawler() instantiation sites in api.py now use concurrency=req.max_concurrency or _config.crawl.concurrency, so per-job override falls back to global default cleanly Tests: 12 new tests in tests/test_crawl_max_concurrency.py (all pass). Full suite: 539 passed, 6 deselected.
- FlockRequest.ignore_invalid_urls: bool (alias: ignoreInvalidURLs) - FlockRequest now uses model_config = ConfigDict(populate_by_name=True) - New _is_valid_http_url() helper in scraper.py — validates scheme in (http, https) + non-empty netloc - flock_scrape endpoint skips invalid URLs with a warning when flag is True; raises 422 when False (default) Tests: 18 new tests in tests/test_ignore_invalid_urls.py (all pass). Full suite: 557 passed, 6 deselected.
- MapRequest.sitemap: Optional[str] (default 'include')
- MapRequest now uses model_config = ConfigDict(populate_by_name=True)
- Mapper.map_site(sitemap='include|skip|only') routes to the right strategies:
'include' (default) — fetch sitemap + crawl page + check sub-sitemaps
'skip' — page crawl only, no sitemap fetch at all
'only' — sitemap URLs only, no page crawl
- Invalid sitemap values fall back to 'include'
- /v1/chart endpoint passes req.sitemap or 'include' as defensive default
Tests: 14 new tests in tests/test_sitemap_mode.py (all pass).
Full suite: 571 passed, 6 deselected.
- ScrapeRequest.change_tracking: bool (alias: changeTracking)
- ScrapeData.change_tracking: Optional[Dict[str, Any]] containing
{previous_hash, current_hash, diff, changed} when populated
- New ChangeTracker module (huginn/change_tracker.py):
- compute_hash(content) — SHA-256 hex prefix (16 chars)
- compute_diff(old, new) — unified diff (empty if identical)
- check_and_store(url, content) — async-safe, returns the dict
- Module-level singleton via get_change_tracker()
- Scraper._scrape_impl() populates result.change_tracking after
markdown extraction when change_tracking=True
- Scraper._change_tracker is lazily initialized on first use
Tests: 22 new tests in tests/test_change_tracking.py (all pass).
Full suite: 593 passed, 6 deselected.
- New _compute_signature(body, secret) helper — HMAC-SHA256 hex digest
- send_webhook() now adds X-Huginn-Signature: sha256=<hex> header
when a secret is set, signed over the JSON-serialized body
- Legacy X-Huginn-Webhook-Secret header still sent for backward compat
- Body is serialized once (with compact separators) so the signature
is computed over the exact bytes that go on the wire
- User-Agent header set to 'Huginn-Webhook/1.0' for receiver-side logs
Receiver-side verification (any language):
body = await request.body() # raw bytes
expected = hmac.new(secret.encode(), body, sha256).hexdigest()
assert request.headers['X-Huginn-Signature'] == f'sha256={expected}'
Tests: 11 new tests in tests/test_hmac_webhook.py (all pass).
Full suite: 604 passed, 6 deselected.
Null-Phnix
marked this pull request as draft
June 15, 2026 21:06
Null-Phnix
marked this pull request as ready for review
June 17, 2026 19:20
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR 1 of 3: Huginn v1.3 — Parity (branding, 11 Firecrawl features, early bug fixes)
Series: 3 stacked PRs to split v1.3 into reviewable chunks.
Merge order: PR 1 → PR 2 → PR 3. This PR is independent (no deps). PR 2 depends on this. PR 3 depends on PR 2.
Status: draft, do not merge until the v1.3 series is reviewed.
What this PR is
19 commits ahead of
main. The "core v1.3" — everything needed for Huginn to be a real Firecrawl replacement._branding.py— single source of truth for user-facing stringsStats
Firecrawl parity features (11/11 + 1 bonus)
includeTags/excludeTagsverification74d9123metadata.languagedetection06fbf6csummaryfield on scrape responsec4cb83cmobile: truedevice emulationb36c2b3blockAdsflagd8a847dremoveBase64Imagesflagd8a847d61aea9emaxConcurrencyb6bf371ignoreInvalidURLs148baccsitemap: "include|skip|only"dd39e02changeTrackingdiff outpute2b1530747ce5fskipTlsVerification(parity bonus)73e54f0Bug fixes (3)
fix(api):version from__version__instead of hardcoded '1.1.0' (51e6810)fix(sdk):default base_url :7432 (was :8000, the only surface not using 7432) (41e426c)fix(config):User-Agent default tracks__version__viadefault_factory(9fb66d2)How to test