Skip to content

feat: recursive media discovery in --extract-scan#16

Merged
xAlcahest merged 7 commits into
masterfrom
feat/recursive-scan
Jun 19, 2026
Merged

feat: recursive media discovery in --extract-scan#16
xAlcahest merged 7 commits into
masterfrom
feat/recursive-scan

Conversation

@xAlcahest

Copy link
Copy Markdown
Contributor

Summary

The scanner in scan.c only looked at the landing page, so sites that load the player one or two clicks away (episode page -> /watch?file= -> the real .mp4) produced an empty skeleton. Now when the landing page has no direct media, scan_page follows internal watch/play/embed links up to --extract-scan-depth hops (default 2, max 3), re-scans each body with the same scoring, and records the capture chain that reached the media so scan_emit_config can write a reusable multi-step get/var config. Series index pages are detected and emit a list plus a per-episode pipeline. The crawl is bounded (visited-set, SCAN_MAX_FETCHES, 8 MiB page cap) and the ad-host blocklist still applies.

Config generation no longer writes into the active dir: run_extract_scan prints the config, stashes a pending copy under .pending/<id>.conf keyed by a stable id derived from the source URL, and suggests flux --save-config <id> to install it active (refusing to clobber). Both the argv id and the config name parsed back out of the pending file are sanitized before they touch a path.

Changes

  • src/scan.h: recursion caps (SCAN_MAX_FOLLOW, SCAN_MAX_CHAIN, SCAN_MAX_DEPTH, SCAN_MAX_FETCHES), a per-candidate capture chain, series fields, and a max_depth argument on scan_page.
  • src/scan.c: link and episode heuristic tables, bounded recursive follow with a visited-set, chain provenance, series detection, and multi-step/series config emission; the direct-media path is unchanged.
  • src/text.c: --extract-scan-depth and --save-config flags, the pending-store and stable-id workflow, path sanitization on save, and depth clamping with a notice.
  • src/test_scan.c: new one-hop, two-hop, and series cases that roundtrip the generated config through extractor_parse/run/list, plus the existing cases adapted to the new scan_page signature.
  • .github/workflows/ci.yml: scan e2e now covers the save-config flow, a path-traversal attempt, and depth validation.
  • README.md, NEWS, ChangeLog: document the recursive scan, the depth flag, and the save-by-id workflow.

Comment thread src/text.c Fixed
Comment thread src/text.c Fixed
@xAlcahest xAlcahest merged commit a826c5f into master Jun 19, 2026
5 checks passed
@xAlcahest xAlcahest deleted the feat/recursive-scan branch June 19, 2026 21:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants