fix(extract): return empty + selector_no_match when a selector matches nothing#292
Merged
Merged
Conversation
…s nothing includeTags (MCP/REST) and --css/--xpath (CLI) previously fell back to the entire page when the selector matched no elements, silently dumping the full document into the output — a footgun for LLM agents whose context fills up (observed in practice). Now an unmatched user-supplied selector yields empty content and pushes a 'selector_no_match' warning into metadata.warnings. Domain-configured default selectors are excluded (a host default that does not apply should still show the page). clean_html's public signature is unchanged; a new clean_html_with_warnings surfaces the warning and is used by the extract pipeline. Changes: clean_html_with_warnings/clean_html_impl split (non-breaking); keep_only_selectors returns empty + warning instead of whole page; extract() css/xpath no-match (user_selected) -> Some(empty) + warning; tests add a no-match case and update the invalid-selector test.
…nclude_tags An unmatched includeTags selector emptied the cleaned HTML, but the pipeline then re-injected the page title and meta description and re-ran the alternates ladder, so callers still received page content plus a duplicated selector_no_match warning. Treat an unmatched include_tags as an intentional empty narrowing: it now takes precedence over any trailing css/xpath selector (an xpath scalar like count(//x) no longer leaks "0"), skips title and description enrichment, and warns exactly once. Adds full-pipeline regression tests for the include_tags, css, both, and xpath-scalar no-match cases.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fixes #291.
When a user-supplied content selector matches no elements, crw now returns empty content and adds
selector_no_matchtometadata.warnings, instead of silently returning the entire page.Applies to all three selector inputs:
includeTags(MCPcrw_scrape/ REST/v1/scrape)--css(CLI)--xpath(CLI)Why
An unmatched selector falling back to the whole page is a footgun for the LLM-agent use case: the output looks normal so it gets trusted, but the full document is dumped into the model's context. I hit this directly when it exhausted an agent's context window mid-session. Repro and root cause in #291.
Changes
crates/crw-extract/src/clean.rskeep_only_selectors: on no match, pushselector_no_matchand return an empty string instead of the wholehtml.clean_htmlkeeps its 4-arg signature (discards warnings) viaclean_html_impl; newclean_html_with_warningssurfaces the warning.crates/crw-extract/src/lib.rsextract()usesclean_html_with_warningsat the two production call sites that carryinclude_tags.user_selectedandapply_selectorreturnsNone, pushselector_no_matchand use empty content instead of falling back to the whole cleaned page. Domain-configured default selectors are excluded (a host default that doesn't apply still shows the page).Behavior change to flag
include_tagswith an invalid selector (parse error) previously fell back to the whole page too; it now yields empty +selector_no_match, consistent with the valid-but-no-match case. The existingclean_html_invalid_css_selectortest was updated to reflect this.Testing
cargo test -p crw-extract→ 173 lib + 17clean_tests+ 11extract_tests, all green. Addedinclude_tags_no_match_returns_empty_and_warns; updatedclean_html_invalid_css_selector.clean_html's public signature is unchanged, so other crates (e.g.crw-serversecurity tests that callclean_html) compile without modification.