[Ha1baraA11]

Summary

Keeps nested inline text when its content-filter paragraph is retained.
Related to #2417

List of files changed and why

• crawl4ai/content_filter_strategy.py: keep HTML phrasing elements from being scored and removed independently of their containing content block.
• crawl4ai/content_filter_strategy_lxml.py: apply the same inline-tag rule in the LXML pruning path.
• tests/async/test_content_filter_prune.py: add the Issue's nested-link and Japanese-text regression for both pruning implementations.

How Has This Been Tested?

• uv run --frozen --python=3.12 --with=pytest python -m pytest tests/async/test_content_filter_prune.py::TestPruningContentFilter::test_pruning_preserves_nested_inline_content_in_kept_paragraphs -q --disable-warnings — 1 passed.
• python3 -m compileall -q crawl4ai/content_filter_strategy.py crawl4ai/content_filter_strategy_lxml.py tests/async/test_content_filter_prune.py — passed.
• uv run --frozen --python=3.12 --with=pytest python -m pytest tests/async/test_content_filter_prune.py -q --disable-warnings — 1 failed, 13 passed; pre-existing base failure: tests/async/test_content_filter_prune.py::TestPruningContentFilter::test_threshold_types.

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added/updated unit tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[SohamKukreti]

Thanks for the fix! The approach is right: inline elements now follow their block's keep/remove decision, in both the BS4 and LXML engines. I checked that the new tests fail on develop and pass with this change, and that the existing prune and parity tests still pass. (test_threshold_types also fails on develop, so it is not caused by this PR.)

I also ran it on 8 real pages (Wikipedia, HN, GitHub, The Verge, BBC, Python docs, MDN, Reddit). The new text is real content that the filter was cutting out of sentences. Nothing that was kept before is lost. A few changes before we merge:

1. Make the tag list smaller. Some entries do nothing: script, noscript and iframe are removed by excluded_tags before pruning starts. Other entries don't hold sentence text, and they change behavior beyond this issue. For example, <img> inside kept paragraphs was always pruned before and is now always kept. Please limit the list to text-level phrasing tags:
   a abbr b bdi bdo cite code data del dfn em i ins kbd mark q ruby s samp small span strong sub sup time u var
   This still fixes #2417. On the Tokyo page it adds 12% to fit_markdown instead of 18%.
2. Put the set on the class (e.g. self.inline_tags, next to tag_weights), not in a private module constant imported from content_filter_strategy_lxml.py. The LXML subclass then gets it for free, and users can change it.
3. Remove the unrelated re-wrap of the _should_remove signature in content_filter_strategy_lxml.py.
4. Remove the duplicated child loop in _prune_tree: only score and maybe remove the node when it is not inline, and then go into the children once.
5. Merge the two tests into one with @pytest.mark.parametrize over PruningContentFilter and PruningContentFilterLXML.