Account for scrapy-poet Zyte API parameters when fingerprinting

[AdrianAtZyte]

Before this change, one of the 2 URLs in the example below would be skipped:

from scrapy import Request, Spider
from scrapy_poet import DummyResponse
from zyte_common_items import JobPostingNavigation

class TestSpider(Spider):
    name = "test"
    custom_settings = {
        "ZYTE_API_LOG_REQUESTS": True,
    }

    def start_requests(self):
        for slug in ("foo", "bar"):
            yield Request(
                f"https://toscrape.com/#/{slug}",
                meta={
                    "zyte_api_provider": {
                        "jobPostingNavigationOptions": {"extractFrom": "browserHtml"},
                    },
                },
            )

    async def parse(self, response: DummyResponse, item: JobPostingNavigation):
        yield item

[codecov]

Codecov Report

❌ Patch coverage is 95.49550% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.43%. Comparing base (8783e37) to head (3a4045d).
⚠️ Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
scrapy_zyte_api/providers.py	92.43%	4 Missing and 5 partials ⚠️
scrapy_zyte_api/_request_fingerprinter.py	99.02%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #281      +/-   ##
==========================================
- Coverage   97.69%   97.43%   -0.26%     
==========================================
  Files          15       15              
  Lines        2172     2304     +132     
  Branches      401      419      +18     
==========================================
+ Hits         2122     2245     +123     
- Misses         21       25       +4     
- Partials       29       34       +5     

Files with missing lines	Coverage Δ
scrapy_zyte_api/_request_fingerprinter.py	98.85% <99.02%> (-0.06%)	⬇️
scrapy_zyte_api/providers.py	93.72% <92.43%> (-1.43%)	⬇️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[wRAR]

My only question is whether this makes costly calculations for requests passed to it that are already made for them somewhere else

[AdrianAtZyte]

I’ve done my best to minimize this now (as of d5b4863).

[Copilot]

Pull request overview

This PR updates request fingerprinting to incorporate scrapy-poet/Zyte API provider parameters so that requests to the same URL with different provider options no longer share the same fingerprint (and therefore won’t be incorrectly deduplicated/skipped).

Changes:

• Extend ScrapyZyteAPIRequestFingerprinter to compute a provider-aware fingerprint (via scrapy-poet’s dependency plan) and combine it with the “regular” Zyte API fingerprint when appropriate.
• Refactor Zyte API provider meta construction into a reusable helper (_build_zyte_api_provider_meta) and add coverage around AnyResponse behavior when HttpResponse is already available.
• Add/adjust tests around the new fingerprint combination logic and provider validation behavior.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
scrapy_zyte_api/_request_fingerprinter.py	Adds provider-aware fingerprinting and combines provider + regular fingerprints, including special handling for provider-only requests.
scrapy_zyte_api/providers.py	Introduces _build_zyte_api_provider_meta helper and reuses it inside ZyteApiProvider.__call__.
tests/test_request_fingerprinter.py	Adds unit tests for provider/regular fingerprint combination and provider-only behavior.
tests/test_providers.py	Imports and tests _build_zyte_api_provider_meta, plus adds a test for unannotated Actions dependencies.
tests/test_sessions_enabled.py	Sets RETRY_TIMES = 0 for test stability/reproducibility.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.