Skip to content

fix(scrapy): dump pydantic models in JSON mode when serializing requests#961

Open
vdusek wants to merge 2 commits into
masterfrom
fix/scrapy-json-pydantic-models
Open

fix(scrapy): dump pydantic models in JSON mode when serializing requests#961
vdusek wants to merge 2 commits into
masterfrom
fix/scrapy-json-pydantic-models

Conversation

@vdusek

@vdusek vdusek commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Pydantic models in a Scrapy request's meta/cb_kwargs were dumped in python mode, so any model with a non-JSON-native field (datetime, UUID, HttpUrl, ...) failed json.dumps and the request was dropped. The v4 upgrade guide promises such models keep working.

  • Dump models with model_dump(mode='json', by_alias=True).
  • Widen the serialization error guards to except (TypeError, ValueError) to cover pydantic's PydanticSerializationError.
  • Add a round-trip test with a datetime-bearing model and update the v4 upgrade guide wording.

A pydantic model in `meta`/`cb_kwargs` with a non-JSON-native field (`datetime`, `UUID`, `HttpUrl`)
failed `json.dumps` and the request was dropped, contradicting the v4 upgrade guide. Also fixes the PyPI
Release Notes URL (pointed at the v2 guide) and stale Python 3.10 claims in the docs.
@vdusek vdusek added adhoc Ad-hoc unplanned task added during the sprint. t-tooling Issues with this label are in the ownership of the tooling team. labels Jun 11, 2026
@vdusek vdusek self-assigned this Jun 11, 2026
@codecov

codecov Bot commented Jun 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.94%. Comparing base (a87e8d1) to head (140fff9).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #961      +/-   ##
==========================================
- Coverage   90.00%   89.94%   -0.07%     
==========================================
  Files          49       49              
  Lines        3053     3053              
==========================================
- Hits         2748     2746       -2     
- Misses        305      307       +2     
Flag Coverage Δ
e2e 35.66% <0.00%> (ø)
integration 56.86% <0.00%> (ø)
unit 78.64% <100.00%> (-0.07%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@vdusek vdusek requested a review from Pijukatel June 11, 2026 12:36
@vdusek vdusek marked this pull request as ready for review June 11, 2026 12:36
@vdusek vdusek requested a review from szaganek as a code owner June 11, 2026 12:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

adhoc Ad-hoc unplanned task added during the sprint. t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants