Skip to content

Give each crawler instance its own default request queue #2132

Closed

Description

@vdusek

BasicCrawler.get_request_manager() (src/crawlee/crawlers/_basic/_basic_crawler.py) opens the same default request queue for every crawler, so two crawlers in one process silently share requests and deduplication state. Only the first instance should use the default queue; the rest should get their own via an internal alias derived from the crawler id.

JS implementation: apify/crawlee#3834

This will be a breaking change.

✍️ Drafted by Claude Code

Activity

  1. added
    enhancementNew feature or request.
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Aug 7, 2026
  2. added this to the 2.0 milestone on Aug 7, 2026
  3. self-assigned this
    on Oct 20, 2026
  4. added a commit that references this issue on Oct 23, 2026
    fe36ab6
  5. vdusek commented on Oct 23, 2026

    @vdusek
    CollaboratorAuthor

    resolved by #2344

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions