Skip to content

fix(sdist): regenerate Cargo.lock when workspace members are removed#3192

Open
ckhordiasma wants to merge 5 commits into
PyO3:mainfrom
ckhordiasma:fix/sdist-cargolock
Open

fix(sdist): regenerate Cargo.lock when workspace members are removed#3192
ckhordiasma wants to merge 5 commits into
PyO3:mainfrom
ckhordiasma:fix/sdist-cargolock

Conversation

@ckhordiasma

@ckhordiasma ckhordiasma commented May 19, 2026

Copy link
Copy Markdown

Summary

Fixes #2609.

When maturin builds an sdist from a workspace, rewrite_cargo_toml strips workspace members
not needed by the package. The Cargo.lock was copied verbatim and still referenced those
removed crates, so cargo build --locked inside the sdist would fail.

After all sdist entries are assembled in the virtual writer, materialize them to a temp
directory, run cargo generate-lockfile to reconcile the lockfile, and replace the tracker
entry with the regenerated Cargo.lock. This only runs for workspace crates that have a
Cargo.lock entry in the sdist.

Changes

  • Add materialize_to and replace_bytes helpers on VirtualWriter<SDistWriter>
  • Add regenerate_cargo_lock step after add_workspace_manifest in the sdist builder
  • Add regression test that builds an sdist from a two-member workspace (where the sibling has
    a unique dependency), unpacks it, and asserts cargo metadata --frozen succeeds

Test plan

  • cargo test --test run -- sdist_workspace_removed_members_cargo_lock passes with the fix
  • Same test fails on main without the fix (verified on a separate branch)
  • cargo fmt and cargo clippy clean

🤖 Generated with Claude Code — ~40 back-and-forth messages covering issue analysis, implementation, line-by-line code walkthrough, test authoring, failure verification, style review, and PR creation.

ckhordiasma and others added 2 commits May 19, 2026 09:43
Fixes PyO3#2609.

When maturin builds an sdist from a workspace, `rewrite_cargo_toml`
strips workspace members not needed by the package. The Cargo.lock was
copied verbatim and still referenced those removed crates, so
`cargo build --locked` inside the sdist would fail.

After all sdist entries are assembled, materialize them to a temp
directory, run `cargo generate-lockfile` to reconcile the lockfile,
and replace the tracker entry with the regenerated Cargo.lock.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Exercises the fix for PyO3#2609: builds an sdist from a workspace member
while a sibling member (`devtool`, with a unique `rand` dependency)
is stripped, then asserts `cargo metadata --frozen` succeeds in the
unpacked sdist.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes sdists produced from Cargo workspaces where rewrite_cargo_toml removes unneeded workspace members but the copied Cargo.lock still references them, causing cargo build --locked / cargo metadata --frozen to fail inside the unpacked sdist.

Changes:

  • Add VirtualWriter<SDistWriter>::materialize_to and replace_bytes helpers to support post-processing tracked sdist entries.
  • After assembling workspace-related sdist entries, materialize the sdist to a temp dir, run cargo generate-lockfile, and replace the sdist’s Cargo.lock with the regenerated version.
  • Add a regression test that constructs a two-member workspace, builds an sdist for one member, unpacks it, and asserts cargo metadata --frozen succeeds.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File Description
tests/run/sdist.rs Adds regression coverage ensuring workspace-member sdists have a usable Cargo.lock.
src/source_distribution/mod.rs Adds regenerate_cargo_lock step in the sdist build pipeline to reconcile lockfiles after workspace-member stripping.
src/module_writer/virtual_writer.rs Adds helpers to materialize tracked sdist entries to disk and replace a tracked file’s bytes in-memory.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/source_distribution/mod.rs Outdated
Comment thread src/module_writer/virtual_writer.rs
Comment thread src/module_writer/virtual_writer.rs
Comment thread src/module_writer/virtual_writer.rs Outdated
ckhordiasma and others added 2 commits May 28, 2026 15:22
- Use `workspace_members.len() > 1` instead of path-inequality heuristic
  so workspace-root packages are also covered.
- Make `replace_bytes` return `Result` and error if the target entry
  does not exist in the tracker.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Set file permissions based on the tracked executable flag after
writing each entry, matching the existing default_permission pattern.
Unix-only via #[cfg(unix)], consistent with the rest of the codebase.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

Comment thread src/module_writer/virtual_writer.rs
Comment thread src/source_distribution/mod.rs
Comment thread tests/run/sdist.rs Outdated
- Preserve original executable flag and source path when replacing
  a tracker entry in replace_bytes.
- Include stdout and working directory in cargo generate-lockfile
  error messages.
- Use output() instead of status() in test for better failure
  diagnostics.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@messense

Copy link
Copy Markdown
Member

Thanks for the PR — the problem is real and reconciling the lockfile against the rewritten manifests is the right general approach. A few things need to change before this can merge though:

1. cargo generate-lockfile re-resolves everything to latest

Per the cargo docs, if the lockfile already exists it "will be rebuilt with the latest available version of every package" — it's effectively a full cargo update. So this doesn't just prune the stale entries: every third-party pin in the sdist's Cargo.lock silently floats to whatever is newest at sdist-build time. That breaks the reproducibility guarantee shipping a lockfile is meant to provide, contradicts a user-supplied --locked, and makes sdists non-reproducible day to day.

Please use cargo update --workspace instead — it conservatively preserves existing pins while re-resolving the graph, which drops the unreachable entries. (Running plain cargo metadata without --frozen for its minimal lockfile sync would also work.)

2. Trigger condition is too coarse

workspace_members.len() > 1 fires even when no member is actually stripped (all siblings are path deps that stay in the sdist). Combined with point 1, that rewrites previously-correct lockfiles. The rewrite code in cargo_toml_rewrite.rs already knows which members were dropped — plumbing that through (or checking whether the lockfile references packages absent from the sdist) would make this a no-op in the common case.

3. Offline / cargo config

This adds a hard network dependency to sdist builds for multi-member workspaces. At minimum, propagate ctx.project.cargo_options.offline as --offline. Also note the temp dir has no .cargo/config.toml, so source replacement / private registries won't be visible to the regeneration run — worth a comment, and possibly degrading to a warning + original lockfile on failure rather than failing the whole build.

Smaller points

  • Pass --manifest-path explicitly instead of relying on current_dir + cargo's upward search (matches cargo_package_file_list in the same file).
  • Missing space in the error message: `{}`(exit status: .
  • Test: please add an assertion that a surviving pinned version is preserved (e.g. cargo update --precise something in the original lockfile, then compare) — that's the test that distinguishes "pruned" from "re-resolved", and it would have caught point 1. Asserting devtool/rand are absent from the regenerated lockfile would also be more direct than relying only on --frozen's exit status.

The VirtualWriter helpers themselves look good — replace_bytes preserving entry metadata and materialize_to reusing default_permission are both right.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sdist generated from workspace contains outdated Cargo.lock

3 participants