Switch pipelining metadata action to hollow rlib (-Zno-codegen) by walter-zeromatter · Pull Request #3870 · bazelbuild/rules_rust

walter-zeromatter · 2026-02-26T02:34:46Z

Replace the --rustc-quit-on-rmeta / .rmeta approach with Buck2-style hollow rlibs: the RustcMetadata action runs rustc to completion with -Zno-codegen, emitting a .rlib archive. This approach mirrors the one used by buck2 and avoids needing to kill rustc mid-output in order to produce metadata.

While not fixing problems with SVH mismatches when non-determinism, this does simplify the codepath and uses a production tested technique that doesn't have any of the dangers associated with killing the rustc process while it's still active.

Previous work on this branch & the claimed prevention of SVH mismatches proved to be from changes which effectively removed all the actual benefits of the pipelined compilation

google-cla · 2026-02-26T02:34:56Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

scentini · 2026-04-01T14:27:02Z

Hey @walter-zeromatter, we'll need a signed CLA to proceed with the review, could you let us know if that works for you?
I looked at the PR a bit and concluded that it's likely taken from @dzbarsky's work; @dzbarsky is it okay from your side that we review/merge this PR?

dzbarsky · 2026-04-01T14:35:08Z

It's Walter's work. We just merged it to my fork a while ago so the world can benefit from it via rules_rs.

scentini · 2026-04-01T16:36:03Z

Thanks for the clarification! In that case we only need to wait for the CLA.

walter-zeromatter · 2026-04-03T22:06:11Z

Sorry, I've been waiting on our legal. Just got approval, should have the cla signed later today

walter-zeromatter · 2026-04-07T18:16:58Z

@scentini Sorry about the delay, should be all good now.

walter-zeromatter · 2026-04-09T20:14:07Z

Apologies - This code was written some time ago while I was doing some early experimenting with LLMs, and while I do think replacing the -emit=metadata,link + kill rustc approach with -Zno-codegen, A number of the claimed benefits here simply aren't real - I've since learned better, and will be overhauling this PR to something more appropriate. Please hold.

walter-zeromatter · 2026-04-10T16:55:56Z

@scentini This is now a proper fix.

Sorry for prior confusion. I do think this is the right approach, and I'll be making a few follow-up PRs to adopt a few other improvements from buck2's rust rules including lightweight metadata for check passes & hopefully some kind of lint check people can run to identify if they have non-deterministic proc macro dependencies.

scentini · 2026-04-15T11:51:30Z

        out_dir = out_dir,
        build_env_files = build_env_files,
        build_flags_files = build_flags_files,
-        emit = ["dep-info", "metadata"],


Why are we removing dep-info?

Mostly because it's not necessary - Nothing in rules_rust uses the outputs

scentini · 2026-04-15T11:54:01Z

        disable_pipelining = getattr(ctx.attr, "disable_pipelining", False),
    ):
        rust_metadata = ctx.actions.declare_file(
-            paths.replace_extension(rust_lib_name, ".rmeta"),


If the .rlib and _meta.rlib files are in the same directory, how will the compiler know which one to pick when it encounters a -Ldependency=blaze-bin/rlib/containing/directory/? I think this won't necessarily be an issue with sandboxed and remote execution, which may only have the _meta.rlib file present, but in local non-sandboxed builds I can see how this can pose a problem.

Yeah, you're right. I ran into this while playing with an experimental mode to allow incremental builds with local execution & I've pushed a fix.

scentini · 2026-04-15T11:57:30Z

+        # RUSTC_BOOTSTRAP=1 is required for -Zno-codegen on stable rustc, and must
+        # be set on both the metadata and full actions for SVH compatibility (since
+        # RUSTC_BOOTSTRAP affects the crate hash).
+        env["RUSTC_BOOTSTRAP"] = "1"


This change makes pipelining work only for users that are fine with relying on unstable features. I think that's fine, however I think we should go through the incompatible change process, where we add an --incompatible_use_unstable_no_codegen_for_pipelining build_setting, and gate the changes in this PR behind it.

scentini · 2026-04-15T11:58:52Z

        experimental_use_cc_common_link = experimental_use_cc_common_link,
    )

+    compile_inputs_metadata = compile_inputs


Seems like compile_inputs_metadata and compile_inputs are now the same thing. If so, could we get rid of this variable?

Good catch, ty

walter-zeromatter · 2026-04-19T17:39:06Z

I've updated to use a gating incompatible flag, and also added machinery so that this doesn't automatically enable all unstable features for downstream users - if you pass RUSTC_BOOTSTRAP or manually specify -Zallow-features, that behavior should still work

Introduce `incompatible_use_unstable_no_codegen_for_pipelining` to switch pipelined compilation from the process-wrapper `--rustc-quit-on-rmeta` approach to Buck2-style `-Zno-codegen` hollow rlibs. Defaults to off; the existing rmeta-interception path remains the default behavior. When enabled: - Metadata action runs `rustc -Zno-codegen --emit=link=<path>` to produce a hollow rlib (`_meta.rlib` in a `_meta/` subdir of the full rlib). - Full action emits only `--emit=link` (no metadata, no dep-info). - `RUSTC_BOOTSTRAP=1` is set on both actions so SVH matches. - Metadata-consuming actions get an extra `-Ldependency` pointing at the `_meta/` subdir for transitive resolution. Tracked in bazelbuild#3977.

scentini self-requested a review April 1, 2026 14:27

scentini added the awaiting-response Maintainers have responded to the pull-request or thread and now await contributor responses. label Apr 1, 2026

scentini mentioned this pull request Apr 1, 2026

Support pipelining via -Zno-codegen #3943

Open

walter-zeromatter force-pushed the user/wgray/pipelining branch from 349e91c to 841c962 Compare April 8, 2026 01:27

scentini reviewed Apr 8, 2026

View reviewed changes

Comment thread rust/private/rustc.bzl Outdated

Comment thread rust/private/rustc.bzl Outdated

walter-zeromatter force-pushed the user/wgray/pipelining branch from 841c962 to b55be1b Compare April 10, 2026 16:50

walter-zeromatter force-pushed the user/wgray/pipelining branch 3 times, most recently from e470d22 to 63ca376 Compare April 10, 2026 19:50

scentini reviewed Apr 15, 2026

View reviewed changes

walter-zeromatter force-pushed the user/wgray/pipelining branch from 63ca376 to ee6c5e6 Compare April 17, 2026 15:35

walter-zeromatter mentioned this pull request Apr 17, 2026

incompatible_use_unstable_no_codegen_for_pipelining #3977

Open

walter-zeromatter force-pushed the user/wgray/pipelining branch from ee6c5e6 to e78e7de Compare April 19, 2026 17:55

walter-zeromatter mentioned this pull request Apr 19, 2026

Add RUSTC_BOOTSTRAP guardrail for -Zno-codegen pipelining hermeticbuild/rules_rust#19

Open

walter-zeromatter force-pushed the user/wgray/pipelining branch from e78e7de to 67f7aaa Compare April 20, 2026 23:50

Conversation

walter-zeromatter commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

google-cla bot commented Feb 26, 2026

Uh oh!

scentini commented Apr 1, 2026

Uh oh!

dzbarsky commented Apr 1, 2026

Uh oh!

scentini commented Apr 1, 2026

Uh oh!

walter-zeromatter commented Apr 3, 2026

Uh oh!

walter-zeromatter commented Apr 7, 2026

Uh oh!

Uh oh!

Uh oh!

walter-zeromatter commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

walter-zeromatter commented Apr 10, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

walter-zeromatter commented Apr 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

walter-zeromatter commented Feb 26, 2026 •

edited

Loading

walter-zeromatter commented Apr 9, 2026 •

edited

Loading