initial SVE intrinsics by davidtwco · Pull Request #2071 · rust-lang/stdarch

davidtwco · 2026-04-05T12:48:30Z

This is a very large patch but the vast majority of it is generated code, the rest should be relatively straightforward to review.

This passes tests for me locally with the latest nightly after rust-lang/rust#153286 has landed.

I've bolded the commits that do the most interesting things - the rest are largely updating the support that has been partially upstreamed in the past to work with the current implementation in rustc.

3489955 re-adds the "missing run-time test" check in stdarch-verify for Arm - it was accidentally removed in 713444d.
d3455ed removes some special-casing in the generator for files being generated from sve.spec.yml and sve2.spec.yml and instead adds a key that can be set in those files to achieve the same behaviour.
3a51a2c removes SvUndef expressions from the generator, these were used in svundef intrinsics and some others. svundef now zeroes and so doesn't need a special generator expression, and the other uses call svundef.
a2f1fd4 changes uses of simd_reinterpret in the generator to transmute_unchecked - simd_reinterpret was expected to be added by rustc_scalable_vector(N) rust#143924 but was changed during reviews.
6fae36b changes the behaviour of auto-llvm-sign-conversion in the generator for return types to match what it does for arguments. auto-llvm-sign-conversion isn't used by any existing intrinsic specification files.
ccf6507 adds a static_assert_range helper that will be used in later commits.
8b544ef defines the scalable vector types and adds the modules that the generated intrinsics will eventually live in. This defines some additional non-public scalable vector types necessary to match the signatures of the LLVM intrinsics, and some traits for conversions between those and the public scalable vector types.
f6adf6f changes the generator to generate calls to SveInto::sve_into (a trait introduced in the previous commit). This trait is necessary because target_feature annotations are necessary on the trait method and that requires the method be unsafe, which is incompatible with the signature of Into::into.
5d9c7cc changes the standard library function linked to from some generated intrinsic doc comments, which was renamed since that part of the generator was written.
552a505 adds the intrinsic definitions - there's a lot of definitions and it likely isn't too practical to look over all of them. We later check that everything that should be being generated is being generated using stdarch-verify. Load/store tests will also be generated from these definitions.
e215315 runs the generator after the above commit to generate the Rust code for all the intrinsic definitions.
c2f8104 updates the intrinsics_data/arm_intrinsics.json file to add all the SVE intrinsics so that stdarch-verify can check all of the intrinsics are present.
1c57120 updates stdarch-verify with the necessary support and special-cases for SVE intrinsics.
1620151 limits SVE intrinsics to only aarch64-*, not arm64ec-*.
f3350a2 updates the intrinsic-test tool so that it can still parse arm_intrinsics.json and skips the SVE intrinsics.
9940d66 updates the assert_instr macro to support instantiating type generics as well as const generics.
98405ca removes assert_instr from svpfalse because its current implementation, while producing the correct result, doesn't generate a pfalse - this will require a rustc intrinsic.
f0dbe83 updates stdarch-test's logic for disassembling used by assert_instr. Previously [us]shll{2} would be rewritten to a normalised alias, but that was triggering for [us]shll[tb] added with SVE, which was incorrect.
b661e23 disables assert_instr for a handful of intrinsics on MSVC that don't seem to be supported by the disassembler.

The intrinsic test tool needs a lot of changes to work with SVE and this patch is getting big as it is, so we're going to do this as a follow-up (agreed in advance with @Amanieu).

This patch is based on #1509 and I've tried to preserve the co-authorship of everyone involved in that patch.

r? @Amanieu

davidtwco · 2026-04-05T12:55:55Z

Looks like this'll need another rustc patch to skip the "scalable vectors aren't supported on this architecture" error for doc builds

davidtwco · 2026-04-05T13:44:17Z

rust-lang/rust#154850 will fix this

…, r=JonathanBrouwer ast_validation: scalable vectors okay for rustdoc Scalable vector types in `core_arch` are cfg'd for aarch64 and for rustdoc, which can successfully document these types given any `--target` (`core_arch` CI uses `i686-unknown-linux-gnu`) - this shouldn't trigger the "scalable vectors not supported on arch" error. This fixes the CI failure in rust-lang/stdarch#2071.

Rollup merge of #154850 - davidtwco:scalable-vectors-rustdoc, r=JonathanBrouwer ast_validation: scalable vectors okay for rustdoc Scalable vector types in `core_arch` are cfg'd for aarch64 and for rustdoc, which can successfully document these types given any `--target` (`core_arch` CI uses `i686-unknown-linux-gnu`) - this shouldn't trigger the "scalable vectors not supported on arch" error. This fixes the CI failure in rust-lang/stdarch#2071.

davidtwco · 2026-04-07T12:09:04Z

rust-lang/rust#154950 will fix the current CI failure

library: no `cfg(target_arch)` on scalable intrinsics These intrinsics don't technically need to be limited to a specific architecture, they'll probably only make sense to use on AArch64, but this just makes it harder to use them in stdarch where it is appropriate (such as on `arm64ec`): requiring a rustc patch to land and be on nightly before stdarch work can proceed. So let's just not `cfg` them at all, they're perma-unstable anyway. Fixes CI failure in rust-lang/stdarch#2071

Rollup merge of #154950 - davidtwco:scalable-no-cfg, r=RalfJung library: no `cfg(target_arch)` on scalable intrinsics These intrinsics don't technically need to be limited to a specific architecture, they'll probably only make sense to use on AArch64, but this just makes it harder to use them in stdarch where it is appropriate (such as on `arm64ec`): requiring a rustc patch to land and be on nightly before stdarch work can proceed. So let's just not `cfg` them at all, they're perma-unstable anyway. Fixes CI failure in rust-lang/stdarch#2071

library: no `cfg(target_arch)` on scalable intrinsics These intrinsics don't technically need to be limited to a specific architecture, they'll probably only make sense to use on AArch64, but this just makes it harder to use them in stdarch where it is appropriate (such as on `arm64ec`): requiring a rustc patch to land and be on nightly before stdarch work can proceed. So let's just not `cfg` them at all, they're perma-unstable anyway. Fixes CI failure in rust-lang/stdarch#2071

adamgemmell

Just to add that we did pretty thorough manual reviews of each intrinsic at the time, and very few should be untested once the intrinsic-test changes go in (svundef comes to mind)

View changes since this review

adamgemmell · 2026-04-08T10:20:45Z

-                    }
-                })
-        };
+        let return_type_conversion = self


It's probably the case that this signed/unsigned conversion isn't needed at all. Doesn't really cause any harm though

adamgemmell · 2026-04-08T10:36:12Z

                );
            }

+            // Newer intrinsics don't have `rustc_legacy_const_generics` - assume they belong at


We could add this attribute for SVE intrinsics - see rust-lang/rust#149654. Some have pointed out that legacy generics is prone to bugs, but:

It aids porting from C in a small way

RfL really wants const args. I'm not sure if they necessarily want it for SIMD intrinsics though.

davidtwco · 2026-04-08T11:15:10Z

rust-lang/rust#154984 will fix this CI failure, eventually I'll just have issues on the stdarch side :)

davidtwco · 2026-04-08T15:47:32Z

rust-lang/rust#154984 will fix this CI failure, eventually I'll just have issues on the stdarch side :)

I've closed this and changed the patch to only add these intrinsics on target_arch = "aarch64" for now.

library: no `cfg(target_arch)` on scalable intrinsics These intrinsics don't technically need to be limited to a specific architecture, they'll probably only make sense to use on AArch64, but this just makes it harder to use them in stdarch where it is appropriate (such as on `arm64ec`): requiring a rustc patch to land and be on nightly before stdarch work can proceed. So let's just not `cfg` them at all, they're perma-unstable anyway. Fixes CI failure in rust-lang/stdarch#2071

davidtwco · 2026-04-10T15:07:21Z

I've fixed the CI failures locally and I'm now hitting into some other issues that I'm working through, which require some rustc patches that I'll upstream as they're ready (rust-lang/rust#155106 to start off)

This was accidentally removed in 713444d.

Instead of generating load/store tests based on the input filename - which no longer works given the expected input file structure of `stdarch-gen-arm` - add a simple global context option that SVE specs can set.

rustbot · 2026-04-13T14:08:50Z

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

davidtwco · 2026-04-13T14:10:51Z

Pushed some more changes - nothing too significant:

I've updated the assert_instr macro to support type generics in addition to const generics, as is required for some of the new intrinsics.
I've disabled the assert_instr for svpfalse just because it doesn't generate a pfalse due to the way its implemented - this will require another follow-up on the rustc side but isn't that important.

This will fail CI until rust-lang/rust#155106 and rust-lang/rust#155243 are merged, both are approved.

Co-authored-by: Adam Gemmell <Adam.Gemmell@arm.com> Co-authored-by: Jamie Cunliffe <Jamie.Cunliffe@arm.com> Co-authored-by: Jacob Bramley <jacob.bramley@arm.com> Co-authored-by: Luca Vizzarro <Luca.Vizzarro@arm.com>

arm64ec doesn't support SVE.

With SVE intrinsics in the `arm_intrinsics.json`, the parsing needs to be updated to know to expect any new fields.

SVE intrinsics have both type and const generics and so the `assert_instr` macro needs to be able to generate test cases with the type generics instantiated with the types provided in the attribute. Co-authored-by: Jamie Cunliffe <Jamie.Cunliffe@arm.com> Co-authored-by: Luca Vizzarro <Luca.Vizzarro@arm.com> Co-authored-by: Adam Gemmell <adam.gemmell@arm.com> Co-authored-by: Jacob Bramley <jacob.bramley@arm.com>

The implementation for this has the same behaviour as a `pfalse` but doesn't currently emit one until an intrinsic is added to emit a `zeroinitializer` for this.

SVE's `[us]shll[tb]` intructions have no aliases unlike Neon's `[us]hll{2}` so this logic needs adjusted to not accidentally rewrite the instruction.

`dumpbin.exe` produces `44a1c000`/`44e1c000`/`44a1c400`/`44e1c400` for `[su]mull[tb]` instead of the instruction name - so skip `assert_instr` for these intrinsics on MSVC targets.

There seemed to be non-deterministic failures on MSVC that looked like corruption of the FFR in CI. Until this can be investigated, to avoid any spurious failures, these tests are disabled.

davidtwco · 2026-04-14T13:09:37Z

CI failure is rust-lang/rust#155277. It should all be passing otherwise, as far as I can tell. I've disabled load/store tests on MSVC because it looked like there was non-deterministic corruption of the FFR in CI, so turning these off out of caution to avoid spurious CI failures until we can investigate as a follow-up.

davidtwco · 2026-04-15T10:13:37Z

CI is passing now so this should be ready for reviews

davidtwco added the A-aarch64 label Apr 5, 2026

rustbot assigned Amanieu Apr 5, 2026

davidtwco mentioned this pull request Apr 5, 2026

ast_validation: scalable vectors okay for rustdoc rust-lang/rust#154850

Merged

davidtwco force-pushed the sve-intrinsics branch from 7203b17 to d473c9a Compare April 7, 2026 11:11

davidtwco mentioned this pull request Apr 7, 2026

library: no cfg(target_arch) on scalable intrinsics rust-lang/rust#154950

Merged

bjorn3 reviewed Apr 7, 2026

View reviewed changes

Comment thread crates/core_arch/src/aarch64/sve/mod.rs

adamgemmell reviewed Apr 8, 2026

View reviewed changes

davidtwco mentioned this pull request Apr 8, 2026

target: support scalable vectors on arm64ec rust-lang/rust#154984

Closed

davidtwco force-pushed the sve-intrinsics branch from d473c9a to 6a6098b Compare April 8, 2026 11:52

Amanieu reviewed Apr 8, 2026

View reviewed changes

Comment thread crates/core_arch/src/macros.rs Outdated

Comment thread crates/core_arch/src/aarch64/sve/mod.rs Outdated

Comment thread crates/core_arch/src/aarch64/sve/mod.rs

Comment thread crates/core_arch/src/lib.rs Outdated

davidtwco force-pushed the sve-intrinsics branch from 6a6098b to 9401448 Compare April 8, 2026 15:45

davidtwco added 2 commits April 13, 2026 05:19

stdarch-verify: re-add runtime test check

3489955

This was accidentally removed in 713444d.

gen-arm: add generate_load_store_tests

d3455ed

Instead of generating load/store tests based on the input filename - which no longer works given the expected input file structure of `stdarch-gen-arm` - add a simple global context option that SVE specs can set.

davidtwco force-pushed the sve-intrinsics branch from ce98d8a to c21192c Compare April 13, 2026 14:11

davidtwco and others added 9 commits April 14, 2026 00:54

intrinsics_data: add sve intrinsics

c2f8104

Co-authored-by: Adam Gemmell <Adam.Gemmell@arm.com> Co-authored-by: Jamie Cunliffe <Jamie.Cunliffe@arm.com> Co-authored-by: Jacob Bramley <jacob.bramley@arm.com> Co-authored-by: Luca Vizzarro <Luca.Vizzarro@arm.com>

stdarch-verify: support sve

1c57120

Co-authored-by: Adam Gemmell <Adam.Gemmell@arm.com> Co-authored-by: Jamie Cunliffe <Jamie.Cunliffe@arm.com> Co-authored-by: Jacob Bramley <jacob.bramley@arm.com> Co-authored-by: Luca Vizzarro <Luca.Vizzarro@arm.com>

core_arch: no SVE on arm64ec

1620151

arm64ec doesn't support SVE.

intrinsic-test: update parsing for SVE intrinsics

f3350a2

With SVE intrinsics in the `arm_intrinsics.json`, the parsing needs to be updated to know to expect any new fields.

gen-arm: disable assert_instr for pfalse

98405ca

The implementation for this has the same behaviour as a `pfalse` but doesn't currently emit one until an intrinsic is added to emit a `zeroinitializer` for this.

stdarch-test: [us]shll[tb] have no aliases

f0dbe83

SVE's `[us]shll[tb]` intructions have no aliases unlike Neon's `[us]hll{2}` so this logic needs adjusted to not accidentally rewrite the instruction.

gen-arm: assert_instr on msvc for [su]mull[tb]

b661e23

`dumpbin.exe` produces `44a1c000`/`44e1c000`/`44a1c400`/`44e1c400` for `[su]mull[tb]` instead of the instruction name - so skip `assert_instr` for these intrinsics on MSVC targets.

core_arch: disable ld/st tests on msvc

ae9cb1c

There seemed to be non-deterministic failures on MSVC that looked like corruption of the FFR in CI. Until this can be investigated, to avoid any spurious failures, these tests are disabled.

davidtwco force-pushed the sve-intrinsics branch 11 times, most recently from d124b9e to b661e23 Compare April 14, 2026 12:17

davidtwco mentioned this pull request Apr 14, 2026

SVE and SME on AArch64 rust-lang/rust-project-goals#270

Open

27 tasks

Amanieu added this pull request to the merge queue Apr 15, 2026

Merged via the queue into rust-lang:main with commit 0956462 Apr 15, 2026
138 of 152 checks passed

davidtwco deleted the sve-intrinsics branch April 15, 2026 13:15

Conversation

davidtwco commented Apr 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davidtwco commented Apr 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davidtwco commented Apr 5, 2026

Uh oh!

davidtwco commented Apr 7, 2026

Uh oh!

Uh oh!

adamgemmell left a comment • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adamgemmell Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

adamgemmell Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

davidtwco commented Apr 8, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

davidtwco commented Apr 8, 2026

Uh oh!

davidtwco commented Apr 10, 2026

Uh oh!

rustbot commented Apr 13, 2026

Uh oh!

davidtwco commented Apr 13, 2026

Uh oh!

davidtwco commented Apr 14, 2026

Uh oh!

davidtwco commented Apr 15, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

davidtwco commented Apr 5, 2026 •

edited

Loading

davidtwco commented Apr 5, 2026 •

edited

Loading

adamgemmell left a comment •

edited by rustbot

Loading