Skip to content

Commit 17749f2

Browse files
amilendraamilendra
committed
Add floating point matrix multiply-add widening intrinsics
Adds intrinsic support for the FMMLA matrix multiply instructions introduced by the 2024 dpISA. FEAT_F8F32MM: Neon/SVE2 FP8 to single-precision FEAT_F8F16MM: Neon/SVE2 FP8 to half-precision FEAT_SVE_F16F32MM: SVE half-precision to single-precision
1 parent a4bc412 commit 17749f2

4 files changed

Lines changed: 68 additions & 0 deletions

File tree

‎main/acle.md‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -466,6 +466,8 @@ Armv8.4-A [[ARMARMv84]](#ARMARMv84). Support is added for the Dot Product intrin
466466
* Added feature test macro for FEAT_SSVE_FEXPA.
467467
* Added feature test macro for FEAT_CSSC.
468468
* Added support for FEAT_FPRCVT intrinsics and `__ARM_FEATURE_FPRCVT`.
469+
* Added support for modal 8-bit floating point matrix multiply-add widening intrinsics.
470+
* Added support for 16-bit floating point matrix multiply-add widening intrinsics.
469471

470472
### References
471473

@@ -2354,6 +2356,27 @@ is hardware support for the SVE forms of these instructions and if the
23542356
associated ACLE intrinsics are available. This implies that
23552357
`__ARM_FEATURE_MATMUL_INT8` and `__ARM_FEATURE_SVE` are both nonzero.
23562358

2359+
##### Multiplication of 8-bit floating-point matrices
2360+
2361+
This section is in
2362+
[**Beta** state](#current-status-and-anticipated-changes) and might change or be
2363+
extended in the future.
2364+
2365+
`__ARM_FEATURE_F8F32MM` is defined to `1` if there is hardware support
2366+
for the SVE 8-bit floating-point matrix multiply (FEAT_SSVE_F8F32MM)
2367+
instructions and if the associated ACLE intrinsics are available.
2368+
2369+
`__ARM_FEATURE_F8F16MM` is defined to `1` if there is hardware support
2370+
for the SVE 8-bit floating-point matrix multiply (FEAT_SSVE_F8F16MM)
2371+
instructions and if the associated ACLE intrinsics are available.
2372+
2373+
##### Multiplication of 16-bit floating-point matrices
2374+
2375+
`__ARM_FEATURE_SVE_F16F32MM` is defined to `1` if there is hardware support
2376+
for the SVE 16-bit floating-point to 32-bit floating-point matrix multiply and add
2377+
(FEAT_SVE_F16F32MM) instructions and if the associated ACLE intrinsics are available.
2378+
This implies that `__ARM_FEATURE_SVE2p1` is nonzero.
2379+
23572380
##### Multiplication of 32-bit floating-point matrices
23582381

23592382
`__ARM_FEATURE_SVE_MATMUL_FP32` is defined to `1` if there is hardware support
@@ -2646,6 +2669,9 @@ be found in [[BA]](#BA).
26462669
| [`__ARM_FEATURE_SVE_BITS`](#scalable-vector-extension-sve) | The number of bits in an SVE vector, when known in advance | 256 |
26472670
| [`__ARM_FEATURE_SVE_MATMUL_FP32`](#multiplication-of-32-bit-floating-point-matrices) | 32-bit floating-point matrix multiply extension (FEAT_F32MM) | 1 |
26482671
| [`__ARM_FEATURE_SVE_MATMUL_FP64`](#multiplication-of-64-bit-floating-point-matrices) | 64-bit floating-point matrix multiply extension (FEAT_F64MM) | 1 |
2672+
| [`__ARM_FEATURE_SVE_F16F32MM`](#multiplication-of-16-bit-floating-point-matrices) | 16-bit floating-point matrix multiply extension (FEAT_SVE_F16F32MM) | 1 |
2673+
| [`__ARM_FEATURE_F8F16MM`](#multiplication-of-8-bit-floating-point-matrices) | Modal 8-bit floating-point matrix multiply extension (FEAT_SSVE_F8F16MM) | 1 |
2674+
| [`__ARM_FEATURE_F8F32MM`](#multiplication-of-8-bit-floating-point-matrices) | Modal 8-bit floating-point matrix multiply extension (FEAT_SSVE_F8F32MM) | 1 |
26492675
| [`__ARM_FEATURE_SVE_MATMUL_INT8`](#multiplication-of-8-bit-integer-matrices) | SVE support for the integer matrix multiply extension (FEAT_I8MM) | 1 |
26502676
| [`__ARM_FEATURE_SVE_PREDICATE_OPERATORS`](#scalable-vector-extension-sve) | Level of support for C and C++ operators on SVE vector types | 1 |
26512677
| [`__ARM_FEATURE_SVE_VECTOR_OPERATORS`](#scalable-vector-extension-sve) | Level of support for C and C++ operators on SVE predicate types | 1 |
@@ -9383,6 +9409,31 @@ BFloat16 floating-point multiply vectors.
93839409
uint64_t imm_idx);
93849410
```
93859411

9412+
### SVE2 floating-point matrix multiply-accumulate instructions.
9413+
9414+
#### FMMLA (widening, FP8 to FP16)
9415+
9416+
8-bit floating-point matrix multiply-add to half-precision.
9417+
```c
9418+
// Only if (__ARM_FEATURE_SVE2 && __ARM_FEATURE_F8F16MM)
9419+
svfloat16_t svmmla[_f16_mf8]_fpm(svfloat16_t zda, svmfloat8_t zn, svmfloat8_t zm, fpm_t fpmd);
9420+
```
9421+
9422+
#### FMMLA (widening, FP8 to FP32)
9423+
9424+
8-bit floating-point matrix multiply-add to single-precision.
9425+
```c
9426+
// Only if (__ARM_FEATURE_SVE2 && __ARM_FEATURE_F8F32MM)
9427+
svfloat32_t svmmla[_f32_mf8]_fpm(svfloat32_t zda, svmfloat8_t zn, svmfloat8_t zm, fpm_t fpmd);
9428+
```
9429+
#### FMMLA (widening, FP16 to FP32)
9430+
9431+
16-bit floating-point matrix multiply-add to single-precision.
9432+
```c
9433+
// Only if __ARM_FEATURE_SVE_F16F32MM
9434+
svfloat32_t svmmla[_f32_f16](svfloat32_t zda, svfloat16_t zn, svfloat16_t zm);
9435+
```
9436+
93869437
### SVE2.1 instruction intrinsics
93879438

93889439
The specification for SVE2.1 is in

‎neon_intrinsics/advsimd.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6202,3 +6202,14 @@ The intrinsics in this section are guarded by the macro ``__ARM_NEON``.
62026202
| <code>float32x4_t <a href="https://developer.arm.com/architectures/instruction-sets/intrinsics/vmlalltbq_laneq_f32_mf8_fpm" target="_blank">vmlalltbq_laneq_f32_mf8_fpm</a>(<br>&nbsp;&nbsp;&nbsp;&nbsp; float32x4_t vd,<br>&nbsp;&nbsp;&nbsp;&nbsp; mfloat8x16_t vn,<br>&nbsp;&nbsp;&nbsp;&nbsp; mfloat8x16_t vm,<br>&nbsp;&nbsp;&nbsp;&nbsp; const int lane,<br>&nbsp;&nbsp;&nbsp;&nbsp; fpm_t fpm)</code> | `vd -> Vd.4S`<br>`vm -> Vn.16B`<br>`vm -> Vm.B`<br>`0 <= lane <= 15` | `FMLALLBB Vd.4S, Vn.16B, Vm.B[lane]` | `Vd.4S -> result` | `A64` |
62036203
| <code>float32x4_t <a href="https://developer.arm.com/architectures/instruction-sets/intrinsics/vmlallttq_lane_f32_mf8_fpm" target="_blank">vmlallttq_lane_f32_mf8_fpm</a>(<br>&nbsp;&nbsp;&nbsp;&nbsp; float32x4_t vd,<br>&nbsp;&nbsp;&nbsp;&nbsp; mfloat8x16_t vn,<br>&nbsp;&nbsp;&nbsp;&nbsp; mfloat8x8_t vm,<br>&nbsp;&nbsp;&nbsp;&nbsp; const int lane,<br>&nbsp;&nbsp;&nbsp;&nbsp; fpm_t fpm)</code> | `vd -> Vd.4S`<br>`vm -> Vn.16B`<br>`vm -> Vm.B`<br>`0 <= lane <= 7` | `FMLALLBB Vd.4S, Vn.16B, Vm.B[lane]` | `Vd.4S -> result` | `A64` |
62046204
| <code>float32x4_t <a href="https://developer.arm.com/architectures/instruction-sets/intrinsics/vmlallttq_laneq_f32_mf8_fpm" target="_blank">vmlallttq_laneq_f32_mf8_fpm</a>(<br>&nbsp;&nbsp;&nbsp;&nbsp; float32x4_t vd,<br>&nbsp;&nbsp;&nbsp;&nbsp; mfloat8x16_t vn,<br>&nbsp;&nbsp;&nbsp;&nbsp; mfloat8x16_t vm,<br>&nbsp;&nbsp;&nbsp;&nbsp; const int lane,<br>&nbsp;&nbsp;&nbsp;&nbsp; fpm_t fpm)</code> | `vd -> Vd.4S`<br>`vm -> Vn.16B`<br>`vm -> Vm.B`<br>`0 <= lane <= 15` | `FMLALLBB Vd.4S, Vn.16B, Vm.B[lane]` | `Vd.4S -> result` | `A64` |
6205+
6206+
## Matrix multiplication intrinsics from Armv9.6-A
6207+
6208+
### Vector arithmetic
6209+
6210+
#### Matrix multiply
6211+
6212+
| Intrinsic | Argument preparation | AArch64 Instruction | Result | Supported architectures |
6213+
|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------|-------------------------------|-------------------|---------------------------|
6214+
| <code>float16x4_t <a href="https://developer.arm.com/architectures/instruction-sets/intrinsics/vmmlaq_f16" target="_blank">vmmlaq_f16</a>(<br>&nbsp;&nbsp;&nbsp;&nbsp; float16x4_t r,<br>&nbsp;&nbsp;&nbsp;&nbsp; mfloat8x16_t a,<br>&nbsp;&nbsp;&nbsp;&nbsp; mfloat8x16_t b)</code> | `r -> Vd.4H`<br>`a -> Vn.16B`<br>`b -> Vm.16B` | `FMMLA Vd.4H, Vn.16B, Vm.16B` | `Vd.4H -> result` | `A64` |
6215+
| <code>float32x4_t <a href="https://developer.arm.com/architectures/instruction-sets/intrinsics/vmmlaq_f32" target="_blank">vmmlaq_f32</a>(<br>&nbsp;&nbsp;&nbsp;&nbsp; float32x4_t r,<br>&nbsp;&nbsp;&nbsp;&nbsp; mfloat8x16_t a,<br>&nbsp;&nbsp;&nbsp;&nbsp; mfloat8x16_t b)</code> | `r -> Vd.4S`<br>`a -> Vn.16B`<br>`b -> Vm.16B` | `FMMLA Vd.4S, Vn.16B, Vm.16B` | `Vd.4S -> result` | `A64` |

‎tools/intrinsic_db/advsimd.csv‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4830,3 +4830,7 @@ float32x4_t vmlalltbq_lane_f32_mf8_fpm(float32x4_t vd, mfloat8x16_t vn, mfloat8x
48304830
float32x4_t vmlalltbq_laneq_f32_mf8_fpm(float32x4_t vd, mfloat8x16_t vn, mfloat8x16_t vm, __builtin_constant_p(lane), fpm_t fpm) vd -> Vd.4S;vm -> Vn.16B; vm -> Vm.B; 0 <= lane <= 15 FMLALLBB Vd.4S, Vn.16B, Vm.B[lane] Vd.4S -> result A64
48314831
float32x4_t vmlallttq_lane_f32_mf8_fpm(float32x4_t vd, mfloat8x16_t vn, mfloat8x8_t vm, __builtin_constant_p(lane), fpm_t fpm) vd -> Vd.4S;vm -> Vn.16B; vm -> Vm.B; 0 <= lane <= 7 FMLALLBB Vd.4S, Vn.16B, Vm.B[lane] Vd.4S -> result A64
48324832
float32x4_t vmlallttq_laneq_f32_mf8_fpm(float32x4_t vd, mfloat8x16_t vn, mfloat8x16_t vm, __builtin_constant_p(lane), fpm_t fpm) vd -> Vd.4S;vm -> Vn.16B; vm -> Vm.B; 0 <= lane <= 15 FMLALLBB Vd.4S, Vn.16B, Vm.B[lane] Vd.4S -> result A64
4833+
4834+
<SECTION> Matrix multiplication intrinsics from Armv9.6-A
4835+
float16x4_t vmmlaq_f16(float16x4_t r, mfloat8x16_t a, mfloat8x16_t b) r -> Vd.4H;a -> Vn.16B;b -> Vm.16B FMMLA Vd.4H, Vn.16B, Vm.16B Vd.4H -> result A64
4836+
float32x4_t vmmlaq_f32(float32x4_t r, mfloat8x16_t a, mfloat8x16_t b) r -> Vd.4S;a -> Vn.16B;b -> Vm.16B FMMLA Vd.4S, Vn.16B, Vm.16B Vd.4S -> result A64

‎tools/intrinsic_db/advsimd_classification.csv‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4717,3 +4717,5 @@ vmlalltbq_lane_f32_mf8_fpm Vector arithmetic|Multiply|Multiply-accumulate and wi
47174717
vmlalltbq_laneq_f32_mf8_fpm Vector arithmetic|Multiply|Multiply-accumulate and widen
47184718
vmlallttq_lane_f32_mf8_fpm Vector arithmetic|Multiply|Multiply-accumulate and widen
47194719
vmlallttq_laneq_f32_mf8_fpm Vector arithmetic|Multiply|Multiply-accumulate and widen
4720+
vmmlaq_f16 Vector arithmetic|Matrix multiply
4721+
vmmlaq_f32 Vector arithmetic|Matrix multiply

0 commit comments

Comments
 (0)