fuzz: differential fuzzer + workarounds for the two Espresso bugs it found (fixes #112, #113) - #114
Merged
Conversation
…nd two real bugs
Three design ideas fitted to this hardware (after a review of NNSmith-style
DNN-compiler fuzzing and EMI/metamorphic compiler testing):
1. Integer-exact mode: fp16 is exact on integers <= 2048 and a large op
subset is closed over them, so graphs whose numpy mirror proves every
intermediate integral get a BIT-EQUALITY oracle - no tolerances, no
false positives. Integer closure survives rewrites, so an
identity-augmented EMI variant probes the canonicalizer exactly.
2. Boundary-lattice sampling: dims and params drawn from the edges where
lowering bugs live (powers of two +/-1, tile factors {2,3,4,8},
nonzero last-axis slice offsets, fp16 danger values).
3. Float mode with a conditioning screen: the mirror runs at fp32 AND
fp64, and graphs where those disagree are discarded as ill-conditioned
rather than judged.
The oracle honors the compiler's contract: opt=0 answers exactly (int
mode), opt=1 answers to the autotuner's accuracy gate, since gated lossy
variants (int8 weights) are by design - the first fuzz batch taught us
that distinction by flagging them. Failures shrink to minimal reproducers
(kind-preserving, invariant-respecting) with replayable JSON specs and
fingerprint dedupe; scripts/fuzz.sh is the community wrapper with a
paste-ready report.
The first corrected 150-graph batch (master seed 42, ~10s on an M5) found
two real silent-wrongness bugs, filed with hand-minimized repros:
floor/ceil/round dropping a following scalar multiply (#112) and softplus
returning 0 for inputs >= 11 = ln(fp16_max) (#113).
Host-side machinery (generator, mirrors, shrinker) is CI-tested off-device
in tests/test_fuzz_harness.py; one requires_ane smoke test runs the full
loop on hardware.
#113) Both bugs receive CORRECT MIL and produce wrong programs, so the fix is to route the emitters around the fusion: - floor/ceil/round followed by a SCALAR multiply: Espresso fuses the scalar into the rounding op's scale slot and the kernel drops it (floor(x)*k returned floor(x)). Probing showed a FULL-shape constant blocks the mis-fusion while a (1,1) broadcast does not, scalar adds are honored, and no other op family is affected. The muls emitter and the binary-mul-with-size-1-const path now materialize a full-shape constant after rounding ops. - softplus: the ANE lowering computes exp(x) in fp16 and returns 0 for every x >= ln(65504) ~= 11.09 (sp(10)=10, sp(11)=0 - silent). The emitter now lowers the stable split softplus(min(x,10)) + relu(x-10), whose exp argument never overflows and whose error is <= 4.5e-5. Also fixed in passing, discovered by the regression test: ANE round is half-AWAY-FROM-ZERO while np.round is banker's half-to-even - the fuzzer mirror and the test reference now model half-away. Verified: both fuzzer finding specs replay as PASSING; 7 new regression tests (rounding x {muls, const-mul, adds}, softplus sweep incl. 30000 and the normal range); 180 tests across test_onnx/test_fuzz_harness/ test_dsp_welch; off-device selection unchanged. A fresh 400-graph fuzz batch found 7 NEW findings (incl. Espresso internal crashes on valid MIL) - triage tracked separately.
sbryngelson
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two commits: the fuzzer, and the fixes for what it caught on its first run.
The fuzzer
Designed after reviewing NNSmith-style DNN-compiler fuzzing and EMI/metamorphic compiler testing, specialized to this hardware: an integer-exact bit-equality oracle (fp16 is exact on integers ≤2048; closure survives rewrites, enabling exact EMI probes of the canonicalizer), boundary-lattice sampling (powers-of-two ±1, tile factors, danger values), a conditioning-screened float mode (fp32-vs-fp64 mirror agreement gates judgment), and a contract-aware oracle (opt=0 exact; opt=1 answers to the autotuner's accuracy gate — verified int8-variant signature). Kind-preserving shrinking to minimal replayable JSON specs;
scripts/fuzz.shcommunity wrapper; host-side machinery CI-tested off-device.What it caught, root-caused to the MIL level
Both bugs receive correct MIL and produce wrong programs — Apple's Espresso/ANECompiler, not ANEForge's emission:
floor/ceil/round+ scalar multiply: Espresso fuses the scalar into the rounding op's scale slot and the kernel drops it. Probed boundary: full-shape const blocks the fusion, (1,1) broadcast doesn't, scalar adds honored, no other op family affected. Emitters now materialize a full-shape constant after rounding ops (mulsand binary-mul-with-size-1-const paths).softplus(x≥11.09) = 0(fp16expoverflow inside the lowering; 11.09 = ln(65504)). Emitter now lowerssoftplus(min(x,10)) + relu(x−10)— never overflows, error ≤ 4.5e-5, verified across ±30000.roundis half-away-from-zero (np.round is banker's) — mirror and tests corrected.Verification (on-device, M5)
--repro).tests/test_espresso_workarounds.py; 180 tests across the ONNX/fuzz/dsp suites; off-device selection unchanged; ruff/pylint/pyright clean.unordered_map::at— on valid MIL); triage is follow-up work, which is rather the point of the tool.