Skip to content

fix: correct cuTile source locations for generated constants - #1385

Open
Arthur031221 wants to merge 5 commits into
NVIDIA:mainfrom
Arthur031221:fix/type-constant-locations
Open

Arthur031221 wants to merge 5 commits into
NVIDIA:mainfrom
Arthur031221:fix/type-constant-locations

Conversation

@Arthur031221

Copy link
Copy Markdown

Summary

cuTile users compiling with debug info get module-header locations for generated constants and wrapper call sites (#1379). Generated spans were resolved against the module's source base. This addresses the frontend part of #1379. Backend trap attribution remains unverified.

Changes

Resolve captured spans with their owning module and use the containing function definition for generated spans. The regression uses @elibol's assertion kernel from #1379.

Testing

Added line/full and cross-file regressions. From cutile-rs, cargo +1.98.0 test -p cutile --locked --test debug_info passes.

  • just -f cuda-oxide/Justfile check passes (the local mirror of CI: fmt, clippy, tests, guards, docs)
  • cargo oxide run <example> passes, or cuda-oxide/scripts/smoketest.sh -o '^<example>$'
  • New example added (if applicable)

Checklist

  • All commits signed off (git commit -s)
  • SPDX headers on new source files

Questions about the review? Ping us in #contributors on Discord.

Co-authored-by: Melih Elibol 1884921+elibol@users.noreply.github.com

Use captured module spans for source locations and the current function
definition for generated types, arrays and wrapper calls. Add regressions
using the assertion kernel from NVIDIA#1379 and a helper in another file.

Co-authored-by: Melih Elibol <1884921+elibol@users.noreply.github.com>
Signed-off-by: Arthur031221 <levi74108520963@gmail.com>
@Arthur031221
Arthur031221 requested a review from elibol as a code owner October 3, 2026 20:44
@copy-pr-bot

copy-pr-bot Bot commented Oct 3, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@Arthur031221

Copy link
Copy Markdown
Author

@elibol, checking in on this one. Anything else you need from me?

@elibol

elibol commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator

Reviewing this against the frontend portion of #1379 and the separate checklist in #1475. The function-definition fallback is a narrower guarantee than attributing every generated constant to its consuming statement, and that is a reasonable scope for this PR.

For acceptance, please tighten the regression around that actual guarantee:

  • The main assertion-kernel test currently accepts any line at or after the function definition. Check the expected file, exact fallback definition line, subprogram, and caller chain for selected generated constants; an unrelated later line should fail. Keep an ordinary user expression in the same test to show its precise location is not replaced by the fallback.
  • Exercise the cross-file/nested-helper case in both Line and Full, and verify the locations survive bytecode serialization. The existing exact cross-file constant test is a good starting point.
  • Record that the focused regression fails on the base revision and passes here. For emitted debug information, check a representative cubin with nvdisasm/DWARF and retain the existing live helper-breakpoint test.

The actual assertion SIGTRAP and reopened GPU-coredump line/frames are a separate backend acceptance test. Passing source-breakpoint or frontend-location tests does not close that portion of #1379; please keep the issue open for it. #1480 owns wrapper signature locations, and #1528 owns the explicit short-circuit expression spans, so these should not become prerequisites for this targeted fix.

Check the file, definition line, subprogram and caller chain of the
generated constants in the assertion kernel and in a nested cross-file
helper, in Line and Full, and that the same locations are present in the
serialized bytecode. A user literal in the same kernel must keep its own
line. An opt-in test checks the cubin line table with nvdisasm and DWARF.

Signed-off-by: Arthur031221 <levi74108520963@gmail.com>
@Arthur031221

Copy link
Copy Markdown
Author

Tightened the regression in a new commit on top of the reviewed one. The assertion-kernel test now compares file, subprogram with its definition line, line and caller chain of every constant against a fixed set, so an unrelated later line fails. The generated tile size must sit exactly on the definition line of checked_scale, the 0.0f32 and 1000.0f32 literals from validate_factor keep their own line with the helper and kernel call frames, and moving the fallback three lines past the definition fails all six new constant tests.

I added generated_nested to the fixture so generated is reached through a second helper in the other file. The direct and nested cross-file cases run in Line and Full, and every location is also looked up in the decoded bytecode debug section.

With _function.rs from the revision before this PR, all six constant tests fail, and so does the new ignored test nested_helper_cubin_lines_stay_inside_their_functions. It runs tileiras, nvdisasm --print-line-info and readelf on the nested kernel in Line and Full and requires every kernel line to lie between the kernel definition and its last statement. On that revision it reports a kernel line above the kernel definition.

On this commit debug_info gives 16 passed with --include-ignored (CUDA 13.2, RTX 5090), including cuda_gdb_stops_in_cross_file_helper. I didn't touch the SIGTRAP and coredump check, so nothing here should close #1379.

Integrate the v0.5.0 main baseline and run the generated-constant cubin regression in the CPU/assembler CI suite. Preserve the distinct backend trap-attribution scope.

Signed-off-by: Melih Elibol <1884921+elibol@users.noreply.github.com>
@elibol

elibol commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator

/ok to test 2794a67

@elibol

elibol commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator

Updated to 2794a67744f2ba34e8320781a3658b9dd289bf4c, preserving the original commits and merging current main. Added the changelog and enabled the nested-helper assembler regression in CPU CI.

Local CUDA 13.3: all 16 debug and six generated-source tests pass, including assembler/GPU/live CUDA-GDB; strict compiler Clippy and formatting pass. scripts/run_all.sh passes with serial tests and the existing temporary benchmark-runner correction to --features smoke-test (separately tracked in #1634, not included here).

Blocker: the new regression fails in CUDA 13.4 CI. The 13.2 lane passes. I assembled identical bytecode with tileiras 13.3.36 and 13.4.92: 13.3 retains helper source lines in Line/Full; 13.4 substitutes generated ./tile-ir-… locations. Full debug has four DW_TAG_inlined_subroutine entries with 13.3 and zero with 13.4. The pre-existing optional cubin_has_rust_lines_and_inline_frames test reproduces this on #1480, without this PR's compiler change. Switching nvdisasm versions does not recover the missing data.

Failing check: https://github.com/NVIDIA/cuda-rust/actions/runs/40389948453/job/121808098508. I retained the strict regression. This assembler/debug-info compatibility regression needs resolution before declaring the PR green. Actual assertion SIGTRAP/coredump attribution under #1379 remains separate. Release tracking: #1606.

Signed-off-by: Melih Elibol <1884921+elibol@users.noreply.github.com>
@elibol

elibol commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator

/ok to test 933e221

@copy-pr-bot

copy-pr-bot Bot commented Oct 26, 2026

Copy link
Copy Markdown

/ok to test 933e221

@elibol, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

@elibol

elibol commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator

/ok to test 933e221

@elibol

elibol commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator

Updated to 933e2216fa5e958076012e7088c7181f8f37deba to separate the frontend fix from the independently reproduced tileiras 13.4.92 debug-information loss.

The assembler regression still runs in CI. Exact frontend/serialized origins, kernel source presence/ranges, surviving helper ranges and successful assembly remain required. Only the exact 13.4 / V13.4.92 version may lack helper locations/inline scopes or report generated ./tile-ir-... paths; CI emits a visible warning when it observes this. All other versions, including 13.5, retain the full assertions. The book documents the limitation.

Validation: 12 CPU debug tests pass with 13.3 and 13.4; the nested-helper assembler regression passes in Line/Full with 13.2, 13.3 and 13.4.92. A test-only wrapper that changes the 13.4 assembler's reported version makes the same debug loss fail, confirming the exception does not cover future versions. This is not validation of CTK 13.5. Formatting, shell syntax and diff checks pass; self-review completed. Fresh CI is running.

This fixes the CI handling, not tileiras itself. The assertion SIGTRAP/GPU-coredump attribution still open in #1379 is not closed by this PR. v0.5.0 scope remains CTK 13.5 or earlier; no dependency on an unreleased CUDA-GDB fix. Tracked in #1606.

Signed-off-by: Melih Elibol <1884921+elibol@users.noreply.github.com>
@elibol

elibol commented Oct 27, 2026

Copy link
Copy Markdown
Collaborator

/ok to test b25b4fe

@elibol

elibol commented Oct 27, 2026

Copy link
Copy Markdown
Collaborator

Updated to b25b4fe275ce943e3fb33aca5f0f53db65fa3c53 after reproducing the failure with the actual binaries extracted from CI's nvidia/cuda:13.4.1-devel-ubuntu24.04 image: tileiras 13.4.59 and nvdisasm 13.4.49. My previous update handled 13.4.92, so it did not recognize the CI assembler.

The known-limit handling now covers exactly 13.4.59 and 13.4.92, and the regression prints the actual assembler fingerprint. Unknown versions, including 13.5, keep the full assertions. Frontend/serialized origins, assembly success, kernel locations/ranges, and any surviving helper ranges remain strict. The backend debug-information loss itself is not fixed here.

Validation:

  • The CI assembler reproduces the failure before this change and passes the Line/Full regression afterward, emitting the expected limitation warnings.
  • The same regression passes with 13.2, 13.3 and 13.4.92. The 12 CPU debug tests pass with 13.4.59.
  • Temporarily restoring the frontend implementation from main makes all six generated-constant location tests fail; restoring this PR's implementation makes them pass. The assembler exception does not mask the frontend regression.
  • A test-only altered version banner makes the missing helper information fail again, verifying that the exception does not silently cover future versions. This is not a CTK 13.5 test.
  • Formatting/diff checks and self-review pass. Fresh CI is running.

Why the other PRs are green: this PR enables nested_helper_cubin_lines_stay_inside_their_functions in CPU CI, and #1613 inherits it. The older optional cubin_has_rust_lines_and_inline_frames test also fails on #1480 with 13.4.59 when run explicitly, although #1480's ordinary CI passes. Thus the extra test exposes an existing assembler limitation rather than a defect introduced by this PR's compiler fix.

@elibol

elibol commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

CI is now green at b25b4fe275ce943e3fb33aca5f0f53db65fa3c53: 25 checks passed, one expected deployment skip, including Rust CodeQL and the full Tile workflow.

The CUDA 13.4 job ran the nested-helper assembler regression with the actual CI version, 13.4.59. Both Line and Full passed with explicit warnings about the existing backend debug-information loss. CUDA 13.2 passes without that exception. Exact frontend origins remain mandatory; removing the compiler fix still fails all six frontend regression tests.

The same test-handling fix is now included in #1613 at a8d91829f; all its applicable CI now passes as well (69 checks passed, one expected deployment skip), including Rust CodeQL. This resolves the CI blocker, not the backend limitation or the remaining assertion/coredump scope of #1379. This PR still needs review approval.

@elibol elibol left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants