proc_macro: Fix cfg_attr inner attrs in file modules

[Dnreikronos]

Fixes #159648

Inner proc-macro attrs produced by cfg_attr only carry the nested attr span. Splitting source around that span tries to lex two unmatched halves, so file modules fail with an unclosed delimiter.

Lex the complete module body first, then remove the expanded attr from its nested token stream. Sibling attrs stay in place, and empty cfg_attr wrappers use their trace span and disappear. imo this is cleaner than teaching source slicing about every nested shape.

Tests exercise sibling attrs on either side of the proc macro, then cover the single-attr case. fyi, denied lints make sibling preservation observable; ltm if another nested case should live here.

[rustbot]

The parser was modified, potentially altering the grammar of (stable) Rust
which would be a breaking change.

cc @fmease

[rustbot]

r? @nnethercote

rustbot has assigned @nnethercote.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Why was this reviewer chosen?

The reviewer was selected based on:

• Owners of files modified in this PR: compiler, parser
• compiler, parser expanded to 74 candidates
• Random selection from 17 candidates

[Dnreikronos]

cc// @mejrs

[petrochenkov]

At this point collecting the tokens properly would probably be simpler.

For an easy solution, I'd suggest removing unwrap_or_emit_fatal and returning None if lexing fail, i.e. using the same fallback as in all the other erroneous cases.

[Dnreikronos]

At this point collecting the tokens properly would probably be simpler.

For an easy solution, I'd suggest removing unwrap_or_emit_fatal and returning None if lexing fail, i.e. using the same fallback as in all the other erroneous cases.

Done. I removed unwrap_or_emit_fatal and return None if lexing fails. I agree with you that this makes more sense here since rebuilding fake tokens is already best effort. fyi, I cancel the diagnostics before returning because they still need to be consumed. ltm if you had a different shape in mind.

[petrochenkov]

(I meant reverting all the other changes besides this one.)

[Dnreikronos]

(I meant reverting all the other changes besides this one.)

sry, I misunderstood or request!
I will work on it rn

[Dnreikronos]

(I meant reverting all the other changes besides this one.)

Done, I reverted everything except the lexer failure fallback you suggested.

The PR diff now only removes unwrap_or_emit_fatal, cancels the lexer diagnostics, and returns None so the existing fallback takes over. btw, I brought the original regression test back locally to check the minimal patch and it passes. I removed it again afterward to keep the diff exactly as requested.

[nnethercote]

This seems fine.

@bors r+ rollup

Inner proc-macro attrs produced by cfg_attr only carry the nested attr span.

FWIW, I'm working on a PR to change Attribute::span so it always covers the full attribute from # to ], because this cfg_attr case has been annoying me.

[rust-bors]

📌 Commit 60dac3d has been approved by nnethercote

It is now in the queue for this repository.

[petrochenkov]

@Dnreikronos
Could you add the test back? And then squash commits.
I'll re-approve after that.
r? @petrochenkov
@bors r-

[rust-bors]

This pull request was unapproved.

This PR was contained in a rollup (#159879), which was unapproved.

View changes since this unapproval

[petrochenkov]

@bors r=nnethercote,petrochenkov

[rust-bors]

📌 Commit 1f8b15f has been approved by nnethercote,petrochenkov

It is now in the queue for this repository.

🌲 The tree is currently closed for pull requests below priority 1000. This pull request will be tested once the tree is reopened.

Reason for tree closure: spurious failures