Add Roslyn-format EnC CustomDebugInformation codec and portable PDB method CDI emission by NatElkins · Pull Request #20018 · dotnet/fsharp

NatElkins · 2026-07-01T21:08:29Z

Summary

Adds an internal AbstractIL module (EncMethodDebugInformation) implementing, byte for byte, the three Portable PDB CustomDebugInformation blob formats Roslyn persists per method for Edit and Continue:

EnC Local Slot Map (755F52A8-91C5-45BE-B4B8-209571E552BD)
EnC Lambda and Closure Map (A643004C-0240-496F-A783-30D64F4979DE)
EnC State Machine State Map (8B78CD68-2EDE-420B-980B-E15884B8AAA3)

with serializers, deserializers, a portable-PDB read-back helper, and an occurrence-key packing helper for deterministic syntax-offset slots. It also plumbs an optional methodCustomDebugInfoRows side channel through the IL binary writer options into the portable PDB generator, so a compilation can attach CDI rows to named methods.

Why

F# hot reload (#19941) needs Roslyn-compatible EnC method debug information in baseline PDBs so deltas can be chained across generations the same way C# Edit and Continue does. This PR carries the format layer only, so it can be reviewed as pure metadata/PDB code with deterministic byte evidence, independent of the hot reload feature.

There is intentionally no in-tree caller populating the side channel yet; the consumer lands with #19941. This follows the same upstreaming pattern as #20017: land isolated, test-covered infrastructure first, wire the feature in a later PR.

Behavior

All existing writer call sites pass an empty map, so emitted PDBs are byte-identical to before (covered by a test asserting zero CustomDebugInformation rows for an empty map).
Method names that do not identify exactly one IL method row are dropped (nothing attached); covered by absent-name and ambiguous-name tests.
All additions are internal; no public surface area change.

Tests

17 tests in CompilerService/EncMethodDebugInformationTests.fs:

blob round-trips for all three formats, including negative baselines, temps, ordinal-flagged slots, and negative state numbers;
golden-byte encodings matching Roslyn's writer;
cross-validation decoding and re-encoding, byte for byte, CDI blobs emitted by a real Roslyn (C#) compilation;
fail-closed occurrence-key packing, including an int32-overflow regression where a wrapped negative key previously escaped the bound check;
end-to-end synthetic PDB emission through the IL writer proving correct MethodDef parenting, zero rows for the empty map, and no rows for absent or ambiguous names.

Sequencing

This PR is part of splitting the F# hot reload work (#19941) into small, independently reviewable PRs. The planned order:

Wave 1 (independent of each other, no cross dependencies): Add ResetCompilerGeneratedNameState to compiler-generated name generators #20017 (generated-name counter reset), Add Roslyn-format EnC CustomDebugInformation codec and portable PDB method CDI emission #20018 (EnC CustomDebugInformation codec and method CDI emission), Add ECMA-335 EnC metadata delta writer #20019 (EnC metadata delta writer), plus two upcoming slices: stable synthesized-name replay and the typed-tree differ with rude-edit classification.
Wave 2: baseline reading and recorded CDI state (depends on Add Roslyn-format EnC CustomDebugInformation codec and portable PDB method CDI emission #20018 and the naming slice).
Wave 3: the delta emitter and symbol matcher.
Wave 4: the hot reload session, FCS surface, and the --test:HotReloadDeltas capture hook (F# hot reload: Edit-and-Continue delta emission behind --test:HotReloadDeltas #19941 in its final, much smaller form).
Last, explicitly experimental: Add an experimental flag-gated in-process compile path for hot reload sessions #20031 (the flag-gated in-process compile perf path).

Each slice is rebased on current main, carries its own tests, and is flag-off zero-cost. Everything stays draft until reviewed in order.
Role in the train: the hot reload baseline reader and PDB delta emission build on this codec to persist and reconstruct Edit and Continue state across processes.

Refresh status (2026-07-17)

Refreshed against dotnet/fsharp main at 5928e91; the current reviewed head is 7aaacc5.
A dedicated review pass completed for this PR, its findings were fixed in the lowest owning slice, and the PR has no unresolved review threads.
The downstream session, in-process compiler, umbrella, and SDK branches were restacked after the fixes, so this slice remains part of the decomposed review train.
The complete compiler stack passed the 11-step hot reload verifier, 456 service tests, 243 component tests with 2 expected skips, and 1411 EmittedIL tests with 3 expected skips. Replacement CI passed on this exact head.

github-actions · 2026-07-01T21:09:15Z

❗ Release notes required

You can open this PR in browser to add release notes: open in github.dev

✅ Found changes and release notes in following paths:

Change path Release notes path Description

src/Compiler docs/release-notes/.FSharp.Compiler.Service/11.0.100.md

Adds an internal, standalone ECMA-335 Edit-and-Continue metadata delta writer to AbstractIL: delta #- table stream and heap construction (DeltaMetadataTables, DeltaMetadataSerializer, DeltaTableLayout, DeltaIndexSizing), ECMA-335 II.24.2.6 coded-index encoding (DeltaMetadataEncoding), EncLog/EncMap emission, generation GUID chaining, user-string and standalone-signature token calculators (IlxDeltaStreams), and the coordinating writer (FSharpDeltaMetadataWriter) over a plain row-description input model (DeltaMetadataTypes, ILDeltaHandles, ILMetadataHeaps). The writer's inputs are row records (names, tokens, signatures, RVAs) plus heap offsets; it has no dependency on any semantic diffing or session machinery. It compiles with no in-tree consumer by design: the consumer is the F# hot reload work in dotnet#19941, following the same upstreaming pattern as dotnet#20017 and dotnet#20018 (land isolated, test-covered infrastructure first, wire the feature in a later PR). One line of ilwrite.fsi is touched to expose the pre-existing markerForUnicodeBytes so the delta writer reuses the exact string-marker logic of the full writer. No behavior change for any existing code path. Tests (130): coded-index encodings asserted against the production definitions and ECMA-335 II.24.2.6 order, System.Reflection.Metadata reader parity over emitted deltas, EncLog/EncMap correctness, stream layout, heap and index sizing, multi-generation heap-offset chaining asserted against computed expected values, standalone-signature rows asserted at baseline+1 from a real seeded baseline, and serializer failure paths.

NatElkins · 2026-07-01T22:53:53Z

/azp run

azure-pipelines · 2026-07-01T22:54:00Z

Commenter does not have sufficient privileges for PR 20018 in repo dotnet/fsharp

Adds an internal, standalone ECMA-335 Edit-and-Continue metadata delta writer to AbstractIL: delta #- table stream and heap construction (DeltaMetadataTables, DeltaMetadataSerializer, DeltaTableLayout, DeltaIndexSizing), ECMA-335 II.24.2.6 coded-index encoding (DeltaMetadataEncoding), EncLog/EncMap emission, generation GUID chaining, user-string and standalone-signature token calculators (IlxDeltaStreams), and the coordinating writer (FSharpDeltaMetadataWriter) over a plain row-description input model (DeltaMetadataTypes, ILDeltaHandles, ILMetadataHeaps). The writer's inputs are row records (names, tokens, signatures, RVAs) plus heap offsets; it has no dependency on any semantic diffing or session machinery. It compiles with no in-tree consumer by design: the consumer is the F# hot reload work in dotnet#19941, following the same upstreaming pattern as dotnet#20017 and dotnet#20018 (land isolated, test-covered infrastructure first, wire the feature in a later PR). One line of ilwrite.fsi is touched to expose the pre-existing markerForUnicodeBytes so the delta writer reuses the exact string-marker logic of the full writer. No behavior change for any existing code path. Tests (130): coded-index encodings asserted against the production definitions and ECMA-335 II.24.2.6 order, System.Reflection.Metadata reader parity over emitted deltas, EncLog/EncMap correctness, stream layout, heap and index sizing, multi-generation heap-offset chaining asserted against computed expected values, standalone-signature rows asserted at baseline+1 from a real seeded baseline, and serializer failure paths.

…ethod CDI emission Adds an internal AbstractIL module implementing, byte for byte, the three Portable PDB CustomDebugInformation blob formats Roslyn persists per method for Edit and Continue (EnC Local Slot Map, EnC Lambda and Closure Map, EnC State Machine State Map), with serializers, deserializers, a portable PDB read-back helper, and an occurrence-key packing helper for deterministic syntax-offset slots. Plumbs an optional methodCustomDebugInfoRows side channel through the IL binary writer options into the portable PDB generator so a compilation can attach CDI rows to named methods. Names that do not identify exactly one method row are dropped. All existing writer call sites pass an empty map, so emitted PDBs are byte-identical to before. No in-tree caller populates the map yet; the consumer is the F# hot reload work in dotnet#19941, following the same pattern as dotnet#20017 (land isolated, test-covered infrastructure first, wire the feature later). Tests: blob round-trips, Roslyn golden-byte encodings, cross-validation against CDI blobs emitted by a real Roslyn compilation, fail-closed occurrence-key packing (including an int32-overflow regression where a wrapped negative key previously escaped the bound check), and end-to-end synthetic PDB emission proving correct MethodDef parenting, zero rows for an empty map, and no rows for absent or ambiguous names.

ProcessStartInfo.ArgumentList does not exist on net472, which the component tests also target on Windows CI. Build the quoted argument string by hand instead.

xunit 3.2.2 no longer discovers internal test classes, so the module was silently skipped after rebasing onto current main (pre-existing internal test modules like CompilerService.Caches are likewise undiscovered there). Public visibility restores discovery; 17 tests run and pass.

…CI images The Roslyn cross-validation test shells out to dotnet build to produce a real Roslyn PDB. The process launch had two problems. It computed the host path by hand from __SOURCE_DIRECTORY__, which misses on CI images that carry no repo-local .dotnet at that depth, failing with Win32Exception before the build starts. It also left UseShellExecute at its default, which is true on net472 and rejects redirected streams, so every Desktop test leg failed deterministically with InvalidOperationException. Resolve the host like the rest of the test framework via TestFramework.initialConfig.DotNetExe, which prefers the repo-local .dotnet and falls back to PATH, and set UseShellExecute to false explicitly. Verified: FSharp.Compiler.ComponentTests builds clean; EncMethodDebugInformationTests 17 passed, 0 failed (net10.0); fantomas clean on the touched file.

Adds an internal, standalone ECMA-335 Edit-and-Continue metadata delta writer to AbstractIL: delta #- table stream and heap construction (DeltaMetadataTables, DeltaMetadataSerializer, DeltaTableLayout, DeltaIndexSizing), ECMA-335 II.24.2.6 coded-index encoding (DeltaMetadataEncoding), EncLog/EncMap emission, generation GUID chaining, user-string and standalone-signature token calculators (IlxDeltaStreams), and the coordinating writer (FSharpDeltaMetadataWriter) over a plain row-description input model (DeltaMetadataTypes, ILDeltaHandles, ILMetadataHeaps). The writer's inputs are row records (names, tokens, signatures, RVAs) plus heap offsets; it has no dependency on any semantic diffing or session machinery. It compiles with no in-tree consumer by design: the consumer is the F# hot reload work in dotnet#19941, following the same upstreaming pattern as dotnet#20017 and dotnet#20018 (land isolated, test-covered infrastructure first, wire the feature in a later PR). One line of ilwrite.fsi is touched to expose the pre-existing markerForUnicodeBytes so the delta writer reuses the exact string-marker logic of the full writer. No behavior change for any existing code path. Tests (130): coded-index encodings asserted against the production definitions and ECMA-335 II.24.2.6 order, System.Reflection.Metadata reader parity over emitted deltas, EncLog/EncMap correctness, stream layout, heap and index sizing, multi-generation heap-offset chaining asserted against computed expected values, standalone-signature rows asserted at baseline+1 from a real seeded baseline, and serializer failure paths.

github-actions · 2026-07-21T10:16:35Z

🔍 Tooling Safety Check — Affects-Compiler-Output
Affects-Compiler-Output: modifies ilwrite.fs, ilwritepdb.fs (EnC debug info)

Generated by PR Tooling Safety Check · opus46 5.4M · ◷

T-Gro

🤖 This review was generated by AI (@expert-reviewer agent). Findings may contain inaccuracies — please verify independently.

Reviewed the EnC CDI codec and the portable-PDB plumbing against Roslyn's EditAndContinueMethodDebugInformation format. The three blob serializers/deserializers match Roslyn byte-for-byte (baseline records, kind+1/0x3F masking, MinClosureOrdinal bias, stable offset sort, per-offset ≤256 relative-ordinal), the fail-closed validations mirror Roslyn's reader, and the writer relies on MetadataBuilder's automatic sort of the CustomDebugInformation table — the same guarantee the pre-existing embedded-source/source-link rows already depend on, so interleaving method-parented rows is safe. All existing call sites pass Map.empty, keeping flag-off PDBs byte-identical. Test coverage (golden bytes, Roslyn cross-validation, int32-overflow occurrence-key regression, synthetic end-to-end plumbing) is strong.

One low-severity, non-blocking behavioral note left inline for the future #19941 consumer regarding the scope of the name-ambiguity guard. No correctness/security/perf blockers found for this isolated infrastructure PR.

T-Gro · 2026-07-21T11:03:53Z

+        else
+            let nameCounts = Dictionary<string, int>()
+
+            for minfo in info.Methods do


Low-severity note for the future consumer (#19941): the uniqueness guard counts names only over info.Methods (the pdbinfo list), which is populated for MethodBody.IL/Abstract/PInvoke bodies but not for the fall-through _ -> 0x0000 bodiless-method case in ilwrite.fs. So the "attach nothing on ambiguity" guarantee is relative to the PDB method set, not the full MethodDef table — a name that is unique within info.Methods but also borne by a method absent from it will still attach. It attaches to a well-defined handle (never the wrong-typed method), so this isn't a corruption risk, but the caller in #19941 should be aware that name-keying resolves against info.Methods only. Not blocking for this codec-only PR.

github-project-automation Bot added this to F# Compiler and Tooling Jul 1, 2026

github-project-automation Bot moved this to New in F# Compiler and Tooling Jul 1, 2026

NatElkins force-pushed the pdb-method-cdi-emission branch from b293239 to b7e5325 Compare July 1, 2026 21:08

NatElkins mentioned this pull request Jul 1, 2026

Add ECMA-335 EnC metadata delta writer #20019

Open

NatElkins force-pushed the pdb-method-cdi-emission branch 2 times, most recently from 2844f2a to 4a3d73c Compare July 2, 2026 22:03

NatElkins force-pushed the pdb-method-cdi-emission branch from f17117d to d967a68 Compare July 7, 2026 14:22

NatElkins force-pushed the pdb-method-cdi-emission branch from d967a68 to 2d722d9 Compare July 8, 2026 18:02

NatElkins force-pushed the pdb-method-cdi-emission branch from 2d722d9 to c3ed355 Compare July 9, 2026 21:44

NatElkins added 4 commits July 17, 2026 12:27

Use ProcessStartInfo.Arguments for net472 compatibility

0d02145

ProcessStartInfo.ArgumentList does not exist on net472, which the component tests also target on Windows CI. Build the quoted argument string by hand instead.

NatElkins force-pushed the pdb-method-cdi-emission branch from c3ed355 to 7aaacc5 Compare July 17, 2026 23:11

NatElkins marked this pull request as ready for review July 18, 2026 01:09

NatElkins requested a review from a team as a code owner July 18, 2026 01:09

github-actions Bot added the ⚠️ Affects-Compiler-Output Tooling check: PR touches IL emission or codegen label Jul 21, 2026

T-Gro reviewed Jul 21, 2026

View reviewed changes

T-Gro added the AI-reviewed PR reviewed by AI review council label Jul 21, 2026

T-Gro self-requested a review July 21, 2026 11:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add Roslyn-format EnC CustomDebugInformation codec and portable PDB method CDI emission#20018

Add Roslyn-format EnC CustomDebugInformation codec and portable PDB method CDI emission#20018
NatElkins wants to merge 4 commits into
dotnet:mainfrom
NatElkins:pdb-method-cdi-emission

NatElkins commented Jul 1, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 1, 2026

Uh oh!

NatElkins commented Jul 1, 2026

Uh oh!

azure-pipelines Bot commented Jul 1, 2026

Uh oh!

github-actions Bot commented Jul 21, 2026

Uh oh!

T-Gro left a comment

Uh oh!

T-Gro Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

NatElkins commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why

Behavior

Tests

Sequencing

Refresh status (2026-07-17)

Uh oh!

github-actions Bot commented Jul 1, 2026

❗ Release notes required

Uh oh!

NatElkins commented Jul 1, 2026

Uh oh!

azure-pipelines Bot commented Jul 1, 2026

Uh oh!

github-actions Bot commented Jul 21, 2026

Uh oh!

T-Gro left a comment

Choose a reason for hiding this comment

Uh oh!

T-Gro Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

NatElkins commented Jul 1, 2026 •

edited

Loading