nullable(#372): per-file #nullable enable remediation of the EmailIntelligence classifier cluster

.claude/agent-memory/atomic-executor/MEMORY.md

@@ -1,10 +1,11 @@
 # Atomic Executor Memory Index
 
+- [#372 email-classifier nullable patterns](project_372_email_classifier_nullable_patterns.md) — engine props set post-ctor=`null!`; factory returns `T?`; Prediction.Class T? cascade=`.Class!`; net481 IsNullOrEmpty no-narrow; `(await Deserialize())!` not `await Deserialize()!`; DTO `= null!` adds a coverage line; measured 30 emitting (not epic ~18)
+- [Timed-out MSTest leaves detached runner](project_timedout_mstest_leaves_detached_runner.md) — a timed-out Invoke-MSTestWithCoverage bash call leaves a detached pwsh runner respawning testhosts; second run contends over user.config (ConfigurationErrorsException in unrelated TaskTree/ToDoModel) and hangs — kill the pwsh runner too, verify 0, then rerun with >=8min timeout
 - [#371 OutlookObjects nullable lessons](project_371_outlookobjects_nullable_lessons.md) — public-signature nullable change regresses OTHER nullable-enabled files in same assembly (DfDeedle) — check TOTAL UtilitiesCS CS86xx per batch, keep public tuples non-null w/ ! at null sites; lazy-field CS8618→Lazy<T>?; ToLazy has class constraint; _item=null! for GetOrLoad ref-match; ForEach commented-but-resolves (grep gate=flag not block)
 - [Sibling-worktree shared-tooling hazard](project_sibling_worktree_shared_tooling_hazard.md) — a concurrent agent in a DIFFERENT worktree crashes your testhost + clobbers /tmp logs via shared global vstest/dotnet-coverage; use session scratchpad for logs, poll for a quiet machine before test runs, trust bash `$?` not log counts, never kill/commit others' processes/memory files
 - [#375 residuals nullable gotchas](project_375_residuals_nullable_gotchas.md) — CS8644 inherited-interface mismatch from oblivious #366 base fixed with a `#nullable disable` island on the class-declaration line (not `!`); full-solution Rebuild cleans SVGControl.dll → isolated gate CS0006, re-run no-TWAE Build first; MeetingItemHelper Lazy<T> classes: bulk `= null!` + `(...)!` getters/lambdas beats widening the API (Python utf-8-sig for CRLF+BOM files)
 - [#364 nullable-gate pre-existing blockers](project_364_nullable_gate_preexisting_blockers.md) — full-solution pragma-only TWAE gate fails at baseline (vendored SVGControl CS0649 + non-HelperClasses CS0618/CS0168); verify CS86xx via isolated UtilitiesCS build w/ BuildProjectReferences=false; analyzer-version drift needs nuget-install into packages/; coverage script single-assembly StrictMode bug
-
 - [Nullable per-file pragma gate mechanics](project_nullable_pragma_gate_mechanics.md) — solution-wide TWAE aborts on vendored SVGControl CS0649; verify via isolated `UtilitiesCS.csproj -t:Rebuild -p:Platform=AnyCPU -p:BuildProjectReferences=false` + grep CS86xx=0
 - [Analyzer version skew on fresh worktree](project_analyzer_version_skew_fresh_worktree.md) — first analyzer build fails CS0006 (Meziantou 3.0.101/Sonar 10.27/BannedApi 3.3.4 missing); nuget install old versions into gitignored packages/, don't edit 16 csproj
 - [Nullable remediation annotation patterns](project_nullable_remediation_annotation_patterns.md) — net481 no post-condition attrs; EmailRecord struct `= default!`; `.ToString()!` for string cells; IsNullOrEmpty overload gotcha; `x!.M()` for defensive-flow-state

.claude/agent-memory/atomic-executor/project_372_email_classifier_nullable_patterns.md

@@ -0,0 +1,21 @@
+---
+name: 372-email-classifier-nullable-patterns
+description: Recurring annotation patterns and gotchas found remediating the UtilitiesCS EmailIntelligence classifier cluster (#372) under per-file #nullable enable
+metadata:
+  type: project
+---
+
+Wave-1 child #372 (`utilitiescs-nullable-email-classifier`) remediated 36 files under `UtilitiesCS/EmailIntelligence/{Bayesian,ClassifierGroups,Flags}`. Measured CS86xx set was **30 emitting files / 188 diagnostics** (research's ~30 static estimate, NOT the epic's ~18); both `Flags/` and `Performance/` were confirmed in scope. Reusable patterns for sibling children:
+
+- **Engine properties set post-construction** (via a builder / `InitAsync` / property-setter that the compiler doesn't track as ctor-init) → `= null!` (or `= default!` for unconstrained `T`). Applies to `Globals`, `ClassifierGroup`, `EngineName`, `TypedItem`, `AsyncAction`, `CgUtilities`, delegate fields, `Combined`, lazy caches. Keeps `IsActivated => X is not null` and hot-path derefs working with no runtime change. Property-setter assignment (`Parent = x`) does NOT satisfy CS8618 for the backing field — hence `= null!` on the field.
+- **Factory/Init returns** that already `return null`/`return default` → `Task<T?>` / `T?`; this cascades to callers (`CreateEngineAsync`, `ValidateJson`, destructuring sites) which then also go nullable.
+- **Prediction<T>.Class became `T?` in Batch A** → cascades everywhere `.Class` feeds a non-null `string` (Triage/Actionable/Category); fix with `.Class!` (classify results are populated in practice).
+- **#363 ThrowIfNull no-narrowing**: after a bare `X.ThrowIfNull()`, the next deref needs a justified `X!` with a `// why` comment (delegates set in InitAsync alongside ClassifierGroup). Never convert to `if (x is null) throw`.
+- **`x as Corpus`/`as MailItem`/MemberwiseClone** → wrap `(x as T)!` (Clone/cast always yields T).
+- **net481 `string.IsNullOrEmpty(x)` does NOT narrow** — after the check, x still needs `!`.
+- **DTO auto-property `= null!` adds an executable line** (BayesianMetricTypes went 97%→93% because a few measurement DTOs aren't instantiated in tests). Not an AC4 regression (no previously-covered line lost coverage; overall coverage rose) but prefer nullable `?` where consumers don't deref-as-non-null to avoid new uncovered lines.
+- **`await Deserialize()!` gotcha**: `x = await Deserialize(...)!` binds `!` to the Task (result stays T?). Must write `x = (await Deserialize(...))!` to null-forgive the awaited result.
+- **FlagDetails.List setter handles null** → `SetXList` assignments use `.List = value!`. Nullable events: `event Handler? Name;`.
+- Interfaces `IFolderPredictor`/`IFlagTranslator` were NOT forced (no CS8766/8767) — left EXCLUDE.
+
+See [[project_nullable_remediation_annotation_patterns]], [[project_nullable_epic_pragma_gate_and_analyzer_restore]], [[project_analyzer_version_skew_fresh_worktree]].

.claude/agent-memory/atomic-executor/project_timedout_mstest_leaves_detached_runner.md

@@ -0,0 +1,23 @@
+---
+name: timedout-mstest-leaves-detached-runner
+description: A timed-out Invoke-MSTestWithCoverage bash call leaves a detached pwsh runner alive that keeps respawning testhosts; a second run then contends over shared user.config and fails/hangs
+metadata:
+  type: project
+---
+
+When a Bash-tool call running `scripts/vscode/Invoke-MSTestWithCoverage.ps1` hits its timeout, the harness kills the bash shell but NOT the detached grandchildren: the `pwsh` process running the script, plus its `dotnet-coverage`/`vstest.console`/`testhost` pipeline, keep running. Re-running tests then produces TWO concurrent pipelines that contend.
+
+Signature of the problem:
+- `Get-Process vstest.console,testhost,dotnet-coverage` shows two pipelines with StartTimes ~minutes apart when you only launched one.
+- A spurious test failure in a totally unrelated assembly (observed: `TaskTree.Test.TaskTreeControllerMoveLogicTests.MoveObjectsToSibling_RootTarget_RemovesFromRootsAndReseeds`) throwing `System.Configuration.ConfigurationErrorsException: The configuration file has been changed by another program` on `...user.config` (via `ToDoModel.IDList.GetNextToDoID` -> `ApplicationSettingsBase.Save()`). Two concurrent testhosts write the same per-user `user.config` -> this error.
+- Coverage XML shows anomalously low branch-rate (~0.61 vs ~0.76 normal) because a contended run is partial.
+- The full suite (5702 tests, ~37s alone) appears to "hang" for minutes.
+
+**Why:** MSTest/settings writers share one machine-global `user.config`; two live pipelines race it.
+
+**How to apply:** Before EVERY test run (and always after a test-run timeout), kill BOTH the pipeline binaries AND the lingering pwsh runner, then verify zero before launching:
+```
+Get-CimInstance Win32_Process -Filter "Name='pwsh.exe'" | ? { $_.CommandLine -match 'Invoke-MSTest' } | % { Stop-Process -Id $_.ProcessId -Force }
+Get-Process vstest.console,testhost,testhost.x86,dotnet-coverage -EA SilentlyContinue | Stop-Process -Force
+```
+Then re-check `Get-Process vstest.console,testhost,dotnet-coverage` == 0 AND the `Invoke-MSTest` pwsh count == 0 before running. A plain `Stop-Process` on the binaries alone is NOT enough — the surviving pwsh runner respawns testhosts. Give the clean run a generous timeout (>= 8 min) so it never times out and re-stacks. Note: leftover idle `MSBuild.exe` `nodemode` reuse-pool workers are harmless (near-zero CPU) and are NOT the cause — do not confuse them with a concurrent build. See [[project_concurrent_executor_same_worktree]].

UtilitiesCS/EmailIntelligence/Bayesian/BayesianClassifierExtensions.cs

@@ -1,4 +1,5 @@
-using System;
+#nullable enable
+using System;
 using System.Collections.Generic;
 using System.Linq;
 using System.Text;

UtilitiesCS/EmailIntelligence/Bayesian/BayesianClassifierGroup.cs

@@ -1,4 +1,5 @@
-using System;
+#nullable enable
+using System;
 using System.Collections.Concurrent;
 using System.Collections.Generic;
 using System.Linq;
@@ -55,7 +56,9 @@ public int TotalEmailCount
         }
         protected int _totalEmailCount;
 
-        public IApplicationGlobals Globals { get; set; }
+        // Injected after construction; null! preserves the non-null posture at the hot-path
+        // call sites (Tokenize/TokenizeAsync) without a per-call guard.
+        public IApplicationGlobals Globals { get; set; } = null!;
 
         [JsonIgnore]
         public Func<object, IApplicationGlobals, IEnumerable<string>> Tokenize

UtilitiesCS/EmailIntelligence/Bayesian/BayesianClassifierShared.cs

@@ -1,4 +1,5 @@
-using System;
+#nullable enable
+using System;
 using System.Collections.Concurrent;
 using System.Collections.Generic;
 using System.Linq;
@@ -243,14 +244,21 @@ public BayesianClassifierGroup Parent
             get => _parent;
             internal set => _parent = value;
         }
-        protected BayesianClassifierGroup _parent;
+
+        // Set via the Parent property (which the compiler does not track as field init) or by
+        // deserialization; the parameterless and single-tag constructors leave it unset
+        // (pre-existing). null! preserves the non-null posture without a hot-path guard.
+        protected BayesianClassifierGroup _parent = null!;
 
         public string Tag
         {
             get => _tag;
             set => _tag = value;
         }
-        private string _tag;
+
+        // Assigned by every functional constructor and by deserialization; the parameterless
+        // constructor leaves it unset (pre-existing). null! preserves the non-null posture.
+        private string _tag = null!;
 
         #endregion public properties
 
@@ -781,14 +789,14 @@ public struct WordStream(string name, string[] words)
             public string[] Words = words;
         }
 
-        public (double, List<(string word, double prob)>) Chi2SpamProb(
+        public (double, List<(string word, double prob)>?) Chi2SpamProb(
             WordStream wordStream,
             bool evidence
         ) => Chi2SpamProb(wordStream.Words, evidence);
 
         public double Chi2SpamProb(WordStream wordStream) => Chi2SpamProb(wordStream, false).Item1;
 
-        public (double, List<(string word, double prob)>) Chi2SpamProb(
+        public (double, List<(string word, double prob)>?) Chi2SpamProb(
             IDictionary<string, int> tokenFrequency,
             bool evidence
         ) => Chi2SpamProb([.. tokenFrequency.Keys], evidence);
@@ -807,7 +815,7 @@ public double Chi2SpamProb(IDictionary<string, int> tokenFrequency) =>
         /// <param name="wordStream"></param>
         /// <param name="evidence"></param>
         /// <returns></returns>
-        public (double, List<(string word, double prob)>) Chi2SpamProb(
+        public (double, List<(string word, double prob)>?) Chi2SpamProb(
             string[] tokens,
             bool evidence
         )
@@ -908,10 +916,10 @@ public double chi2Q(double x2, int v)
         //public List<(double prob, string word, WordInfo record)> GetClues(WordStream wordStream) => GetClues([.. wordStream.Words]);
         //public List<(double prob, string word, WordInfo record)> GetClues(IDictionary<string, int> tokenFrequency) => GetClues([.. tokenFrequency.Keys]);
 
-        public List<(double prob, string word, WordInfo record)> GetClues(HashSet<string> tokenSet)
+        public List<(double prob, string word, WordInfo? record)> GetClues(HashSet<string> tokenSet)
         {
             var mindist = Knobs.MinDist;
-            var clues = new List<(double distance, double prob, string word, WordInfo record)>();
+            var clues = new List<(double distance, double prob, string word, WordInfo? record)>();
             var push = clues.Add;
             foreach (string token in tokenSet)
             {
@@ -930,11 +938,11 @@ public double chi2Q(double x2, int v)
             return selectedClues;
         }
 
-        public (double distance, double prob, string word, WordInfo record) GetWordDistance(
+        public (double distance, double prob, string word, WordInfo? record) GetWordDistance(
             string word
         )
         {
-            WordInfo record = GetWordInfo(word);
+            WordInfo? record = GetWordInfo(word);
             double prob;
             if (record is null)
             {
@@ -949,7 +957,7 @@ string word
             return (distance, prob, word, record);
         }
 
-        public WordInfo GetWordInfo(string word)
+        public WordInfo? GetWordInfo(string word)
         {
             int m = _match.TokenFrequency.TryGetValue(word, out int bCount) ? bCount : 0;
             int nm = Parent.SharedTokenBase.TokenFrequency.TryGetValue(word, out int gCount)

UtilitiesCS/EmailIntelligence/Bayesian/Corpus.cs

@@ -1,4 +1,5 @@
-using System;
+#nullable enable
+using System;
 using System.Collections.Concurrent;
 using System.Collections.Generic;
 using System.Collections.Specialized;
@@ -91,7 +92,11 @@ public ConcurrentDictionary<string, int> TokenFrequency
             get => _tokenFrequency;
             protected set => _tokenFrequency = value;
         }
-        protected ConcurrentDictionary<string, int> _tokenFrequency;
+
+        // Non-null in normal use; two legacy concurrency-level constructors leave it unset
+        // (pre-existing behavior, preserved). null! keeps the analyzer's non-null posture
+        // without changing runtime state (the field already defaulted to null).
+        protected ConcurrentDictionary<string, int> _tokenFrequency = null!;
 
         public int TokenCount
         {
@@ -176,7 +181,8 @@ out int result
 
         public object Clone()
         {
-            var result = this.MemberwiseClone() as Corpus;
+            // MemberwiseClone on a Corpus instance always yields a Corpus, so the as-cast never returns null.
+            var result = (this.MemberwiseClone() as Corpus)!;
             var tokenFrequency = this.TokenFrequency ?? new ConcurrentDictionary<string, int>();
             result.TokenFrequency = new ConcurrentDictionary<string, int>(tokenFrequency);
             return result;
@@ -188,7 +194,8 @@ public object Clone()
 
         public static Corpus operator +(Corpus c1, Corpus c2)
         {
-            var result = c1.Clone() as Corpus;
+            // Clone() always returns a Corpus, so the as-cast never returns null.
+            var result = (c1.Clone() as Corpus)!;
 
             foreach (var kvp in c2.TokenFrequency)
             {
@@ -205,13 +212,14 @@ public static async Task<Corpus> SubtractAsync(
             Corpus c1,
             Corpus c2,
             CancellationToken token,
-            SegmentStopWatch sw = null
+            SegmentStopWatch? sw = null
         )
         {
             sw ??= new SegmentStopWatch().Start();
             sw.LogDuration("SubtractAsync Start");
 
-            var result = c1.Clone() as Corpus;
+            // Clone() always returns a Corpus, so the as-cast never returns null.
+            var result = (c1.Clone() as Corpus)!;
             sw.LogDuration("clone universe");
 
             var processors = Math.Max(Environment.ProcessorCount - 2, 1);
@@ -252,7 +260,8 @@ public static async Task<Corpus> SubtractAsync(
 
         public static Corpus operator -(Corpus c1, Corpus c2)
         {
-            var result = c1.Clone() as Corpus;
+            // Clone() always returns a Corpus, so the as-cast never returns null.
+            var result = (c1.Clone() as Corpus)!;
             foreach (var kvp in c2.TokenFrequency)
             {
                 if (result.TokenFrequency.TryGetValue(kvp.Key, out int count))
@@ -277,8 +286,9 @@ public static (Corpus NotMatchFiltered, Corpus MatchFiltered) SubtractFilter(
             int minCt
         )
         {
-            var result = all.Clone() as Corpus;
-            var matchClone = match.Clone() as Corpus;
+            // Clone() always returns a Corpus, so the as-casts never return null.
+            var result = (all.Clone() as Corpus)!;
+            var matchClone = (match.Clone() as Corpus)!;
 
             result.TokenFrequency = new ConcurrentDictionary<string, int>(
                 result.TokenFrequency.Where(x => x.Value * negTokenWt >= minCt)

UtilitiesCS/EmailIntelligence/Bayesian/CorpusInherit.cs

@@ -1,4 +1,5 @@
-using System;
+#nullable enable
+using System;
 using System.Collections.Concurrent;
 using System.Collections.Generic;
 using System.IO;
@@ -56,8 +57,8 @@ public CorpusInherit(int concurrencyLevel, int capacity, IEqualityComparer<strin
 
         #region Public Properties and Methods
 
-        private string _id;
-        public string Id
+        private string? _id;
+        public string? Id
         {
             get => _id;
             set => _id = value;
@@ -152,7 +153,7 @@ bool askUserOnError
 
         internal static CorpusInherit Deserialize(FilePathHelper disk, bool askUserOnError)
         {
-            CorpusInherit dictionary = null;
+            CorpusInherit? dictionary = null;
             bool writeDictionary = false;
             DialogResult response = DialogResult.Ignore;
 
@@ -189,14 +190,18 @@ internal static CorpusInherit Deserialize(FilePathHelper disk, bool askUserOnErr
 
             if (writeDictionary)
             {
-                dictionary.Serialize();
+                // Reached only from the catch paths, where dictionary was assigned by
+                // CreateEmpty (which returns non-null or throws); never null here.
+                dictionary!.Serialize();
             }
-            return dictionary;
+            // All reaching paths leave dictionary non-null: the success path throws when
+            // DeserializeJson returns null, and each catch path assigns CreateEmpty (non-null) or throws.
+            return dictionary!;
         }
 
-        protected static CorpusInherit DeserializeJson(FilePathHelper disk)
+        protected static CorpusInherit? DeserializeJson(FilePathHelper disk)
         {
-            CorpusInherit collection;
+            CorpusInherit? collection;
             var settings = new JsonSerializerSettings();
             settings.TypeNameHandling = TypeNameHandling.Auto;
             settings.Formatting = Formatting.Indented;
@@ -279,7 +284,7 @@ public void SerializeThreadSafe(string filePath)
         }
 
         private ThreadSafeSingleShotGuard _serializationRequested = new();
-        private TimerWrapper _timer;
+        private TimerWrapper? _timer;
 
         protected void RequestSerialization(string filePath)
         {

UtilitiesCS/EmailIntelligence/Bayesian/DoNotSerializeContractResolver.cs

@@ -1,4 +1,5 @@
-using System;
+#nullable enable
+using System;
 using System.Collections.Generic;
 using System.Linq;
 using System.Reflection;

UtilitiesCS/EmailIntelligence/Bayesian/FolderHierarchyNode.cs

@@ -1,3 +1,4 @@
+#nullable enable
 using System;
 using Newtonsoft.Json;