Windows: UnicodeEncodeError crashes the process *after* the workflow succeeds, truncating the --silent JSON result

[franklixuefei]

Summary

On Windows with a non-UTF-8 locale, conductor run crashes with UnicodeEncodeError after the workflow has already completed successfully, while serializing its final JSON result to stdout. The workflow's real output is written correctly to disk, but:

• the JSON result on stdout is truncated mid-field, so it cannot be parsed by any caller;
• the process exits non-zero, so automation concludes the run failed;
• with --web-bg, the dashboard shows "Connection lost — the Conductor process may have crashed", which reads as data loss.

The work is complete and safe on disk. Nothing in the UI or the exit status says so.

Environment

Conductor	v0.1.26
Python	3.10.11
OS	Windows 11, PowerShell 7
sys.stdout.encoding	cp1252 (both redirected and at the console)
PYTHONUTF8 / PYTHONIOENCODING	unset (default)

What I observed

A plan workflow ran ~55 minutes across 6 agent turns and finished. From the captured stderr, in this order:

✓ decision_summarizer  (171.97s, claude-sonnet-4.6, 91851 in/59 out, $0.3046, → ['file_path'])
   → $end
⏱ Total workflow execution: 3317.76s
Workflow completed successfully

Token Usage Summary
  Input:  1,050,049 tokens
  ...
  Total: $6.1694

Event log written to: ...events.jsonl

+--------------------------- ❌ UnicodeEncodeError ---------------------------+
|  'charmap' codec can't encode character '\u2192' in position 190:           |
|  character maps to <undefined>                                              |
+-----------------------------------------------------------------------------+

Workflow completed successfully is at stderr line 27412; UnicodeEncodeError at line 27432. The crash is strictly after successful completion.

The stdout JSON stops dead here:

{
  "purpose": "...",
  "plan_file": "...plan.md",
  "decisions_file": "...decisions.md",
  "technical_review_score": 92,
  "readability_review_score": 93,
  "technical_review_feedback":        <-- ends mid-field, file ends here

ConvertFrom-Json rejects it. Both output documents (88 KB and 13 KB) were written to disk correctly.

Root cause

U+2192 (→) is not encodable in cp1252. Python 3.10 on Windows uses the locale encoding for sys.stdout unless PYTHONUTF8/PYTHONIOENCODING is set — and it does so whether or not stdout is redirected (verified below, this is not a redirection-only issue as I first assumed).

The failing write is the final result JSON, because it embeds agent-generated content — here technical_review_feedback, which contains →. Agent prose routinely contains → — ✅ ⚠️, so this is data-dependent: it fires whenever a reviewer/agent happens to use one, which makes it look intermittent.

Standalone reproduction (no workflow needed)

python -c "print('x'*185 + '\u2192')" > out.txt
# exit code: 1
# UnicodeEncodeError: 'charmap' codec can't encode character '\u2192'
#   in position 185: character maps to <undefined>
#   File "...\lib\encodings\cp1252.py", line 19, in encode

Identical codec, character, and error class as the conductor crash — only the offset differs, being a function of the string.

Encoding is cp1252 regardless of redirection

python -c "import sys; print(sys.stdout.encoding)"           # cp1252
python -c "import sys; ..." > file.txt                        # cp1252

So --web-bg does not cause it — it makes it maximally alarming, because the dashboard interprets the dead process as a crashed run.

Impact

1. Looks like total data loss. ~55 min and $6.17 of agent work appeared lost. It wasn't — but recovering it meant knowing to go read %LOCALAPPDATA%\Temp\conductor\*.log and checking the output paths by hand.
2. Breaks --silent as a machine interface. Its contract is "JSON result only"; a truncated document is worse than none, because naive callers may parse a prefix.
3. Non-zero exit on a successful run — CI and wrapper scripts will report failure.
4. Silently intermittent. Depends on whether an agent emitted a non-cp1252 glyph, so it will not reproduce reliably for anyone triaging it.

Suggested fix

Force UTF-8 on conductor's own streams at entry, independent of locale:

# conductor/cli entrypoint, before any output
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
sys.stderr.reconfigure(encoding="utf-8", errors="replace")

errors="replace" is worth having as a belt-and-braces measure: conductor should never die while reporting a result it already computed. Serializing the final result with ensure_ascii=True would also make the JSON path safe by construction.

Given that the same crash would kill a run mid-flight if the offending glyph appeared in progress output, the entrypoint fix looks preferable to patching individual call sites.

Related

• Windows: Unicode characters in agent output are garbled (charmap encoding) #89 (Windows: Unicode characters in agent output are garbled) — same area, different failure. That one is mojibake in inter-agent passing; this one is a hard crash on the final-result write. Windows: Unicode characters in agent output are garbled (charmap encoding) #89's environment notes say PYTHONIOENCODING=utf-8 "fixes Python's own stdout", which suggests the stdout path was assumed handled; this shows it is not, in the default case where that variable is unset.
• --web-bg child process dies when launched from a job-managed shell on Windows (missing CREATE_BREAKAWAY_FROM_JOB) #195 (--web-bg child dies on Windows) — also --web-bg + Windows, but that is process-lifetime (job objects); this is stream encoding. Distinct.

Secondary observation (not diagnosed)

With conductor --silent run ... --web-bg, the dashboard URL never reached my console — I found Dashboard: http://127.0.0.1:52913 only inside the background process's redirected stderr log, and initially had to recover it by scanning listening TCP ports for the conductor PID. I have not verified whether this is --silent suppressing it or --web-bg emitting it only to the child's stream, so I am reporting the observation rather than asserting a cause. It matters because tooling built on this workflow is instructed to surface that URL to the user.

[jrob5756]

Starting to take a look at this.

[jrob5756]

Verified this one. Your diagnosis is right and the standalone repro made it quick to confirm.

One correction, and it moves where the fix belongs. json.dumps(result) at cli/app.py:543 already defaults to ensure_ascii=True, so conductor hands off a pure-ASCII string with \u2192 in it. What un-escapes it is rich: Console.print_json defaults to ensure_ascii=False (rich 14.3.1), so it re-parses and re-serializes the literal → back in immediately before the write. The JSON channel is being made unsafe by the pretty-printer, not by our serialization.

That gives a more surgical fix than the entrypoint reconfigure: pass ensure_ascii=True to print_json. It's lossless, since \u2192 is valid JSON and round-trips through ConvertFrom-Json, where errors="replace" would quietly turn your review feedback into ?. Five sites: cli/app.py:543,556,998,1004 and cli/doctor.py:94.

Two other things I turned up.

First, why it truncated instead of writing nothing. Rich's _write_buffer only takes a chunked write path (≤8192 chars per file.write) if WINDOWS, as a workaround for CPython #82052. Each chunk is a separate write, so the ASCII chunks ahead of the arrow land on disk and the one containing it dies. I reproduced it: 8181 of 12176 bytes persisted. On POSIX rich does a single write and you'd have gotten an empty file instead. That's why your JSON ends mid-document.

Second, why the crash is strictly after success. CPython gives sys.stderr errors='backslashreplace', which never raises, while sys.stdout gets strict under a non-UTF-8 locale. So all 27,411 lines of progress output got through and only the final stdout write died. Your stderr arrows would have been written as literal \u2192 escapes rather than →, which is #89 in different clothes.

I still want your entrypoint fix as defence in depth, because show, validate, and the registry commands all print YAML-sourced text to stdout. conductor/__init__.py is the place: it's the earliest common import for both the console script and python -m conductor, and the faulthandler block already sitting there is the pattern to copy (guarded, warns, never breaks startup). I'd use errors="backslashreplace" over replace for the reason above, and skip the reconfigure entirely when PYTHONIOENCODING is set so a deliberate choice wins.

Worth folding in too: _build_bg_env in cli/bg_runner.py doesn't set PYTHONUTF8=1, unlike executor/script.py:140 and mcp_auth.py:61. Your --web-bg observation is exactly this. The parent opens the log files with encoding="utf-8" at bg_runner.py:318-323, but the child inherits only the fd and builds its own cp1252 sys.stdout from the locale.

There's a trap in the testing. ci.yml runs the whole suite on ubuntu-latest; only the install_scripts job touches windows-latest. So the tests have to simulate the condition rather than rely on the platform: build a TextIOWrapper with encoding="cp1252", render through it, and assert both that nothing raises and that json.loads round-trips back to the original dict. Please don't gate anything on sys.platform or it will never actually run.

On your secondary observation: it's --silent, not --web-bg. cli/app.py:509 is gated by if is_verbose(), and the docstring on _print_web_bg_human_gate_notice says so outright. I think it's the wrong call, since the URL is a result rather than progress, but it's deliberate today so I'd rather it got its own issue than rode along here. Same for adding a Windows CI job.

One question. You listed Python 3.10.11, but requires-python is >=3.12. It doesn't change anything above, the codec behaviour is identical, but I'd like to know which interpreter actually ran in case there's a separate install bug hiding there.

Assigned it to you. Are you OK picking this one up? Happy to take it back if you'd rather just review.