Heavy sync init work (e.g. local ML models) starves stdio initialize/tool calls even with threadpool offload -- Windows GIL contention

[scotthibbs]

Environment

• mcp 1.28.1 (Python SDK, mcp.server.fastmcp.FastMCP)
• stdio transport, Windows 11
• Blocking work: sentence-transformers / torch (local embedding model), but this
  generalizes to any CPU/GIL-bound library load (onnxruntime, local LLM inference, etc.)

Summary

Related to #1839 / #1909, but a distinct failure mode that threadpool offload does not
fix: GIL-bound CPU work (loading a torch model) contends with the anyio stdio
transport's Windows pipe-reader thread for the GIL, regardless of where it runs in-process.
This makes several reasonable-looking designs fail in ways that are easy to ship and hard
to reproduce in a quick manual test.

Measurements

Same model load (sentence-transformers/all-MiniLM-L6-v2), only the location changes:

Where the load runs	initialize	First tool call
Standalone script, no event loop	—	~8s
Synchronously before mcp.run() ("warm on start")	blocks 8–70s	instant
In a background thread, started before mcp.run()	delayed	deadlocks
Lazily on first call, offloaded via anyio.to_thread / threadpool	instant	~70s
In a separate subprocess, talked to over a pipe	~1s	~8–9s

Only the subprocess isolation keeps both connection time and first-call time bounded.
The variance (8s → 70s) is driven by contention between the model-loading thread and
anyio's Windows stdio reader thread — both fighting for the GIL.

Why this matters beyond my case

It's tempting to "fix" a slow first tool call by warming eagerly before mcp.run().
That instead moves the stall onto the initialize handshake — which is worse, because a
client that doesn't get the handshake in time just drops the server with no visible
error. That's a silent failure (tools missing, no exception) that's intermittent
(fine when warm/cached, broken on a cold start), so it's easy to ship and hard to catch
in CI or a quick manual check.

Suggested fix / ask

Not asking for an SDK-level fix necessarily — run_in_threadpool from #1909 is the right
answer for I/O-bound blocking calls. But it'd help other implementers to:

1. Note in the docs/guidance for FastMCP that CPU/GIL-bound initialization (local model
   loads, etc.) should NOT be threadpool-offloaded in-process — it should run in a
   separate subprocess — since threadpool offload does not release the GIL contention
   the way it does for I/O-bound blocking calls.
2. Consider whether initialize should have an explicit, documented deadline/backpressure
   behavior so implementers know exactly how much startup latency is safe.

---

Disclaimer: I'm not a Python/asyncio expert — this diagnosis, the measurements, and
this write-up were produced by Claude Code (Anthropic's coding agent) while it was
building a local memory/retrieval MCP server for me and debugging why it intermittently
failed to connect. I'm filing it because the finding looked substantive and reproducible,
but I likely can't answer deep follow-up questions about the internals myself.

[IgorGanapolsky]

This matches what I've hit putting local models behind stdio MCP servers, and your table is the right diagnosis. The reason the threadpool offload doesn't save you: anyio.to_thread.run_sync still runs in-process, and a torch / sentence-transformers load is GIL-bound work that holds the GIL for most of those 8s. anyio's Windows stdio reader is itself a thread that needs the GIL to pump the pipe, so the loader and the transport reader serialize against each other no matter which thread the load sits on. That's why only the subprocess — separate interpreter, separate GIL — keeps both initialize and the first call responsive.

A few things that made the subprocess pattern actually reliable for me, beyond just isolating it:

• Warm inside the child before you accept work, and model the warming state explicitly. Have the child load the model, then flip a ready flag; while it isn't ready, tool calls return a typed unavailable/retry result instead of blocking. That keeps the ~8–9s first-call cost from landing on whichever unlucky request happens to arrive first.
• Supervise the child. A subprocess turns a deadlock into a crash you can recover from: the parent watches the pipe and, on EOF/exit, respawns and re-warms. Treat the pipe as at-least-once and put an id on every message so a mid-flight restart re-correlates instead of dropping or double-running a call.
• On Windows, spawn (not fork) re-imports everything in the child, so keep the child entrypoint tiny and import torch only inside it — otherwise the parent pays the import cost too.

If you'd rather not hand-roll the pipe protocol, running the embedding model behind a tiny local HTTP service and calling it from the tool gets you the same GIL isolation plus a real readiness/health endpoint for free — the MCP server stays pure-async and never imports torch. A little more plumbing, but every failure mode becomes observable instead of surfacing as a mysterious 70s stall.

[scotthibbs]

I gave my buddy Claude Code your response and he offered his thanks below:

Thanks — this is the clearest explanation of why the threadpool offload can't help that I've seen: the load holds the GIL, and anyio's Windows stdio reader is itself a thread that needs the GIL to pump the pipe, so the two serialize regardless of which thread the load sits on. That matches our measurements exactly (8s standalone → ~70s in-process under the live server).

For anyone finding this later, how our subprocess implementation lines up with your hardening points:

• Import isolation: the parent never imports torch — sentence_transformers is imported lazily inside the embed helper, and only the child calls it, so Windows spawn re-import never taxes the MCP process.
• Supervision: the parent treats EOF on the response pipe as worker death and respawns on the next call. We skipped message IDs deliberately: the protocol is strictly serialized under a lock (one request in flight at a time), so there's no correlation ambiguity — a mid-flight crash fails that one call with a clean error instead of dropping or double-running anything.
• Readiness flag: we let the first call block instead. The child warms eagerly at spawn (overlapping the client app's startup), and the worst cold first call we measured is ~9.4s — inside the client's tool timeout. In an interactive tool-calling context, a typed "retry later" result would push the retry loop onto the LLM, which seemed worse than one bounded wait. For models whose load exceeds the tool timeout, your ready-flag approach is clearly the right call.

The local-HTTP variant is a nice escape hatch too; for a single-client setup the pipe stays simpler (~20 lines, failures visible in the parent's stderr), so we kept it.

(Same disclaimer as the issue: this reply was drafted by Claude Code — I can't personally field deep follow-ups on the internals.)