Add multi-project daemon support for next dev/build#92022
Draft
Add multi-project daemon support for next dev/build#92022
Conversation
When multiple --project flags are detected, spawn a shared Turbopack daemon process and N worker Next.js processes that communicate via Unix domain sockets (or Windows named pipes). This commit adds: - CLI: --project flag parsing, --turbopack-daemon hidden flag, `next internal turbopack-daemon` subcommand, multi-project orchestrator - Rust IPC: bincode protocol types, daemon server, client stub in next-api - NAPI: startTurbopackDaemon and connectTurbopackDaemon entry points - JS bindings: daemon handle threading through createProject - Test fixtures and parseProjectGroups unit tests The daemon server currently returns stub responses for most operations. The actual ProjectContainer dispatch will be wired up in a follow-up. Co-Authored-By: Claude <noreply@anthropic.com>
…tInstance - Add optional `daemon` parameter to `project_new` for IPC path - Refactor `ProjectInstance` to support both local and remote (daemon-backed) projects via `Option` fields and `RemoteProject` struct - Add `ctx()` and `container()` accessor methods on `ProjectInstance` - When daemon is provided, short-circuit to IPC call instead of full in-process setup (tracing, profiling, TurboTasks creation, etc.) Co-Authored-By: Claude <noreply@anthropic.com>
nextjs-bot
added
created-by: Turbopack team
The WASM fallback binding object was missing startTurbopackDaemon and connectTurbopackDaemon, and the generated-native.d.ts declaration for projectNew was missing the new optional daemon parameter. Both caused type errors during pnpm build / tsc. Co-Authored-By: Claude <noreply@anthropic.com>
Merging this PR will not alter performance
Comparing 
Footnotes
|
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
The WASM binding stub added for startTurbopackDaemon/connectTurbopackDaemon introduced a new error message string that must be registered in errors.json. Running pnpm build detected it automatically via the check_error_codes step. Co-Authored-By: Claude <noreply@anthropic.com>
The initial `inner` allocation at the top of `connect` was unused — the real client is built a few lines later after the socket is opened. Pointed out in code review. Co-Authored-By: Claude <noreply@anthropic.com>
Collaborator
Failing test suitesCommit: 2b37d21 | About building and testing Next.js
Expand output● dynamic-css-client-navigation react lazy nodejs › should not remove style when navigating from static imported component to react lazy at runtime nodejs
Expand output● i18n Support basePath › development mode › should navigate with locale prop correctly GSP ● i18n Support basePath › development mode › should navigate with locale false correctly
Expand output● parallel-routes-and-interception (trailingSlash: true) › parallel routes › should support parallel route tab bars ● parallel-routes-and-interception (trailingSlash: true) › parallel routes › should match parallel routes ● parallel-routes-and-interception (trailingSlash: true) › parallel routes › should match parallel routes in route groups ● parallel-routes-and-interception (trailingSlash: true) › parallel routes › should throw a 404 when no matching parallel route is found ● parallel-routes-and-interception (trailingSlash: true) › parallel routes › should render nested parallel routes ● parallel-routes-and-interception (trailingSlash: true) › parallel routes › should support layout files in parallel routes ● parallel-routes-and-interception (trailingSlash: true) › parallel routes › should only scroll to the parallel route that was navigated to ● parallel-routes-and-interception (trailingSlash: true) › parallel routes › should apply the catch-all route to the parallel route if no matching route is found ● parallel-routes-and-interception (trailingSlash: true) › parallel routes › should match the catch-all routes of the more specific path, if there is more than one catch-all route ● parallel-routes-and-interception (trailingSlash: true) › parallel routes › should navigate with a link with prefetch=false ● parallel-routes-and-interception (trailingSlash: true) › parallel routes › should display all parallel route params with useParams ● parallel-routes-and-interception (trailingSlash: true) › parallel routes › should load CSS for a default page that exports another page ● parallel-routes-and-interception (trailingSlash: true) › parallel routes › should handle a loading state ● parallel-routes-and-interception (trailingSlash: true) › route intercepting with dynamic routes › should render intercepted route ● parallel-routes-and-interception (trailingSlash: true) › route intercepting with prerendered dynamic routes › should render intercepted route ● parallel-routes-and-interception (trailingSlash: true) › route intercepting with dynamic optional catch-all routes › should render intercepted route ● parallel-routes-and-interception (trailingSlash: true) › route intercepting with dynamic catch-all routes › should render intercepted route ● parallel-routes-and-interception (trailingSlash: true) › route intercepting › should render intercepted route ● parallel-routes-and-interception (trailingSlash: true) › route intercepting › should render an intercepted route from a slot ● parallel-routes-and-interception (trailingSlash: true) › route intercepting › should render an intercepted route at the top level from a nested path ● parallel-routes-and-interception (trailingSlash: true) › route intercepting › should render intercepted route from a nested route ● parallel-routes-and-interception (trailingSlash: true) › route intercepting › should re-render the layout on the server when it had a default child route ● parallel-routes-and-interception (trailingSlash: true) › route intercepting › should render modal when paired with parallel routes ● parallel-routes-and-interception (trailingSlash: true) › route intercepting › should support intercepting with beforeFiles rewrites ● parallel-routes-and-interception (trailingSlash: true) › route intercepting › should support intercepting local dynamic sibling routes ● parallel-routes-and-interception (trailingSlash: true) › route intercepting › should intercept on routes that contain hyphenated/special dynamic params ● parallel-routes-and-interception (trailingSlash: true) › route intercepting › should not have /default paths in the prerender manifest ● parallel-routes-and-interception (trailingSlash: false) › parallel routes › should support parallel route tab bars ● parallel-routes-and-interception (trailingSlash: false) › parallel routes › should match parallel routes ● parallel-routes-and-interception (trailingSlash: false) › parallel routes › should match parallel routes in route groups ● parallel-routes-and-interception (trailingSlash: false) › parallel routes › should throw a 404 when no matching parallel route is found ● parallel-routes-and-interception (trailingSlash: false) › parallel routes › should render nested parallel routes ● parallel-routes-and-interception (trailingSlash: false) › parallel routes › should support layout files in parallel routes ● parallel-routes-and-interception (trailingSlash: false) › parallel routes › should only scroll to the parallel route that was navigated to ● parallel-routes-and-interception (trailingSlash: false) › parallel routes › should apply the catch-all route to the parallel route if no matching route is found ● parallel-routes-and-interception (trailingSlash: false) › parallel routes › should match the catch-all routes of the more specific path, if there is more than one catch-all route ● parallel-routes-and-interception (trailingSlash: false) › parallel routes › should navigate with a link with prefetch=false ● parallel-routes-and-interception (trailingSlash: false) › parallel routes › should display all parallel route params with useParams ● parallel-routes-and-interception (trailingSlash: false) › parallel routes › should load CSS for a default page that exports another page ● parallel-routes-and-interception (trailingSlash: false) › parallel routes › should handle a loading state ● parallel-routes-and-interception (trailingSlash: false) › route intercepting with dynamic routes › should render intercepted route ● parallel-routes-and-interception (trailingSlash: false) › route intercepting with prerendered dynamic routes › should render intercepted route ● parallel-routes-and-interception (trailingSlash: false) › route intercepting with dynamic optional catch-all routes › should render intercepted route ● parallel-routes-and-interception (trailingSlash: false) › route intercepting with dynamic catch-all routes › should render intercepted route ● parallel-routes-and-interception (trailingSlash: false) › route intercepting › should render intercepted route ● parallel-routes-and-interception (trailingSlash: false) › route intercepting › should render an intercepted route from a slot ● parallel-routes-and-interception (trailingSlash: false) › route intercepting › should render an intercepted route at the top level from a nested path ● parallel-routes-and-interception (trailingSlash: false) › route intercepting › should render intercepted route from a nested route ● parallel-routes-and-interception (trailingSlash: false) › route intercepting › should re-render the layout on the server when it had a default child route ● parallel-routes-and-interception (trailingSlash: false) › route intercepting › should render modal when paired with parallel routes ● parallel-routes-and-interception (trailingSlash: false) › route intercepting › should support intercepting with beforeFiles rewrites ● parallel-routes-and-interception (trailingSlash: false) › route intercepting › should support intercepting local dynamic sibling routes ● parallel-routes-and-interception (trailingSlash: false) › route intercepting › should intercept on routes that contain hyphenated/special dynamic params ● parallel-routes-and-interception (trailingSlash: false) › route intercepting › should not have /default paths in the prerender manifest ● parallel-routes-and-interception-conflicting-pages › should gracefully handle when two page segments match the ● Test suite failed to run |
- Remove 4 unused imports in ipc/protocol.rs (DebugBuildPaths, DefineEnv, DraftModeOptions, WatchOptions) flagged by -D unused-imports - Remove unused HashMap import in ipc/server.rs - Rewrite loop+match as while-let in ipc/client.rs per clippy::while-let-loop - Reformat multi-project.ts with prettier (trailing whitespace) - Fix multi-project unit test: remove explicit @jest/globals import (globals are available via root tsconfig), fix module path to relative reference Co-Authored-By: Claude <noreply@anthropic.com>
Collaborator
Stats from current PR🔴 1 regression
📊 All Metrics📖 Metrics GlossaryDev Server Metrics:
Build Metrics:
Change Thresholds:
⚡ Dev Server
📦 Dev Server (Webpack) (Legacy)📦 Dev Server (Webpack)
⚡ Production Builds
📦 Production Builds (Webpack) (Legacy)📦 Production Builds (Webpack)
📦 Bundle SizesBundle Sizes⚡ TurbopackClient Main Bundles
Server Middleware
Build DetailsBuild Manifests
📦 WebpackClient Main Bundles
Polyfills
Pages
Server Edge SSR
Middleware
Build DetailsBuild Manifests
Build Cache
🔄 Shared (bundler-independent)Runtimes
📎 Tarball URL |
- Add #[allow(dead_code)] to ProjectInstance::remote field and is_remote() method, which are scaffolded for future daemon dispatch wiring - Remove two needless &-borrows on project.ctx() calls (clippy::needless_borrow) - Update next-server-nft inline snapshot to include multi-project.js, which is now compiled into dist/lib/ and correctly traced by standalone output Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What?
Adds scaffolding for running multiple Next.js projects simultaneously with a shared Turbopack daemon process. When two or more
--projectflags are passed tonext devornext build, the CLI:next internal turbopack-daemon <socket>)--turbopack-daemon <socket>CLI syntax
Single-project usage (
next dev/next buildwithout--project) is completely unchanged.Why?
Multi-project (monorepo) setups currently require running separate
next dev/next buildprocesses, each with its own Turbopack instance. Sharing a single Turbopack daemon across projects enables:How?
TypeScript layer
packages/next/src/lib/multi-project.tsparseProjectGroups()argv parser, socket path generation, daemon/worker orchestrationpackages/next/src/cli/internal/turbopack-daemon.tsnext internal turbopack-daemonsubcommandpackages/next/src/bin/next.ts--turbopack-daemonhidden option on dev/build, multi-project detection, daemon subcommand registrationpackages/next/src/cli/next-dev.tsturbopackDaemonoption,NEXT_TURBOPACK_DAEMON_SOCKETenv propagation to forked serverpackages/next/src/cli/next-build.tspackages/next/src/build/swc/types.tsstartTurbopackDaemon,connectTurbopackDaemon,daemonparam oncreateProjectpackages/next/src/build/swc/index.tsloadNative, threadsdaemonthroughcreateProject→projectNewRust IPC layer (in
crates/next-api/src/ipc/)protocol.rsDaemonRequest/DaemonResponse/DaemonResultenums. ReusesProjectOptions,PartialProjectOptions, etc. directly fromnext-api(no Wire* duplication). AddsTurboEngineOptionsandStackFramefor types that don't yet have pure-Rust equivalents.client.rsDaemonClient— connects to socket, length-prefixed framing, async call/subscribe withoneshot/mpscchannelsserver.rsUnixListener/ Windows named pipes), "READY" stdout signaling, per-connection handler, stub request dispatcherdaemon.rsstart_daemon()/connect_daemon()APINAPI layer (in
crates/next-napi-bindings/)start_turbopack_daemon(socket_path)— starts the daemon server (blocks forever)connect_turbopack_daemon(socket_path)— returns opaqueDaemonHandleExternalproject_newgains optionaldaemon: External<DaemonHandle>parameter — when set, short-circuits to IPC instead of full in-process TurboTasks setupProjectInstancerefactored withOptionfields +RemoteProjectstruct to support both local and remote (daemon-backed) projectsturbopack_ctx,container) migrated toctx()/container()accessor methodsCurrent state
The daemon server returns stub responses for most operations — full
ProjectContainerdispatch will be wired up in follow-up PRs. The IPC protocol, framing, client/server connection, and CLI orchestration are complete and functional.Test plan
parseProjectGroupsunit tests (6 tests) —pnpm jest test/e2e/multi-project/multi-project.test.tscargo test -p next-api --lib ipc::protocol::testscargo check -p next-api -p next-napi-bindingscompiles cleanly