Skip to content

feat(devenv/pnpm): per-host shared store + hardlink import (ext4/Linux)#773

Draft
schickling-assistant wants to merge 1 commit into
mainfrom
schickling/2026-06-09-pnpm-shared-store
Draft

feat(devenv/pnpm): per-host shared store + hardlink import (ext4/Linux)#773
schickling-assistant wants to merge 1 commit into
mainfrom
schickling/2026-06-09-pnpm-shared-store

Conversation

@schickling-assistant

@schickling-assistant schickling-assistant commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

Problem

The devenv pnpm store is per-worktree (storeDir: .devenv/pnpm-store-pure-v1) with packageImportMethod: clone-or-copy. On Linux ext4 (no reflink) clone-or-copy degrades to a full copy, so every worktree/checkout of a repo duplicates its entire dependency set on disk — the dominant disk consumer on Linux dev/build hosts. On macOS APFS the same setting is a cheap native reflink, so the problem is Linux/ext4-specific.

Change

Make the pnpm store location + import method a per-host trait (gated on pkgs.stdenv.hostPlatform.isLinux), in the devenv pnpm module (nix/devenv-modules/tasks/shared/pnpm.nix):

  • Linux: one shared content-addressed store at ${PNPM_SHARED_STORE_DIR:-$HOME/.local/share/pnpm/store-shared-v1} + --config.package-import-method=hardlink, so node_modules across all worktrees/sub-workspaces hardlink into a single store (marginal cost per worktree ≈ 0 for shared versions). This generalizes the existing files/-only shared-files symlink into a fully shared store (the now-redundant files-symlink is skipped on Linux).
  • macOS: unchanged — keeps clone-or-copy + the per-worktree store + the shared-files symlink (native reflink is already cheap; no regression).
  • CI stays job-local (the existing [ -z "$CI" ] gating is preserved — this only affects local dev/build hosts).

The committed storeDir:/packageImportMethod: policy keys are not changed; the store path + import method are driven via env/CLI config so the per-host behavior is purely additive.

Validation (representative isolated workspaces incl. a native-build dep, pnpm 11.3.0, ext4)

  • node_modules are hardlinks into the store (link-count 2, shared blob inode).
  • A second fresh worktree adds ~0 KB store delta (cross-worktree content shares inodes).
  • Concurrent installs into the shared store are safe (serialized by the existing store-dir flock); store stays intact.
  • Global virtual store single-instance / TS2742 identity holds — store location is orthogonal to the resolution model (node-linker=hoisted + global virtual store); two packages depending on the same dep resolve to one physical instance.
  • Lifecycle-script packages (e.g. node-pty node-gyp rebuild) are always copy-imported even under hardlink (link-count 1) — an in-place rebuild provably cannot mutate the shared store for other worktrees. Direct mutation test confirmed. No copy-pinning needed.
  • macOS trait verified unchanged via Darwin eval.

Scope / review note

This is the source-of-truth piece in the shared devenv pnpm module, opened in isolation for review. Consumers of this module pick up the per-host trait automatically; a consumer that sets PNPM_STORE_DIR itself will need a matching adjustment, which will follow separately once this is reviewed. The change is self-contained and safe to evaluate standalone.

Cutover note: store-shared-v1 is a fresh path, so the first install per host repopulates it and the old ~/.local/share/pnpm/shared-files blob can be GC'd; existing per-worktree .devenv/pnpm-store* can be cleaned to realize the disk reclaim.

Fleet decision 0002 (2a). On ext4/Linux hosts clone-or-copy degrades to a
full copy (no reflink), duplicating the JS closure per worktree (~750G
fleet-wide). Make store location + import method a per-host trait:

- Linux: one shared content-addressed store under
  $HOME/.local/share/pnpm/store-shared-v1 + package-import-method=hardlink,
  no per-workspace store suffix (pnpm keys projects/ by abs path), and the
  files-only split symlink becomes a no-op (whole store is already shared).
- macOS/APFS: unchanged (per-worktree .devenv store + committed
  clone-or-copy); the trait must not regress mac.

Driven via PNPM_STORE_DIR default + a module-appended
--config.package-import-method=hardlink (after the impurity guard, like
--config.store-dir). Committed storeDir/packageImportMethod in
genie/external.ts are untouched. CI stays job-local and never takes the
hardlink branch.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jun 9, 2026

Copy link
Copy Markdown

Storybook Previews

No storybooks were deployed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants