feat(devenv/pnpm): per-host shared store + hardlink import (ext4/Linux)#773
Draft
schickling-assistant wants to merge 1 commit into
Draft
feat(devenv/pnpm): per-host shared store + hardlink import (ext4/Linux)#773schickling-assistant wants to merge 1 commit into
schickling-assistant wants to merge 1 commit into
Conversation
Fleet decision 0002 (2a). On ext4/Linux hosts clone-or-copy degrades to a full copy (no reflink), duplicating the JS closure per worktree (~750G fleet-wide). Make store location + import method a per-host trait: - Linux: one shared content-addressed store under $HOME/.local/share/pnpm/store-shared-v1 + package-import-method=hardlink, no per-workspace store suffix (pnpm keys projects/ by abs path), and the files-only split symlink becomes a no-op (whole store is already shared). - macOS/APFS: unchanged (per-worktree .devenv store + committed clone-or-copy); the trait must not regress mac. Driven via PNPM_STORE_DIR default + a module-appended --config.package-import-method=hardlink (after the impurity guard, like --config.store-dir). Committed storeDir/packageImportMethod in genie/external.ts are untouched. CI stays job-local and never takes the hardlink branch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Storybook PreviewsNo storybooks were deployed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The devenv pnpm store is per-worktree (
storeDir: .devenv/pnpm-store-pure-v1) withpackageImportMethod: clone-or-copy. On Linux ext4 (no reflink)clone-or-copydegrades to a full copy, so every worktree/checkout of a repo duplicates its entire dependency set on disk — the dominant disk consumer on Linux dev/build hosts. On macOS APFS the same setting is a cheap native reflink, so the problem is Linux/ext4-specific.Change
Make the pnpm store location + import method a per-host trait (gated on
pkgs.stdenv.hostPlatform.isLinux), in the devenv pnpm module (nix/devenv-modules/tasks/shared/pnpm.nix):${PNPM_SHARED_STORE_DIR:-$HOME/.local/share/pnpm/store-shared-v1}+--config.package-import-method=hardlink, sonode_modulesacross all worktrees/sub-workspaces hardlink into a single store (marginal cost per worktree ≈ 0 for shared versions). This generalizes the existingfiles/-only shared-files symlink into a fully shared store (the now-redundant files-symlink is skipped on Linux).clone-or-copy+ the per-worktree store + the shared-files symlink (native reflink is already cheap; no regression).[ -z "$CI" ]gating is preserved — this only affects local dev/build hosts).The committed
storeDir:/packageImportMethod:policy keys are not changed; the store path + import method are driven via env/CLI config so the per-host behavior is purely additive.Validation (representative isolated workspaces incl. a native-build dep, pnpm 11.3.0, ext4)
node_modulesare hardlinks into the store (link-count 2, shared blob inode).flock); store stays intact.TS2742identity holds — store location is orthogonal to the resolution model (node-linker=hoisted+ global virtual store); two packages depending on the same dep resolve to one physical instance.node-gyp rebuild) are always copy-imported even underhardlink(link-count 1) — an in-place rebuild provably cannot mutate the shared store for other worktrees. Direct mutation test confirmed. No copy-pinning needed.Scope / review note
This is the source-of-truth piece in the shared devenv pnpm module, opened in isolation for review. Consumers of this module pick up the per-host trait automatically; a consumer that sets
PNPM_STORE_DIRitself will need a matching adjustment, which will follow separately once this is reviewed. The change is self-contained and safe to evaluate standalone.Cutover note:
store-shared-v1is a fresh path, so the first install per host repopulates it and the old~/.local/share/pnpm/shared-filesblob can be GC'd; existing per-worktree.devenv/pnpm-store*can be cleaned to realize the disk reclaim.