vfs: add minimal node:vfs subsystem#63115
Conversation
|
Review requested:
|
The docs in this PR claim that you can call |
30f5755 to
779fc37
Compare
Fixed, good spot. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #63115 +/- ##
==========================================
- Coverage 91.94% 90.21% -1.73%
==========================================
Files 362 726 +364
Lines 155999 231034 +75035
Branches 24057 43545 +19488
==========================================
+ Hits 143429 208430 +65001
- Misses 12295 14363 +2068
- Partials 275 8241 +7966
🚀 New features to boost your workflow:
|
|
Can you add a test for #63158? |
Adds the node:vfs builtin module with VirtualFileSystem and provider classes. No integration with fs, modules, or SEA. Assisted-by: Claude-Opus4.7 Signed-off-by: Matteo Collina <hello@matteocollina.com>
Adapts tests that exercised behavior through fs integration so they call the VFS API directly instead. Assisted-by: Claude-Opus4.7 Signed-off-by: Matteo Collina <hello@matteocollina.com>
Cover VirtualDir iteration and disposal, MemoryFileHandle read/write methods via the provider, and the VirtualProvider base class (capability flags, readonly stubs, default implementations). Assisted-by: Claude-Opus4.7 Signed-off-by: Matteo Collina <hello@matteocollina.com>
Covers MemoryProvider, copyFile mode, rm edge cases, hardlinks, bigint read positions, and parent timestamps via the VFS API. Assisted-by: Claude-Opus4.7 Signed-off-by: Matteo Collina <hello@matteocollina.com>
Cover the callback-style async API, additional read/write stream flows, the promises.watch async iterable, and async methods of RealFSProvider. Assisted-by: Claude-Opus4.7 Signed-off-by: Matteo Collina <hello@matteocollina.com>
Removes the unused createEXDEV error helper, adds direct tests for MemoryProvider numeric flags / symlink loops / utimes variants, and adds a base-class VirtualFileHandle test. Assisted-by: Claude-Opus4.7 Signed-off-by: Matteo Collina <hello@matteocollina.com>
Adds targeted tests covering the lazy population, dynamic content provider, readonly-mode, and symlink-traversal paths in MemoryProvider; the path-escape and RealFileHandle EBADF paths in RealFSProvider; the abort/buffer-encoding/recursive watch paths in VFSWatcher; and the empty-file / EBADF fd / explicit-fd-with-start paths in the streams. Assisted-by: Claude-Opus4.7 Signed-off-by: Matteo Collina <hello@matteocollina.com>
Adds direct unit tests for stats default-option paths (including the process.getuid?.() fallback), file-handle base-class branches, the empty-options provider write/append paths, the access-mode permission denials, the watcher closed-state and async-iterable resolver-drain branches, and various RealFSProvider escape and EBADF paths. Brings overall branch coverage from 89% to 95.7%, and stats.js to 100% branch coverage. Assisted-by: Claude-Opus4.7 Signed-off-by: Matteo Collina <hello@matteocollina.com>
Replaces the -coverage / -branches / -misc suffixes with focused files named after the API or behaviour they exercise. Splits the larger multi-topic files into one-topic-per-file. Renames: - callbacks.js → callback-api.js - stats-defaults.js → stats-helpers.js - file-handle-base.js → virtual-file-handle.js - provider-base.js → virtual-provider.js - provider-memory.js → memory-provider.js - real-provider-async.js → real-provider-promises.js - mkdir-recursive-return.js → mkdir.js New files (split out of -coverage/-branches/-misc): - access-modes, create, link, mkdtemp, rename, symlinks, utimes, write-options - memory-file-handle, memory-provider-dynamic, memory-provider-flags - real-provider-handle, real-provider-symlinks, real-provider-watch - stream-errors, stream-explicit-fd - watch, watch-abort-signal, watch-encoding, watch-promises, watch-recursive Coverage maintained at 97.6% line / 95.2% branch / 95.3% function. Assisted-by: Claude-Opus4.7 Signed-off-by: Matteo Collina <hello@matteocollina.com>
Adds an --experimental-vfs runtime option that gates loading of the
node:vfs builtin module, matching the pattern used by node:quic and
node:stream/iter. Without the flag, require('node:vfs') / import
'node:vfs' throw ERR_UNKNOWN_BUILTIN_MODULE.
All VFS test files are updated to pass --experimental-vfs.
Assisted-by: Claude-Opus4.7
Signed-off-by: Matteo Collina <hello@matteocollina.com>
Fixes the JS lint warnings on the VFS subsystem and tests: primordial alphabetical ordering, em-dash → hyphen, error-codes multiline destructuring, removal of unused JSDoc @returns, and the test-side mustSucceed / async-iife-no-unused-result rules. Assisted-by: Claude-Opus4.7 Signed-off-by: Matteo Collina <hello@matteocollina.com>
internal/vfs/fd.js doesn't require any internal/vfs/* modules so there's no circular dependency to defer. Replace the getLazy wrapper with a direct import. Assisted-by: Claude-Opus4.7 Signed-off-by: Matteo Collina <hello@matteocollina.com>
Removes mount/unmount, virtualCwd, overlay mode, fs/module integration sections, SEA usage, and worker-thread guidance since none of that ships in this PR. Assisted-by: Claude-Opus4.7 Signed-off-by: Matteo Collina <hello@matteocollina.com>
Adds 'vfs' to the C++ cannot_be_required list so existing tests (test-code-cache, test-process-get-builtin, test-require-resolve) treat it like other flagged experimental modules. Adds the flag to doc/node.1 and reorders the entry in doc/api/cli.md. Assisted-by: Claude-Opus4.7 Signed-off-by: Matteo Collina <hello@matteocollina.com>
The first block used persistent: false plus setTimeout to trigger the write. The watcher's poll timer was unref'd, so on slow runners the write timer could fire before the first poll and the change event would be missed. Use the same await-once pattern as the other blocks in the file with a content-length change so the size-based stat-change detector always fires. Assisted-by: Claude-Opus4.7 Signed-off-by: Matteo Collina <hello@matteocollina.com>
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
Mirrors the recently-merged handling for node:ffi (see nodejs#63158): when --experimental-vfs is not set, node:vfs is filtered out of the public Module.builtinModules list. Adds matching coverage in test-vfs-flag. Refs: nodejs#63158 Assisted-by: Claude-Opus4.7 Signed-off-by: Matteo Collina <hello@matteocollina.com>
aduh95
left a comment
There was a problem hiding this comment.
Did another pass on the docs
| `readFile`, `writeFile`, `stat`, `lstat`, `readdir`, `realpath`, `readlink`, | ||
| `access`, `open`, `close`, `read`, `write`, `rm`, `fstat`, `truncate`, | ||
| `ftruncate`, `link`, `mkdtemp`, `opendir`. Each takes a Node.js-style | ||
| callback `(err, ...result)`. |
There was a problem hiding this comment.
Without the arrow, I find it a bit confusing
| callback `(err, ...result)`. | |
| callback `(err, ...result) => {}`. |
| <!-- source_link=lib/vfs.js --> | ||
|
|
||
| The `node:vfs` module provides an in-memory virtual file system with an | ||
| `fs`-like API. It is useful for tests, fixtures, embedded assets, and other |
There was a problem hiding this comment.
| `fs`-like API. It is useful for tests, fixtures, embedded assets, and other | |
| `node:fs`-like API. It is useful for tests, fixtures, embedded assets, and other |
| The base class for all VFS providers. Subclasses implement the essential | ||
| primitives (`open`, `stat`, `readdir`, `mkdir`, `rmdir`, `unlink`, | ||
| `rename`, ...) and inherit default implementations of the derived | ||
| methods (`readFile`, `writeFile`, `exists`, `copyFile`, `access`, ...). |
There was a problem hiding this comment.
According to https://learn.microsoft.com/en-us/style-guide/punctuation/ellipses, we should not be using an ellipsis here
| methods (`readFile`, `writeFile`, `exists`, `copyFile`, `access`, ...). | |
| The base class for all VFS providers. Subclasses implement the essential | |
| primitives (such as `open`, `stat`, `readdir`, `mkdir`, `rmdir`, `unlink`, | |
| `rename`, etc.) and inherit default implementations of the derived | |
| methods (such as `readFile`, `writeFile`, `exists`, `copyFile`, `access`, etc.). |
(etc. is also no go according to https://learn.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/e/etc but our own style guide uses it so 🤷)
Line 97 in a159b57
Alternatively, we can list them all, or not at all
| methods (`readFile`, `writeFile`, `exists`, `copyFile`, `access`, ...). | |
| The base class for all VFS providers. |
| added: REPLACEME | ||
| --> | ||
|
|
||
| A provider that wraps a real file system directory and exposes its |
There was a problem hiding this comment.
| A provider that wraps a real file system directory and exposes its | |
| A provider that wraps a directory (i.e. one on the actual file system) and exposes its |
| * `rootPath` {string} The absolute file system path to use as the root. | ||
| Must be a non-empty string. |
There was a problem hiding this comment.
| * `rootPath` {string} The absolute file system path to use as the root. | |
| Must be a non-empty string. | |
| * `rootPath` {string} The absolute file-system path to use as the root. | |
| Must be a non-empty string. |
We should probably document what happens if rootPath doesn't resolve to an existing dir (and maybe there should be an option to auto-create it, but that should be a follow up)
|
|
||
| `true` when the underlying provider is read-only. | ||
|
|
||
| ### File system methods |
There was a problem hiding this comment.
On fs.md, we call it "API", which I find more straight forward
| ### File system methods | |
| ### APIs |
| `VirtualFileSystem` implements the following methods, with the same | ||
| signatures as their [`node:fs`][] counterparts: | ||
|
|
||
| #### Synchronous methods |
There was a problem hiding this comment.
| #### Synchronous methods | |
| #### Synchronous API |
| * Streams: `createReadStream`, `createWriteStream` | ||
| * Watchers: `watch`, `watchFile`, `unwatchFile` | ||
|
|
||
| #### Callback-style asynchronous methods |
There was a problem hiding this comment.
| #### Callback-style asynchronous methods | |
| #### Callback API |
| `ftruncate`, `link`, `mkdtemp`, `opendir`. Each takes a Node.js-style | ||
| callback `(err, ...result)`. | ||
|
|
||
| #### Promise methods |
There was a problem hiding this comment.
| #### Promise methods | |
| #### Promise API |
| /** | ||
| * Gets the real file descriptor. | ||
| * @returns {number} | ||
| */ | ||
| get fd() { | ||
| return this.#fd; | ||
| } | ||
|
|
There was a problem hiding this comment.
This is not documented. Do we actually want to expose it?
| /** | |
| * Gets the real file descriptor. | |
| * @returns {number} | |
| */ | |
| get fd() { | |
| return this.#fd; | |
| } |
| return new Promise((resolve, reject) => { | ||
| fs.read(this.#fd, buffer, offset, length, position, (err, bytesRead) => { | ||
| if (err) reject(err); | ||
| else resolve({ __proto__: null, bytesRead, buffer }); | ||
| }); | ||
| }); | ||
| } |
There was a problem hiding this comment.
It would probably make more sense to reuse FileHandle methods here, I guess this can be a follow up
| if (typeof rootPath !== 'string' || rootPath === '') { | ||
| throw new ERR_INVALID_ARG_VALUE('rootPath', rootPath, 'must be a non-empty string'); | ||
| } | ||
| // Resolve to absolute path and normalize | ||
| this.#rootPath = path.resolve(rootPath); |
There was a problem hiding this comment.
error handling and URL support for free with getValidatedPath
| if (typeof rootPath !== 'string' || rootPath === '') { | |
| throw new ERR_INVALID_ARG_VALUE('rootPath', rootPath, 'must be a non-empty string'); | |
| } | |
| // Resolve to absolute path and normalize | |
| this.#rootPath = path.resolve(rootPath); | |
| // Resolve to absolute path and normalize | |
| this.#rootPath = path.resolve(getValidatedPath(rootPath, 'rootPath')); |
| ObjectDefineProperty(this, 'readonly', { __proto__: null, value: false }); | ||
| ObjectDefineProperty(this, 'supportsSymlinks', { __proto__: null, value: true }); |
There was a problem hiding this comment.
nit: we have a helper for that
| ObjectDefineProperty(this, 'readonly', { __proto__: null, value: false }); | |
| ObjectDefineProperty(this, 'supportsSymlinks', { __proto__: null, value: true }); | |
| setOwnProperty(this, 'readonly', false); | |
| setOwnProperty(this, 'supportsSymlinks', true); |
|
|
||
| [SymbolAsyncIterator]() { | ||
| return this.entries(); | ||
| } | ||
|
|
||
| [SymbolAsyncDispose]() { | ||
| return this.close(); | ||
| } | ||
|
|
||
| [SymbolDispose]() { | ||
| if (!this.#closed) { | ||
| this.closeSync(); | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
nit: we can do like the spec does, like Array.prototype[Symbol.iterator] === Array.prototype.entries
| [SymbolAsyncIterator]() { | |
| return this.entries(); | |
| } | |
| [SymbolAsyncDispose]() { | |
| return this.close(); | |
| } | |
| [SymbolDispose]() { | |
| if (!this.#closed) { | |
| this.closeSync(); | |
| } | |
| } | |
| } | |
| [SymbolDispose]() { | |
| if (!this.#closed) { | |
| this.closeSync(); | |
| } | |
| } | |
| } | |
| VirtualDir.prototype[SymbolAsyncIterator] = VirtualDir.prototype.entries; | |
| VirtualDir.prototype[SymbolAsyncDispose] = VirtualDir.prototype.close; |
|
|
||
| [SymbolAsyncDispose]() { | ||
| return this.close(); | ||
| } | ||
| } |
There was a problem hiding this comment.
| [SymbolAsyncDispose]() { | |
| return this.close(); | |
| } | |
| } | |
| } | |
| VirtualFileHandle.prototype[SymbolAsyncDispose] = VirtualFileHandle.prototype.close; | |
| VirtualFileHandle.prototype[SymbolDispose] = VirtualFileHandle.prototype.closeSync; |
| const dir = { __proto__: Object.getPrototypeOf(root) }; | ||
| Object.assign(dir, { |
There was a problem hiding this comment.
nit: it feels weird to use Object.assign here, do we need it (if so we should add a comment) or?
| const dir = { __proto__: Object.getPrototypeOf(root) }; | |
| Object.assign(dir, { | |
| const dir = { | |
| __proto__: Object.getPrototypeOf(root), |
Adds an experimental
node:vfsbuiltin (gated behind--experimental-vfs) withVirtualFileSystem,VirtualProvider,MemoryProvider, andRealFSProvider. No integration withnode:fs, the module loader, or SEA those are intended to land in follow-up PRs.Extracted from: #61478
Approximate line counts: code ~4k / docs ~1k / tests ~5k — total ~10k lines, with tests being the largest share.