zlib, vfs: add ZIP archive support and an archive vfs provider#64339
zlib, vfs: add ZIP archive support and an archive vfs provider#64339pipobscure wants to merge 1 commit into
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #64339 +/- ##
==========================================
+ Coverage 90.23% 90.31% +0.07%
==========================================
Files 741 743 +2
Lines 240979 243856 +2877
Branches 45401 46181 +780
==========================================
+ Hits 217449 220236 +2787
- Misses 15112 15172 +60
- Partials 8418 8448 +30
🚀 New features to boost your workflow:
|
|
Just to show where this is coming from and where I'm going with this. pipobscure#3 is a follow on to handle resolving inside a vfs/archive. And based on both of those this is how one could nicely build SEA apps and/or distribute bundled apps. |
0f2b879 to
4258e94
Compare
Codecov flagged low patch coverage on lib/internal/zip.js and lib/internal/vfs/providers/archive.js in nodejs#64339. Add tests exercising Zip64 extra-field parsing, DOS date/time edge cases, streaming-entry state guards, decodeMemberStream/decodeMemberSync's duplicated error branches, the ZipBuffer/ZipFile iteration protocols, several on-disk ZipFile error paths, and ArchiveFileHandle's direct read/write/stat/ truncate surface plus a handful of provider-level error branches the existing tests didn't reach.
2a56a15 to
27ab6b4
Compare
|
See also #45651 |
Thanks @bakkot !!! I think the time has come for it on the one hand, and on the other I added some „motivation“ links earlier. Here some more detail: Based on this, we can modify the loader to directly load from an archive. If we do that, we get application bundles. pipobscure#3 Which can then in turn be used to easily create SEA apps. https://github.com/pipobscure/experimental-sea So the world had changed enough that it‘s worth proposing again. |
Add ZIP archive read/write support to zlib, plus a VFS provider that
mounts a ZIP archive as a browsable, read/write file system.
zlib gains three classes and a pair of helper functions:
* ZipEntry represents a single archive member: name, metadata, and
content, created from a Buffer/stream (`create()`/`createStream()`)
or read back out of raw entry bytes (`read()`). It is otherwise
immutable; every accessor and static creator has a synchronous and
an asynchronous form.
* ZipFile wraps a real ZIP file on disk (`open()`/`openSync()`),
exposing get/add/delete/stream by name plus `compact()` to reclaim
space left behind by deleted entries. Opened read-only unless
`{ writable: true }` is passed.
* ZipBuffer is the equivalent fully in-memory representation,
serializable back to a Buffer with `toBuffer()`/`toBufferSync()`.
Always writable, since nothing is written until asked to serialize.
* `createZipArchive()`/`createZipArchiveSync()` build a brand-new
archive (as a stream or a Buffer) from a list of entries, returned
as a `Readable` that can be piped directly. Their trailing
comment/options arguments collapse into a single optional argument -
either a plain string (an archive comment) or an options object with
`comment` and `baseOffset` fields; `ZipBuffer`'s `toBuffer()`/
`toBufferSync()` accept the same shape. `baseOffset` seeds the
offsets an archive records, so it can be made self-describing even
when it will end up somewhere other than byte 0 of its eventual file
- for example, appended after a shebang line or other prefix already
written to the same output.
Every read path enforces `getMaxZipContentSize()`/
`setMaxZipContentSize()` limits and rejects malformed local/central
directory records, guarding against zip-bomb and corrupt-archive
inputs. New ERR_ZIP_* error codes cover corrupt entries, missing
entries, oversized entries, invalid archives, non-writable archives,
and unsupported ZIP features.
node:vfs gains ZipProvider, a VirtualProvider backed by an
already-open ZipBuffer or ZipFile: mounting one exposes the archive's
entries through the same fs-like API any other VFS provider offers.
Directories are inferred from entry-name prefixes rather than stored
explicitly. Because a ZIP member can only be written or read in full,
never edited in place, a file opened for writing is buffered and only
committed as a new archive entry when its handle is closed. The
provider's readonly flag mirrors the underlying archive's own
writable state.
Signed-off-by: Philipp Dunkel <pip@pipobscure.com>
Summary
node:zlib— ZIP archive support (ZipEntry,ZipFile,ZipBuffer)ZipEntry: reads and builds individual archive members, supportingstore,deflate, andzstdcompression methods. Includes a streaming variant (createStream()/contentStream()) for incremental production/consumption without buffering a whole member in memory.ZipBuffer: an in-memory, always-writable view over an archive buffer. Entries can be added, replaced, or deleted;toBuffer()serializes the current live set into a fresh archive with a new central directory.ZipFile: a disk-backed archive, read-only by default and writable via{ writable: true }. Adding an entry appends it where the central directory used to be and rewrites the central directory in place; deleting rewrites the central directory only (no growth).compact()streams a fresh archive with dead entries removed, without touching the still-open file.*Synccounterpart, documented as event-loop-blocking (mirroringnode:fs's own wording). An internal busy-flag guard prevents a sync call from racing an in-flight async mutation on the sameZipFile.ERR_ZIP_ENTRY_CORRUPT,ERR_ZIP_ENTRY_NOT_FOUND,ERR_ZIP_ENTRY_TOO_LARGE,ERR_ZIP_INVALID_ARCHIVE,ERR_ZIP_NOT_WRITABLE,ERR_ZIP_UNSUPPORTED_FEATURE.zlib.getMaxZipContentSize()/setMaxZipContentSize()bound decompressed entry size by default, to guard against zip-bomb-style entries.content(),contentSync(), andcontentStream().node:vfs—ArchiveProviderZipBufferorZipFileas a virtual file system.provider.readonlyreflects the underlying archive's own writability./) and implicitly (any entry prefixed"<dir>/").openSync,statSync,readdirSync,readFileSync/writeFileSync,mkdirSync,rmdirSync,unlinkSync,renameSync, ...), backed byZipBuffer/ZipFile's own sync surface and documented as event-loop-blocking.Test plan
test/parallel/test-zlib-zip*.js,test/pummel/test-zlib-zip-slow.js—ZipEntry/ZipFile/ZipBuffercreation, round-tripping, writability, sync parity, zstd, fuzzing, hardening (encrypted entries, unsupported methods, truncated/garbage archives), zip64 boundaries, metadata, and property-based tests.test/parallel/test-vfs-archive-provider.js— construction validation, stat/readdir over explicit and implicit directories, full async and sync CRUD against both aZipBufferand aZipFilesource, read-only enforcement (EROFS).node:zlib/node:vfsparallel suite run against a local build (171 tests) to confirm no regressions.P.S.: my CLA should be on file and I wrote this myself so COO is declared