Skip to content

improvement(file-parsers): parse spreadsheets on a worker thread - #9313

Merged
waleedlatif1 merged 2 commits into
stagingfrom
perf/file-parse-worker
Oct 22, 2026
Merged

waleedlatif1 merged 2 commits into
stagingfrom
perf/file-parse-worker

Conversation

@waleedlatif1

Copy link
Copy Markdown
Collaborator

Summary

  • XLSX.read is synchronous, so every spreadsheet parse through parseBuffer blocked the event loop for the whole parse (1.1–2s on 16–23 MB workbooks on Bun)
  • Spreadsheet buffers (xlsx/xls/xlsm/xlsb/xltx/ods) now parse on a worker_threads pool running the same XlsxParser; output is identical
  • Pool is bounded to one worker (in-process parses were already serialized by the sync parse, so peak workbook memory stays where it is); jobs queue, idle workers terminate after 30s
  • Input bytes are copied into a fresh ArrayBuffer and transferred, so the caller's Buffer is never detached; options cross without the signal
  • Abort: a queued job is dropped, a running job's worker is terminated and the call rejects with signal.reason (same value the in-process path throws)
  • Errors rebuild as the same class (FileParserError with code/message/cause name+message, ZipBombError, ArchiveIntegrityError), so getFileParserErrorCode callers see no change
  • Safe fallback: missing bundle, load failure, startup that never reports ready, or a crash mid-parse disables the pool for the process (one warn) and runs the job in-process exactly as before
  • Enabled only for the production app server: dev/tests parse the live source, Trigger.dev workers (Node, no bundle shipped) stay in-process
  • Bundle: build:xlsx-worker (bun build --target=node, builtins only, ~0.8 MB) runs in bun run build; shipped via Dockerfile COPY (fails the image build if missing) plus an outputFileTracingIncludes entry, resolved from cwd candidates like the isolated-vm worker
  • Bun ignores resourceLimits, so the worker heap counts against the process exactly like an in-process parse; the bound is concurrency, not heap size
  • PDF is left in-process: fix(parsers): yield the event loop during PDF extraction and CSV row coercion #9311 time-slices its loops, and pdf.js needs the native canvas binding inside the worker

Type of Change

  • Improvement

Testing

  • xlsx-worker-pool.test.ts builds the real bundle with bun run build:xlsx-worker and checks in-process parity (preview, complete, text-budget error, archive-guard rejection, corrupt ZIP), caller buffer intact, fallback for a missing bundle / a bundle that throws on load / a worker that exits mid-parse, abort of a running job (frees the slot of a spinning worker) and of a queued job. Each guard was reverted and its test went red
  • Bun 1.4.2 event-loop gap (max timer gap during one parse):
Workbook Mode In-process Worker Output hash
16 MB preview 1112 ms 10 ms identical
16 MB complete 1380 ms 8 ms identical
23 MB preview 1525 ms 10 ms identical
23 MB complete 1934 ms 14 ms identical
  • End to end through parseBuffer with NODE_ENV=production: worker 66 ms gap (cold spawn included); bundle removed → one warn, in-process, same hash; bundle replaced with a throwing file → one warn, in-process, same hash
  • lib/file-parsers/ suite (316 tests), bun run lint, focused tsc on the new modules and test, check:source-conventions, check:utils, check:comment-hygiene, check:file-names, check:test-patterns, check:boundaries

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing (new tests pass the test-audit authoring gate)
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

XLSX.read is synchronous and blocks the event loop for the full parse (about
1.1-2s on 16-23 MB workbooks). Spreadsheet buffers going through parseBuffer
now run on a bounded worker_threads pool (one worker) built from the same
XlsxParser, with an in-process fallback for a missing or broken bundle, worker
startup failure, or crash.
@waleedlatif1
waleedlatif1 requested a review from a team as a code owner October 22, 2026 20:26
@vercel

vercel Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
docs Ready Ready Preview Oct 22, 2026 8:48pm UTC

Request Review

@greptile-apps

greptile-apps Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The latest changes appear safe to merge; no new actionable issues remain.

Summary

Moves spreadsheet parsing to a single worker thread, with in-process parsing when the worker is unavailable.

  • Production spreadsheet buffers are parsed on a worker thread.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  Request[Spreadsheet buffer] --> Enabled{Production app worker available?}
  Enabled -->|No| Fallback[Parse in process]
  Enabled -->|Yes| Queue[Queue job]
  Queue --> Worker[Single spreadsheet worker]
  Worker -->|Result or parser error| Caller[Finish caller request]
  Worker -->|Startup failure or crash| Disable[Disable pool and drain queue]
  Disable --> Fallback
  Fallback --> Caller
  Queue -->|Caller aborts| Cancel[Remove job or terminate its worker]
Loading

Reviews (2) · Last reviewed commit: "fix(file-parsers): drain queued jobs on ..." · Reviewed by Greptile

Comment thread apps/sim/lib/file-parsers/xlsx-worker-pool.ts
Comment thread apps/sim/lib/file-parsers/xlsx-worker-pool.ts

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 9 files

Heads up: you’re close to your flex budget. Increase your flex budget so reviews don’t pause.

Turn on auto-fix | Re-trigger cubic

Comment thread docker/app.Dockerfile Outdated
Comment thread apps/sim/lib/file-parsers/xlsx-worker-pool.ts
Comment thread apps/sim/lib/file-parsers/index.ts
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Oct 22, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 9 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Heads up: you’re close to your flex budget. Increase your flex budget so reviews don’t pause.

Turn on auto-fix | Re-trigger cubic

@waleedlatif1
waleedlatif1 merged commit d0c10bb into staging Oct 22, 2026
65 checks passed
@waleedlatif1
waleedlatif1 deleted the perf/file-parse-worker branch October 22, 2026 21:10

This branch was successfully deployed

1 active deployment
Preview — 7e176925 Deployed Oct 22, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant