Skip to content

feat(backend): add extract-text node for PDF/DOCX documents#29

Open
LukasHirt wants to merge 3 commits into
mainfrom
feat/extract-text-node
Open

feat(backend): add extract-text node for PDF/DOCX documents#29
LukasHirt wants to merge 3 commits into
mainfrom
feat/extract-text-node

Conversation

@LukasHirt

Copy link
Copy Markdown
Collaborator

Summary

Implements the "Extract Text" node (previously just a design proposal in docs/superpowers/specs/2026-07-24-new-workflow-node-types-design.md) end to end.

Bug this fixes: Executor.Run (backend/pkg/executor/executor.go) has always done vars["file.content"] = string(content) unconditionally for every file type. That's correct for .txt/.md, but for a PDF or .docx, an LLM Prompt node referencing {{file.content}} silently got that binary format's raw bytes stringified instead of the document's actual text — a real, user-facing gap for a perfectly reasonable "summarize this uploaded PDF" workflow.

What's in scope: plain-document text extraction (PDF and DOCX). OCR is explicitly out of scope — recovering text from an image (a scanned PDF, a photo of a page) needs either a bundled OCR engine or a call out to an external service, a meaningfully bigger lift than parsing an already-textual document format. That remains a natural, separate follow-up; nothing here attempts it.

Backend (backend/pkg/textextract, backend/pkg/executor)

  • New textextract package: DetectFormat (by extension, falling back to content-sniffing magic bytes/zip structure) and Extract, which converts PDF/DOCX to plain text and passes through anything else (including plain text) unchanged — no behavior change for existing .txt/.md workflows.
  • PDF parsing uses github.com/ledongthuc/pdf — pure Go, BSD-3-Clause (a maintained fork of the Go team's own rsc.io/pdf), no cgo, license compatible with this project's existing dependencies. License text and NOTICE.md/REUSE.toml entries added; reuse lint passes.
  • DOCX parsing uses only the standard library (archive/zip + encoding/xml) — a .docx is just a zip with an XML part, and plain-text extraction needs nothing beyond unzip-and-walk-the-XML, so no new dependency was needed for that half.
  • New executor node kind extractText (runExtractText, parallel to runLLM/runAction): reads the same file.content/file.name vars Run already loads, extracts text, and writes it to vars["file.text"] by default or node.Data["outputVariable"] when set. result.Output gets a short "N characters extracted" summary (per spec, not the full text — unlike llm.output, which can be arbitrarily large here).

Frontend

  • Modeled as a new CanvasNodeKind ('extractText'), not a new actionType. Rationale: it has no "target file" side effect the way tag/move/comment/etc. do — it's a content-transform step, like the LLM Prompt node. The LLM node is already precedent in this codebase for a single-purpose top-level kind with no sub-types, so this keeps the picker/canvas/NDV wiring consistent with how llm already works instead of overloading the action-node switch.
  • New AI-category picker entry (file-text icon), a new canvas node component (ExtractTextNode.vue, mirroring LlmNode.vue), and the optional output-variable-override field wired into NodeDetailsPanel.vue.
  • Reuses the existing outputVariable field already declared (but unused) on WorkflowNodeData rather than adding a new field — same name, same shape, for consistency. (The LLM node's own NDV doesn't read it yet — this PR doesn't touch that, only extractText does.)

Test plan

  • Backend: TDD — wrote pkg/textextract tests first (confirmed they failed to compile with no implementation), then implemented DetectFormat/Extract/PDF/DOCX extraction. Tests build minimal-but-valid PDF/DOCX fixtures in-process (real byte-accurate PDF with xref table; real DOCX zip with word/document.xml) rather than checking in binary fixtures.
  • Backend: extended pkg/executor/executor_test.go with runExtractText tests — default output var, custom output var, real end-to-end PDF extraction through the executor, plain-text passthrough, and failure propagation on a corrupt document (execution stops, downstream llm node doesn't run).
  • cd backend && go build ./... && go vet ./... && go test ./... — all green.
  • Frontend: added tests/unit/nodeTypes.spec.ts (Extract Text registered in NODE_TYPES under AI_CATEGORY) and tests/unit/NodeDetailsPanel.spec.ts (output-variable field renders and emits correctly), confirmed both fail before implementation, pass after.
  • pnpm test:unit, pnpm check:types, pnpm lint, pnpm build — all green.
  • reuse lint — compliant (added LICENSES/BSD-3-Clause.txt, NOTICE.md/REUSE.toml entries for ledongthuc/pdf).

🤖 Generated with Claude Code

@LukasHirt
LukasHirt requested a review from a team as a code owner July 24, 2026 16:29
@LukasHirt LukasHirt self-assigned this Jul 24, 2026
Pure-Go, BSD-3-Clause-licensed PDF parser needed by the upcoming
Extract Text node's PDF support. DOCX extraction needs no new
dependency (archive/zip + encoding/xml, both stdlib, are enough for
that format), so this is the only new module required.

Signed-off-by: Lukas Hirt <info@hirt.cz>
Executor.Run has always done vars["file.content"] = string(content)
unconditionally, for every file type. That's correct for .txt/.md, but
for a PDF or .docx an LLM Prompt node referencing {{file.content}}
silently got that binary format's raw bytes stringified instead of the
document's actual text.

Add a new "extractText" node kind (pkg/textextract) that detects a
file's format by extension, falling back to content sniffing, and
converts PDF/DOCX into plain text. Plain-text and unrecognized formats
pass through unchanged. Wired into the executor as runExtractText,
writing to vars["file.text"] by default or the node's configured
outputVariable, mirroring the pattern where llm.output is also written
into result.Output.

Image OCR (scanned documents/photos) is explicitly out of scope: it
needs a bundled OCR engine or an external service call, a meaningfully
bigger lift than parsing an already-textual document format. Left as a
natural follow-up.

Signed-off-by: Lukas Hirt <info@hirt.cz>
Modeled as a new CanvasNodeKind ('extractText') rather than a new
actionType: it has no meaningful "target file" side effect the way
tag/move/comment/etc. do, it's a content-transform step like the LLM
Prompt node, and the LLM node itself is already precedent for a
single-purpose top-level kind with no sub-types. This keeps the
picker, canvas node component, and NDV wiring consistent with how llm
already works instead of overloading the action-node switch.

Adds an AI-category picker entry, a canvas node component
(ExtractTextNode.vue, mirroring LlmNode.vue), and reuses the existing
(previously unused) outputVariable field on WorkflowNodeData for the
optional output-variable override, wired into NodeDetailsPanel's NDV.

Signed-off-by: Lukas Hirt <info@hirt.cz>
@LukasHirt
LukasHirt force-pushed the feat/extract-text-node branch from 1ccbcf9 to 32a191c Compare July 24, 2026 21:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant