feat(components): add MinerU Document Loader with flash and precision modes#6063
feat(components): add MinerU Document Loader with flash and precision modes#6063chaserRen wants to merge 4 commits intoFlowiseAI:mainfrom
Conversation
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces a new MinerU Document Loader into Flowise, significantly expanding its document processing capabilities. Users can now leverage MinerU's powerful API to extract content from a wide array of document formats, choosing between a rapid, token-free 'Flash' mode or a more detailed, token-required 'Precision' mode with advanced configuration options. This integration streamlines the ingestion of complex documents into Flowise workflows, providing greater flexibility and accuracy in data extraction. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new MinerU Document Loader, enabling document parsing in both 'flash' and 'precision' modes, supporting file uploads and URL inputs for various document types. The review identifies a high-severity security concern due to the custom ZIP parsing implementation, recommending the use of a robust, battle-tested library. Additionally, it suggests improving performance by parallelizing task processing and addressing the non-deterministic generation of filenames.
| const results: MinerUTaskResult[] = [] | ||
| for (const task of tasks) { | ||
| const result = await this.runTask(task, config) | ||
| results.push(result) | ||
| } |
There was a problem hiding this comment.
The current implementation processes tasks sequentially using a for...of loop. This can be inefficient when there are multiple files or URLs to process. To improve performance, you can execute these tasks in parallel using Promise.all.
| const results: MinerUTaskResult[] = [] | |
| for (const task of tasks) { | |
| const result = await this.runTask(task, config) | |
| results.push(result) | |
| } | |
| const results: MinerUTaskResult[] = await Promise.all(tasks.map((task) => this.runTask(task, config))) |
|
|
||
| const mimeMatch = contentPart.match(/^data:([^;]+);base64,/i) | ||
| const ext = mimeMatch?.[1] ? MIME_EXTENSION_MAP[mimeMatch[1].toLowerCase()] || 'bin' : 'bin' | ||
| const guessedName = fileNameFromPayload || `upload_${Date.now()}_${index}.${ext}` |
There was a problem hiding this comment.
Using Date.now() to generate filenames introduces non-determinism, which can make testing more difficult and less reliable. While collisions are unlikely, it's a good practice to use a more deterministic approach. Consider using a cryptographic hash of the file content for a unique and deterministic name, or a simpler counter if the scope is limited.
|
Pushed fixes for the MinerU review feedback in MinerU.ts:
Additional:
|
1 participant
Summary
Add MinerU Document Loader to Flowise for parsing documents via MinerU APIs in flash (token-free) and precision (token-required) modes.
Changes
Add new node implementation at packages/components/nodes/documentloaders/MinerU/MinerU.ts and icon at packages/components/nodes/documentloaders/MinerU/mineru.svg
Support two input modes: URL and file upload
Support MinerU flash and precision workflows
Support precision options: model (vlm/pipeline/html), OCR, formula, table, language, page range, timeout
Support split-pages behavior for PDF sources when page range is provided
Return Flowise Document/Text outputs with metadata (source, mode, language, model, page info)
Add env fallbacks for configuration: MINERU_TOKEN, MINERU_FLASH_BASE_URL, MINERU_API_BASE_URL, MINERU_SOURCE_HEADER
Include polling, timeout, and explicit error handling for MinerU task lifecycle
Why
This integration enables Flowise users to parse PDF/image/office/html documents with MinerU directly in Document Loaders, covering both fast extraction and higher-accuracy extraction
scenarios.
Testing
Confirmed branch diff only adds MinerU loader source and icon files
Verified code paths enforce mode-specific token/file-type requirements and produce clear errors for invalid inputs/timeouts
Verified node outputs and metadata mapping logic for both flash and precision modes