tokimo-package-fileparser

Pure-Rust unified file parser: PDF / DOCX / XLSX / PPTX / DOC / XLS / PPT / TXT / CSV / JSON → Markdown, with embedded images extracted to disk.

Features

• Pure Rust, no C/C++ deps — no pdfium, no libreoffice, no *-sys crates. Builds cleanly on any rustc target with no system libraries.
• Unified API — one entry point parse(input, out_dir). The parser dispatches on file extension and writes a folder <out_dir>/<input-stem>/ containing per-page / per-sheet / per-slide Markdown plus an images/ directory.
• Per-page splitting
  • PDF → page-001.md, page-002.md, …
  • XLSX / XLSM → one section-NNN-<sheet>.md per worksheet
  • PPTX → one section-NNN-<slide-title>.md per slide
  • DOCX → splits on <w:br w:type="page"/> page breaks (the common Word page break). Falls back to H1-based chapter split, then to a fixed-element cap so even unstructured long documents don't collapse into one file.
• Embedded asset extraction
  • OOXML (DOCX/XLSX/PPTX):
    • Images under */media/*.{png,jpg,gif,bmp,tif,webp,emf,wmf,svg} → images/
    • Other media (audio/video) under */media/* → media/
    • Embedded objects (e.g. xl/embeddings/oleObject1.bin, embedded xlsx-in-docx) → embeddings/
  • PDF: every Subtype=Image XObject is decoded (DCTDecode → .jpg, JPXDecode → .jp2, raster DeviceGray/DeviceRGB → re-encoded .png) into images/. PDF /EF file attachments (embedded files) are extracted with their original filenames into embeddings/.
  • Each non-empty bucket gets a sibling images.md / media.md / embeddings.md listing for easy review.
• CJK / Chinese support — text files auto-detect UTF-8 / GBK / UTF-16, and properly-tagged PDFs (Word / WeasyPrint / Acrobat output) extract mixed Chinese + English without issue.
• Structured return value — parse() returns a ParseOutput { dir, files, tree } that serialises directly to JSON, with both a recursive files tree ({type, name, path, size, children}) and a human-readable tree(1)-style string.

Supported formats

Extension	Splitting	Notes
.pdf	per page	pdf-extract + lopdf for images
.docx, .doc	per page break (with fallbacks)	office_oxide IR
.xlsx, .xlsm, .xls	per worksheet	office_oxide IR
.pptx, .ppt	per slide	office_oxide IR
.csv, .tsv	single Markdown table
.txt, .log	opt-in via --features text	off by default
.htm, .html	opt-in via --features html	off by default
.json	opt-in via --features json	off by default
.zip, .tar, .tar.gz/.tgz, .tar.bz2/.tbz2, .tar.xz/.txz, .tar.zst/.tzst, .7z, .rar, .gz, .bz2, .xz, .zst	fully extracted into <out_dir>/<stem>/ (default cap 1 GiB, override via ParseOptions::max_archive_bytes)	via tokimo-universal-archiver

Anything else returns ParseError::UnsupportedExtension.

Cargo features

Feature	Default	Purpose
text	off	Routes .txt/.log through a plain-text Markdown extractor. Off by default. With the feature disabled, text inputs return ParseError::UnsupportedExtension.
html	off	Routes .htm/.html through a tag-stripping Markdown extractor. Off by default — most callers should not be parsing arbitrary HTML through a document parser. With the feature disabled, HTML inputs return ParseError::UnsupportedExtension.
json	off	Routes .json through a fenced code block extractor. Off by default. With the feature disabled, JSON inputs return ParseError::UnsupportedExtension.

Usage

use tokimo_package_fileparser::parse;

let output = parse("report.pdf", "out/")?;

println!("{}", output.dir.display());
//   out/report
println!("{}", output.tree);
//   report
//   ├── images/
//   │   ├── img-001.png  (120.6 KB)
//   │   └── img-002.jpg  (96.5 KB)
//   ├── page-001.md  (4.2 KB, 87 lines)
//   ├── page-002.md  (5.5 KB, 102 lines)
//   └── …

println!("{}", output.to_json_pretty());
// { "dir": "...", "files": { "type": "dir", "children": [ ... ] }, "tree": "..." }

CLI example (cargo run --example parse):

cargo run --example parse -- attention.pdf ./out
# prints JSON describing ./out/attention/{page-001.md, ..., images/img-001.png}

Output layout

<out_dir>/<input-stem>/
├── page-001.md            (PDF / DOCX-with-page-breaks)
├── page-002.md
├── section-001-<title>.md (XLSX sheets / PPTX slides / DOCX sections)
├── content.md             (txt/csv/json)
├── images.md              (gallery — only when images were extracted)
├── media.md               (audio/video listing — OOXML only, when present)
├── embeddings.md          (embedded objects/attachments — when present)
├── images/
│   ├── img-001.png
│   └── img-002.jpg
├── media/
│   └── media-001.mp3
└── embeddings/
    └── embed-001-oleObject1.bin

Building

cargo build
cargo test

Fixture documents under tests/fixtures/ are generated by tests/gen_fixtures.py (requires Python with python-docx, openpyxl, python-pptx, weasyprint, plus a CJK font such as WenQuanYi Zen Hei or Droid Sans Fallback).

License

MIT OR Apache-2.0