Skip to content
BranchesTags

Repository files navigation

Docvion 📄⚡

One schema. Every document parser. Zero lock-in.

PyPI version Docvion CI Python Version License: MIT Ruff

1. Problem

Every document-parsing engine (Docling, Tesseract, PaddleOCR, Azure Document Intelligence, AWS Textract) returns output in its own incompatible shape. A developer building a RAG pipeline or data extraction system against one engine's native output is locked in — switching parsers or comparing accuracy requires rewriting all downstream code.

There has been no neutral, engine-agnostic representation of a parsed document — until now.

2. What Docvion Does

docvion is a schema-first Python library built by Prolixis that:

  1. Defines one canonical document model — DocvionDocument (Pydantic v2)
  2. Adapts any supported engine's raw output into that model
  3. Exports that model to Markdown, JSON, HTML, CSV, or a pandas DataFrame
  4. Chunks it structure-aware (tables never split, sentences never cut, headings preserved)
  5. Scores the quality of any parse with a composite signal
  6. Diffs outputs from two different engines on the same document

Before Docvion — Parser Lock-In

# Locked to Docling's specific dict layout
def build_prompt(docling_json):
    for item in docling_json["texts"]:
        if item["label"] == "title":
            prompt += f"# {item['text']}\n"

After Docvion — Engine-Agnostic

from docvion import parse

doc = parse("invoice.pdf", engine="docling")   # or "tesseract", "paddleocr" — same output shape
prompt = doc.export_to_markdown()              # identical regardless of engine

3. Landscape Comparison

Dimension Docvion Unstructured.io Raw Engine (Docling etc.) LlamaParse
Architecture Schema-First Neutral Layer Document ETL Suite Specific Parser/OCR Cloud Parsing API
Schema Portability High — one DocvionDocument for all engines Medium — custom Element list None — engine-native JSON None — proprietary JSON
Engine Lock-In Zero — swap engines freely Low/Medium 100% locked 100% locked to LlamaCloud
Runs Locally / Air-Gapped ✅ Yes Hybrid ✅ Yes ❌ Cloud required
Base Package Weight Ultra-light (pydantic + psutil only) Heavy (torch, transformers) Heavy (ML models) Light (API client)
Cost Free & MIT Freemium Free & OSS Paid per page
Downstream Reusability Write once, swap engines freely Medium Low — rewrite per engine Low — LlamaIndex ecosystem

4. Installation

# Core (zero heavy dependencies)
pip install docvion

# With specific engines
pip install "docvion[docling]"     # Docling layout + table AI
pip install "docvion[tesseract]"   # pytesseract + Pillow
pip install "docvion[paddleocr]"   # PaddleOCR PP-Structure
pip install "docvion[pandas]"      # pandas DataFrame support for tables
pip install "docvion[all]"         # everything

5. Quickstart

Parse a document

from docvion import parse

doc = parse("report.pdf", engine="docling")

# Export to Markdown (for LLM prompts)
print(doc.export_to_markdown())

# Export to clean HTML (for web/CMS)
print(doc.export_to_html(styled=True))

# Export to JSON (for APIs)
print(doc.export_to_json())

# Export all tables as CSV
csvs = doc.export_to_csv()   # list of CSV strings, one per table

Bring your own engine output

from docvion import convert

doc = convert(engine="docling", raw_output=my_docling_result)
doc = convert(engine="tesseract", raw_output=tesseract_data, image_width=1200, image_height=1600)
doc = convert(engine="paddleocr", raw_output=paddle_result)

Register a custom engine (plugin API)

from docvion import register_adapter, parse

def my_azure_adapter(raw_output, **kwargs):
    # Convert Azure Document Intelligence output -> DocvionDocument
    return DocvionDocument(...)

register_adapter("azure", my_azure_adapter)
doc = parse("invoice.pdf", engine="azure")   # works like any built-in engine

6. Structure-Aware Chunking

Standard chunkers split by raw token count — routinely breaking tables mid-row and cutting sentences in half. SemanticChunker uses the document's own block structure to make smarter decisions:

  • Tables are atomic — never split across chunks
  • Sentences are never cut — boundary protection at sentence level
  • Heading context propagated — every chunk carries the nearest parent heading for retrieval
  • Sliding window overlap — optional overlap_tokens for dense retrieval models
  • Page citations built-in — every chunk knows which pages it spans
from docvion import parse, SemanticChunker

doc = parse("report.pdf", engine="docling")
chunker = SemanticChunker(target_tokens=512, overlap_tokens=64)
chunks = chunker.chunk(doc)

for chunk in chunks:
    print(f"[{chunk.heading_context}] p.{chunk.page_range} → {chunk.token_count} tokens")
    print(chunk.text[:120])

Token counting: Uses a fast character-based approximation (~4 chars/token). For exact counts for a specific model, use that model's tokenizer post-export.

7. Document Quality Scoring

Before ingesting thousands of documents, know which parses are trustworthy and which need a better engine:

q = doc.quality_score()
# {
#   "overall_score": 0.83,
#   "confidence_coverage_pct": 91.0,
#   "avg_confidence": 0.887,
#   "empty_block_ratio": 0.04,
#   "schema_completeness_pct": 78.5,
#   "table_completeness_scores": [0.96, 1.0, 0.72]
# }

Composite weighted score: confidence coverage (30%) + schema completeness (30%) + non-empty ratio (40%).

8. Batch & Async Processing

from docvion import parse_batch, parse_async, DocvionDocument

# Parse 100 files in parallel (thread pool)
docs = parse_batch(
    ["contract_01.pdf", "contract_02.pdf", "invoice.pdf"],
    engine="docling",
    max_workers=4,
    on_error="collect",   # "skip" | "collect" | "raise"
)
valid = [d for d in docs if isinstance(d, DocvionDocument)]

# Async — safe for FastAPI, aiohttp
@app.post("/parse")
async def parse_endpoint(filename: str):
    doc = await parse_async(filename, engine="docling")
    return doc.export_to_json()

9. Engine Comparison (diff_documents)

from docvion import parse, diff_documents

doc_a = parse("report.pdf", engine="docling")
doc_b = parse("report.pdf", engine="tesseract")

diff = diff_documents(doc_a, doc_b)
print(diff["summary"])
# "tesseract vs docling: block delta=+12, table delta=+1,
#  quality delta=+0.043, text overlap=78%. Better quality: tesseract."

10. Table Utilities

block = doc.blocks[3]   # a TABLE block

# Markdown
print(block.table.to_markdown())

# CSV
print(block.table.to_csv())

# HTML (valid <thead>/<tbody> structure)
print(block.table.to_html())

# pandas DataFrame (requires pip install docvion[pandas])
df = block.table.to_dataframe()

# Quality signal
score = block.table.completeness_score()   # 0.0-1.0 ratio of non-empty grid positions

11. CLI Reference

# Parse and export
docvion convert invoice.pdf --engine docling --format markdown
docvion convert invoice.pdf --engine docling --format html -o output.html
docvion convert invoice.pdf --engine docling --format csv -o tables.csv
docvion convert invoice.pdf --engine docling --format json -o output.json

# Structure-aware chunking
docvion chunk report.pdf --engine docling --target-tokens 512 --overlap 64 --format json

# Fast document statistics + quality score
docvion info report.pdf --engine docling

# Benchmark latency, memory, schema completeness across engines
docvion benchmark invoice.pdf --engines docling,tesseract,paddleocr --format markdown

# System health check — engine deps, registered adapters, export formats
docvion doctor

12. Supported Engines & Confidence

Engine Layout Tables BBoxes Confidence
Docling ✅ ✅ ✅ Native block-level scores (0.0–1.0)
Tesseract Paragraphs ❌ ✅ Averaged word-level confidence
PaddleOCR ✅ ✅ HTML ✅ Region detection + text recognition
Custom (via plugin API) Your engine Your engine Your engine Whatever you provide

Transparency note: Docvion never fabricates confidence scores. If an engine doesn't provide them, Block.confidence stays None.

13. Core Schema (DocvionDocument)

All models are Pydantic v2 — strict validation, JSON serialization, full IDE autocomplete.

DocvionDocument
├── metadata: DocumentMetadata
│   ├── source_filename, source_engine, engine_version
│   ├── language, n_pages, processing_time_seconds
├── pages: List[PageMetadata]
│   └── page_number, width, height, rotation
├── blocks: List[Block]
│   ├── id, type (BlockType), text, table (Table)
│   ├── bbox (BoundingBox — normalized 0.0–1.0)
│   ├── confidence, page, order
│   └── language, is_rtl, reading_order_confidence  ← v0.2.0
└── chunks: List[Chunk]
    ├── id, text, block_ids, token_count
    ├── heading_context
    └── page_range, source_heading_id               ← v0.2.0

Block Types:

TITLE · HEADING · PARAGRAPH · TABLE · IMAGE · FIGURE
LIST_ITEM · CAPTION · HEADER · FOOTER · FOOTNOTE · CODE · EQUATION · OTHER

Design principles:

  • Bounding boxes normalized to [0.0, 1.0] — resolution independent
  • Tables stored as flat cell lists with row/col/span indices — handles merged cells
  • schema_version field for forward compatibility

14. Examples

python examples/basic_usage.py         # Quickstart: parse + export
python examples/chunking_demo.py       # SemanticChunker with overlap
python examples/benchmark_usage.py     # Benchmark report generation
python examples/generate_benchmark_charts.py   # Visual Mermaid charts

15. Development & Testing

# Install with dev dependencies
pip install -e ".[dev]"

# Run full test suite (32 tests)
python -m unittest discover tests

# Lint
python -m ruff check .

16. Changelog

See CHANGELOG.md for full version history.

Latest: v0.2.1 — Deep code audit bug fixes (HTML table structure, Tesseract confidence filter, PaddleOCR fallback text, asyncio Python 3.12 compatibility, traceback preservation in batch).

v0.2.0 — HTML/CSV export, quality_score(), parse_batch(), parse_async(), diff_documents(), register_adapter() plugin API, chunk/info CLI subcommands, sliding window overlap chunking, schema enrichments (Block.language, Chunk.page_range, Table.completeness_score(), BlockType.CODE/EQUATION).

17. Third-Party Notices

See THIRD_PARTY_NOTICES.md for full attribution for Docling, Tesseract OCR, PaddleOCR, and Pydantic.

18. Docvion & Contivon

Docvion turns any document into a clean, engine-agnostic structure.
Contivon turns that structure into governed, retrievable memory.

Docvion is the open-source document layer. Contivon is the enterprise memory layer built on top of it.

🏢 Enterprise & Integration Support

Building a RAG pipeline, document AI product, or high-volume ingestion system?

We offer:

  • Custom adapter development — Azure Document AI, AWS Textract, Google Document AI, proprietary engines
  • Integration support — embedding Docvion into existing pipelines
  • Hosted API access — managed parsing endpoint, no infra to run
  • Enterprise licensing — SLA, private support channel, custom feature development

📩 engineering@prolixis.ai

19. License & Company

Maintained by Prolixis (OPC) Pvt Ltd.
Released under the MIT License.

PyPI GitHub Stars

About

One canonical schema for every document parser. Engine-agnostic adapters for Docling, Tesseract, and PaddleOCR — swap OCR/layout engines without rewriting your pipeline. Built for RAG and LLM document ingestion.

pypi.org/project/docvion/

Topics

data-pipelinedoclingdocument-aidocument-intelligencedocument-intelligence-ragdocument-parsinginformation-extractionllmnlpocrpaddleocrpydanticpythonragschemaschema-orgtesseract

Resources

Readme
MIT license

Code of conduct

Code of conduct

Contributing

Contributing

Security policy

Security policy
Activity
Custom properties

Stars

2 stars

Watchers

1 watching

Forks

1 fork
Report repository

Releases

Packages

Contributors

Languages