readers (txt, url, wikipedia)

Author: telmomenezes

CHANGELOG.md

@@ -3,9 +3,8 @@
 ## [0.8.1] - work in progress
 
 ### Added
-- cli interface.
-- repl cli entry point.
-- parsers cli entry point.
+- readers (txt, url, wikipedia).
+- cli interface with repl, parsers, readers.
 - hyperedge.Hyperedge.match function (calls parsers.match_pattern).
 
 ### Changed

docs/manual/readers.md

@@ -0,0 +1,140 @@
+# Readers
+
+The `hyperbase.readers` module provides a way to read and parse text from various sources directly into Semantic Hypergraphs. A reader handles the extraction and segmentation of text into paragraph-sized blocks, which can then be fed to a parser. Hyperbase ships with three built-in readers -- plain text files, URLs and Wikipedia articles -- and provides a registration mechanism for custom readers.
+
+## Reading and parsing sources
+
+The preferred way to read and parse a source is through the `Parser` methods `read_source()` and `read_source_to_jsonl()`. These handle reader selection automatically, so you only need a parser instance:
+
+```python
+from hyperbase import get_parser
+
+parser = get_parser("generative")
+
+# Iterate over parse results block by block
+for results in parser.read_source("article.txt"):
+    for result in results:
+        print(result["edge"])
+
+# Or write everything to a JSONL file in one call
+parser.read_source_to_jsonl("article.txt", "output.jsonl", progress=True)
+```
+
+Both methods accept an optional `reader` argument to force a specific reader instead of auto-detection:
+
+```python
+# Force the generic URL reader on a Wikipedia link
+for results in parser.read_source(
+    "https://en.wikipedia.org/wiki/Hypergraph", reader="url"
+):
+    ...
+```
+
+### Extracting raw text (no parsing)
+
+To extract text blocks without parsing, use a reader directly via `get_reader()` (available from `hyperbase.readers`):
+
+```python
+from hyperbase.readers import get_reader
+
+reader = get_reader("article.txt")
+
+# Iterate over text blocks
+for block in reader.read("article.txt"):
+    print(block)
+
+# Or write blocks to a plain text file
+reader.read_to_text("article.txt", "output.txt", progress=True)
+```
+
+When a named reader is given, the source is not required to obtain the reader instance:
+
+```python
+reader = get_reader(reader="wikipedia")
+```
+
+Either `source` or a named `reader` must be provided -- calling `get_reader()` with neither raises a `ValueError`.
+
+## CLI
+
+The `hyperbase read` command provides a convenient way to read and parse sources from the command line:
+
+```bash
+# Parse a local file to JSONL
+hyperbase read article.txt -o output.jsonl
+
+# Extract raw text blocks (no parsing)
+hyperbase read article.txt -o output.txt
+
+# Parse a Wikipedia article
+hyperbase read https://en.wikipedia.org/wiki/Hypergraph -o output.jsonl
+
+# Specify reader and parser explicitly
+hyperbase read source.txt -o output.jsonl --reader plain_text --parser alphabeta --language en
+```
+
+## Built-in readers
+
+### `plain_text`
+
+Reads local text files. Accepts any source that is a valid file path. The text is split into paragraph-sized blocks: if blank lines are found, they are used as paragraph separators; otherwise each line becomes its own block.
+
+### `url`
+
+Reads web pages via HTTP/HTTPS. Uses [trafilatura](https://trafilatura.readthedocs.io/) to extract the main text content from the HTML, stripping navigation, ads and other boilerplate.
+
+### `wikipedia`
+
+Reads Wikipedia articles directly from the MediaWiki API. Accepts any URL matching `*.wikipedia.org/wiki/*`. It parses the wikicode markup to extract clean text, organized by section. Boilerplate sections (e.g. "References", "See also", "External links") are automatically discarded based on the article language. The Wikipedia reader declares `url` as `more_general`, so it takes priority over the URL reader when the source is a Wikipedia link.
+
+## Auto-detection
+
+When a reader is not explicitly specified, all registered readers are checked and those whose `accepts()` method returns `True` for the given source are collected. If more than one reader matches, the `more_general` mechanism is used to pick the most specific one. For example, a Wikipedia URL is accepted by both the `url` and `wikipedia` readers, but because `WikipediaReader` declares `more_general = ['url']`, the Wikipedia reader is selected.
+
+## Custom readers
+
+You can create and register your own readers. A custom reader must subclass `Reader` and implement two methods:
+
+- `accepts(source)` -- a static method that returns `True` if the reader can handle the given source string.
+- `read(source)` -- a generator that yields text blocks from the source.
+
+Optionally, you can implement `block_count(source)` to return the total number of blocks (enabling progress bars), and set the `more_general` class attribute to declare that this reader is more specific than others.
+
+Here is an example:
+
+```python
+from hyperbase.readers import Reader, register_reader
+
+class RSSReader(Reader):
+    more_general = ['url']  # take priority over the generic URL reader
+
+    @staticmethod
+    def accepts(source: str) -> bool:
+        return source.endswith('.rss') or source.endswith('/feed')
+
+    def read(self, source: str):
+        import feedparser
+        feed = feedparser.parse(source)
+        for entry in feed.entries:
+            # yield the text content of each entry as a block
+            yield entry.get('summary', '')
+
+register_reader('rss', RSSReader)
+```
+
+After registration, the new reader is automatically considered during auto-detection. It can also be requested by name:
+
+```python
+parser.read_source_to_jsonl("https://example.com/feed", "feed.jsonl", reader="rss")
+```
+
+## Listing registered readers
+
+To see all currently registered readers:
+
+```python
+from hyperbase.readers import list_readers
+
+for name, cls in list_readers().items():
+    print(f"{name}: {cls.__name__}")
+```

mkdocs.yml

@@ -67,6 +67,7 @@ nav:
     - Hyperedge Operations: manual/hyperedge-operations.md
     - Patterns: manual/patterns.md
     - Parsing: manual/parsing.md
+    - Readers: manual/readers.md
     - API Reference: manual/api.md
   - Authors: authors.md
 

pyproject.toml

@@ -30,6 +30,8 @@ dependencies = [
     "tqdm>=4.65.0",
     "prompt-toolkit>=3.0.0",
     "rich>=13.0.0",
+    "mwparserfromhell>=0.7.2",
+    "trafilatura>=2.0.0",
 ]
 
 [project.scripts]

src/hyperbase/cli/__init__.py

@@ -15,6 +15,59 @@ def main():
         help="List installed parser plugins",
     )
 
+    # --- read subcommand ---------------------------------------------------
+    read_parser = subparsers.add_parser(
+        "read",
+        help="Read a source, parse it, and write JSONL output",
+    )
+    read_parser.add_argument(
+        "source",
+        type=str,
+        help="Source to read (file path or URL)",
+    )
+    read_parser.add_argument(
+        "-o", "--output",
+        type=str,
+        required=True,
+        help="Output file path (.jsonl for parsed output, .txt for raw text)",
+    )
+    read_parser.add_argument(
+        "--parser",
+        type=str,
+        default="generative",
+        help="Parser plugin name (default: generative)",
+    )
+    read_parser.add_argument(
+        "--reader",
+        type=str,
+        default="auto",
+        help="Reader name or 'auto' (default: auto)",
+    )
+    read_parser.add_argument(
+        "--model_path",
+        type=str,
+        default=None,
+        help="Path to trained model (generative parser)",
+    )
+    read_parser.add_argument(
+        "--language",
+        type=str,
+        default=None,
+        help="Language for alphabeta parser",
+    )
+    read_parser.add_argument(
+        "--device",
+        type=str,
+        default=None,
+        help="Device to use (cuda/cpu/mps)",
+    )
+    read_parser.add_argument(
+        "--batch_size",
+        type=int,
+        default=8,
+        help="Batch size for parsing (default: 8)",
+    )
+
     # --- repl subcommand ---------------------------------------------------
     repl_parser = subparsers.add_parser(
         "repl",
@@ -94,6 +147,11 @@ def main():
         run_parsers()
         sys.exit(0)
 
+    if args.command == "read":
+        from hyperbase.cli.read import run_read
+        run_read(args)
+        sys.exit(0)
+
     if args.command == "repl":
         from hyperbase.cli.repl import run_repl
         run_repl(args)

src/hyperbase/cli/read.py

@@ -0,0 +1,70 @@
+import argparse
+import json
+import os
+import sys
+
+from hyperbase.parsers import get_parser
+from hyperbase.readers import get_reader
+
+
+def run_read(args: argparse.Namespace):
+    ext = os.path.splitext(args.output)[1].lower()
+
+    if ext == '.txt':
+        try:
+            reader = get_reader(args.source, reader=args.reader)
+        except ValueError as e:
+            print(f"Error: {e}", file=sys.stderr)
+            sys.exit(1)
+        print(f"Reader: {type(reader).__name__}", file=sys.stderr)
+        reader.read_to_text(args.source, args.output, progress=True)
+        print(f"\nOutput: {args.output}", file=sys.stderr)
+        return
+
+    if ext != '.jsonl':
+        print(f"Error: unsupported output extension {ext!r}"
+              " (use .jsonl or .txt)", file=sys.stderr)
+        sys.exit(1)
+
+    # Build parser kwargs
+    kwargs = {}
+    if args.parser == 'generative':
+        if args.model_path:
+            kwargs['model_path'] = args.model_path
+        if args.device:
+            kwargs['device'] = args.device
+    elif args.parser == 'alphabeta':
+        if args.language:
+            kwargs['lang'] = args.language
+
+    try:
+        parser = get_parser(args.parser, **kwargs)
+    except ValueError as e:
+        print(f"Error: {e}", file=sys.stderr)
+        sys.exit(1)
+
+    print(f"Parser: {args.parser}", file=sys.stderr)
+
+    sentences = 0
+    edges = 0
+    errors = 0
+
+    with open(args.output, 'w') as f:
+        for results in parser.read_source(
+            args.source, reader=args.reader,
+            batch_size=args.batch_size, progress=True,
+        ):
+            for result in results:
+                sentences += 1
+                if result.get('failed'):
+                    errors += 1
+                elif result.get('edge') is not None:
+                    edges += 1
+                    result['edge'] = str(result['edge'])
+                f.write(json.dumps(result, ensure_ascii=False,
+                                   default=str) + '\n')
+
+    print(f"\nSentences: {sentences}", file=sys.stderr)
+    print(f"Edges:     {edges}", file=sys.stderr)
+    print(f"Errors:    {errors}", file=sys.stderr)
+    print(f"Output:    {args.output}", file=sys.stderr)