add convert_arrow_to_parquet_streaming.py to tools

Author: Ricardo Decal

.github/copilot-instructions.md

@@ -13,6 +13,42 @@ Inspired by [Simon Willison's tools collection](https://github.com/simonw/tools)
 
 ## Available Tools
 
+### [convert_arrow_to_parquet_streaming.py](python/convert_arrow_to_parquet_streaming.py)
+
+Output of `uv run https://tools.ricardodecal.com/python/convert_arrow_to_parquet_streaming.py --help`:
+
+```text
+Usage: convert_arrow_to_parquet_streaming.py [OPTIONS]
+
+  Convert Arrow shards to Parquet.
+
+  - Discovers all .arrow files under a given source directory - Converts each
+  file to Parquet - Uses streaming in order to keep memory bounded and convert
+  files larger than available RAM - Handles both Arrow IPC File and Stream
+  formats (tries file, falls back to stream)
+
+  Notes: - Use `--preserve-subdirs` to mirror the input directory tree under
+  the output dir. - Use `--overwrite` to re-create files; otherwise existing
+  outputs are skipped.
+
+  Arguments:
+      SOURCE_DIR: Directory containing .arrow files.
+      OUTPUT_DIR: Directory to write .parquet files.
+
+  Examples:
+
+      uv run https://tools.ricardodecal.com/python/convert_arrow_to_parquet_streaming.py --source-dir ./arrow_data --output-dir ./parquet_data
+      uv run https://tools.ricardodecal.com/python/convert_arrow_to_parquet_streaming.py --source-dir ./arrow_data --output-dir ./parquet_data --preserve-subdirs
+
+Options:
+  --source-dir DIRECTORY  Directory containing .arrow files  [required]
+  --output-dir DIRECTORY  Directory to write .parquet files
+  --overwrite             Overwrite existing parquet files
+  --preserve-subdirs      Preserve input subdirectory structure inside output
+                          dir
+  --help                  Show this message and exit.
+```
+
 ### [yt_transcript.py](python/yt_transcript.py)
 
 Output of `uv run https://tools.ricardodecal.com/python/yt_transcript.py --help`:

README.md

@@ -170,6 +170,38 @@ <h1>Ricardo Decal's Tools</h1>
 <p>This is an experiment in low-stakes vibe coding. The code lives in <a href="https://github.com/crypdick/tools"><code>crypdick/tools</code></a>.</p>
 <p>Inspired by <a href="https://github.com/simonw/tools">Simon Willison's tools collection</a>.</p>
 <h2>Available Tools</h2>
+<h3><a href="python/convert_arrow_to_parquet_streaming.py">convert_arrow_to_parquet_streaming.py</a></h3>
+<p>Output of <code>uv run https://tools.ricardodecal.com/python/convert_arrow_to_parquet_streaming.py --help</code>:</p>
+<pre><code class="language-text">Usage: convert_arrow_to_parquet_streaming.py [OPTIONS]
+
+  Convert Arrow shards to Parquet.
+
+  - Discovers all .arrow files under a given source directory - Converts each
+  file to Parquet - Uses streaming in order to keep memory bounded and convert
+  files larger than available RAM - Handles both Arrow IPC File and Stream
+  formats (tries file, falls back to stream)
+
+  Notes: - Use `--preserve-subdirs` to mirror the input directory tree under
+  the output dir. - Use `--overwrite` to re-create files; otherwise existing
+  outputs are skipped.
+
+  Arguments:
+      SOURCE_DIR: Directory containing .arrow files.
+      OUTPUT_DIR: Directory to write .parquet files.
+
+  Examples:
+
+      uv run https://tools.ricardodecal.com/python/convert_arrow_to_parquet_streaming.py --source-dir ./arrow_data --output-dir ./parquet_data
+      uv run https://tools.ricardodecal.com/python/convert_arrow_to_parquet_streaming.py --source-dir ./arrow_data --output-dir ./parquet_data --preserve-subdirs
+
+Options:
+  --source-dir DIRECTORY  Directory containing .arrow files  [required]
+  --output-dir DIRECTORY  Directory to write .parquet files
+  --overwrite             Overwrite existing parquet files
+  --preserve-subdirs      Preserve input subdirectory structure inside output
+                          dir
+  --help                  Show this message and exit.
+</code></pre>
 <h3><a href="python/yt_transcript.py">yt_transcript.py</a></h3>
 <p>Output of <code>uv run https://tools.ricardodecal.com/python/yt_transcript.py --help</code>:</p>
 <pre><code class="language-text">Usage: yt_transcript.py [OPTIONS] URL [OUTPUT_FILE]

index.html

@@ -35,17 +35,6 @@ no_implicit_optional = true
 show_error_codes = true
 show_column_numbers = true
 
-# Exclude test files from strict checking
-[[tool.mypy.overrides]]
-module = "tests.*"
-ignore_errors = true
-
-[tool.pytest.ini_options]
-testpaths = ["tests"]
-python_files = "test_*.py"
-python_functions = "test_*"
-addopts = "-v --strict-markers"
-
 [tool.ruff]
 target-version = "py312"
 
@@ -64,8 +53,5 @@ select = [
 ]
 ignore = ["E501"]
 
-[tool.ruff.lint.per-file-ignores]
-"tests/*" = ["S101"]  # Allow assert in tests
-
 [tool.ruff.lint.isort]
 known-first-party = ["tools"]

pyproject.toml

@@ -12,21 +12,16 @@ uv run https://tools.ricardodecal.com/python/foo.py [args]
 uv run python/foo.py [args]
 ```
 
-## Available Tools
-
-- `yt_transcript.py`: Fetch YouTube transcripts for a single video or a whole playlist into a single file.
-
----
-
 ## Creating Tools
 
 ### Required Structure
 
 **CRITICAL:** The tool's `--help` output is used to **auto-generate the main README.md and the website**.
 
 - Ensure your tool supports `--help` (standard with `click`).
-- The docstring of your main command function (decorated with `@click.command`) provides the help description.
-- Keep the first line as a clear, concise summary.
+- **Module Docstring**: Keep the top-level module docstring to a single sentence summary.
+- **Main Docstring**: The docstring of your main command function (decorated with `@click.command`) provides the full help description.
+- Keep the first line of the main docstring as a clear, concise summary.
 - Manually document `@click.argument` arguments in an `Arguments:` section (Click does not auto-document them).
 - Include concrete `Examples:` section showing exactly how to run it.
 - Do not use generic `Usage:` placeholders.
@@ -52,6 +47,9 @@ def main(name: str, output: str | None) -> None:
     """
     Tool description that will be used to auto-generate the main README.md and the website.
 
+    This docstring is the source of truth for the tool's documentation.
+    It should include a detailed description, arguments list, and usage examples.
+
     Arguments:
         NAME: The name of the person to greet.
 
@@ -138,40 +136,6 @@ for item in track(items, description="Processing..."):
 
 ---
 
-## Testing
-
-Create `tests/test_your_tool.py`:
-
-```python
-from pathlib import Path
-import subprocess
-
-TOOL = Path(__file__).parent.parent / "python" / "foo.py"
-
-def run_tool(*args):
-    return subprocess.run(
-        ["uv", "run", str(TOOL), *args],
-        capture_output=True,
-        text=True,
-        check=False,
-    )
-
-def test_help():
-    result = run_tool("--help")
-    assert result.returncode == 0
-    assert "Usage:" in result.stdout
-
-def test_basic():
-    result = run_tool("arg")
-    assert result.returncode == 0
-```
-
-Run: `uv run pytest tests/test_your_tool.py -v`
-
-See [tests/README.md](../tests/README.md) for details.
-
----
-
 ## Common Patterns
 
 ### File I/O
@@ -258,12 +222,11 @@ uv add --script python/foo.py click requests
 # Test
 uv run python/foo.py --help
 
-# Add tests
-touch tests/test_my_tool.py
-uv run pytest tests/test_my_tool.py -v
+# Bump dependencies to latest versions
+uv remove --script python/foo.py click requests && uv add --script python/foo.py click requests
 
 # Commit
-git add python/foo.py tests/test_my_tool.py
+git add python/foo.py
 git commit -m "Add foo: description"
 
 # Test from URL after push