Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 15 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

## [4.0.1] - 2026-10-19
### Changed
- Improved CLI error message when `libclang` is unavailable: now lists platform
package manager commands (including Alpine Linux `apk add clang-dev`), details
the `autopxd2[treesitter]` alternative, and explicitly summarizes tree-sitter's
limitations (parses single file only, no `#include` recursion, no `#define` macros).

### Documentation
- Added Alpine Linux (`apk add clang-dev`) to system package manager installation
instructions in `README.md` and `docs/getting-started/installation.md`.
- Added a dedicated "Headless CI & Build Containers (cibuildwheel, Docker)" guide
to `README.md` and `docs/getting-started/installation.md` for projects using
autopxd2 in automated container builds.
- Expanded the documentation of the `tree-sitter` backend's limitations and
trade-offs versus `libclang` in `README.md`, `installation.md`, and
`docs/user-guide/backends.md`.

### Changed
- Updated test snapshot fixtures (`issue_039_opaque_struct.test`, `keywords.test`, `array_dimensions.test`, `simple_c.expected.pxd`, and `json_lib.expected.pxd`) to reflect HeaderKit 0.46.2 struct/variable ordering, `ctypedef struct` opaque struct typedef semantics, and topological dependency sort order.
Expand Down
38 changes: 32 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ You only need the system `libclang` shared library (`libclang.so`, `libclang.dyl
If `libclang` is not already installed on your system, install it via your system package manager:
- **macOS**: `brew install llvm`
- **Ubuntu/Debian**: `sudo apt-get install libclang-dev`
- **Alpine Linux (musl)**: `apk add clang-dev`
- **Fedora/RHEL**: `sudo dnf install clang-devel`
- **Windows**: `winget install LLVM.LLVM`

Expand All @@ -93,12 +94,37 @@ Then specify `--backend tree-sitter`:
autopxd --backend tree-sitter myheader.h myheader.pxd
```

The tree-sitter backend parses the given file only. Compared to libclang it does not:

- follow `#include` directives, so declarations from included headers are absent
- extract `#define` macros as constants
- detect system include paths
- accept `--std` or `--clang-arg` (both are rejected with an error)
> **Limitations:** The tree-sitter backend is more limited than `libclang`. It operates purely on the syntax tree of the given file. Compared to libclang, it:
> - **Does not follow `#include` directives**: Declarations from included headers are absent.
> - **Does not extract `#define` macros**: Macro constants and macro functions are not emitted.
> - **Does not detect system include paths**: Local/system header lookups are skipped.
> - **Rejects compiler driver options**: Options like `--std` or `--clang-arg` are rejected with an error.

Use `tree-sitter` for self-contained headers where installing system LLVM is not feasible. For umbrella headers and full macro extraction, use `libclang`.

### Headless CI & Build Containers (cibuildwheel, Docker)

In minimal Docker containers or CI build environments (such as Alpine/`musllinux`, `manylinux`, or slim Debian images) where `libclang` is not present, you can:

1. **Install `libclang` in the container:**
- **Alpine / musllinux**: `apk add clang-dev`
- **Debian / Ubuntu**: `apt-get install -y libclang-dev`
- **RHEL / manylinux**: `dnf install -y clang-devel`

*(For example, in `cibuildwheel` config in `pyproject.toml`:)*
```toml
[[tool.cibuildwheel.overrides]]
select = ["*-musllinux_*"]
before-all = "apk add clang-dev"
```

2. **Or use the Tree-sitter backend (no system packages required):**
Declare `autopxd2[treesitter]` in your build dependencies:
```toml
[build-system]
requires = ["meson-python", "autopxd2[treesitter]"]
```
and pass `--backend tree-sitter` in your build script (e.g. `meson.build`).

## Quick Start

Expand Down
21 changes: 18 additions & 3 deletions autopxd/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -346,11 +346,26 @@ def _print_backends_json() -> None:


DOCKER_DOCS_URL = "https://elijahr.github.io/python-autopxd2/getting-started/docker/"
INSTALL_DOCS_URL = "https://elijahr.github.io/python-autopxd2/getting-started/installation/"

LIBCLANG_REQUIRED_ERROR = f"""Error: libclang backend not available.
Install libclang with headerkit: python -m headerkit.install_libclang
Or install LLVM/Clang manually (e.g., apt install libclang-dev, brew install llvm)
Or use Docker: {DOCKER_DOCS_URL}
To use libclang, install it via your system package manager:
- Ubuntu/Debian: sudo apt-get install libclang-dev
- Alpine Linux (musl): apk add clang-dev
- Fedora/RHEL: sudo dnf install clang-devel
- macOS: brew install llvm
- Automated installer: python -m headerkit.install_libclang

Alternatively, use the tree-sitter backend (no LLVM / system library required):
pip install "autopxd2[treesitter]"
autopxd --backend tree-sitter <header.h> [<output.pxd>]

Note: The tree-sitter backend is more limited than libclang: it parses the input
file only (does not follow #include directives), does not extract #define macros,
and cannot detect system include paths.

See installation docs: {INSTALL_DOCS_URL}
Or Docker usage: {DOCKER_DOCS_URL}
"""


Expand Down
34 changes: 32 additions & 2 deletions docs/getting-started/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,8 @@ autopxd --backend tree-sitter myheader.h myheader.pxd
```

This backend parses only the file you give it: it does not follow `#include`
directives, does not extract `#define` macros, and rejects `--std` and
`--clang-arg`. See [Parser Backends](../user-guide/backends.md).
directives, does not extract `#define` macros, does not detect system include
paths, and rejects `--std` and `--clang-arg`. See [Parser Backends](../user-guide/backends.md).

## System Package Managers

Expand All @@ -54,6 +54,12 @@ brew install llvm
sudo apt-get install libclang-dev llvm
```

### Alpine Linux / musl

```bash
apk add clang-dev
```

### Fedora / RHEL

```bash
Expand All @@ -65,6 +71,30 @@ sudo dnf install clang-devel llvm
1. Install LLVM from [releases.llvm.org](https://releases.llvm.org/) or via `winget install LLVM.LLVM`
2. Ensure the LLVM `bin` directory is in your `PATH`

## Headless CI & Build Containers (cibuildwheel, Docker)

In minimal Docker containers or CI build environments (such as Alpine/`musllinux`, `manylinux`, or slim Debian images) where `libclang` is not present, you can:

1. **Install `libclang` in the container:**
- **Alpine / musllinux**: `apk add clang-dev`
- **Debian / Ubuntu**: `apt-get install -y libclang-dev`
- **RHEL / manylinux**: `dnf install -y clang-devel`

*(For example, in `cibuildwheel` config in `pyproject.toml`:)*
```toml
[[tool.cibuildwheel.overrides]]
select = ["*-musllinux_*"]
before-all = "apk add clang-dev"
```

2. **Or use the Tree-sitter backend (no system packages required):**
Declare `autopxd2[treesitter]` in your build dependencies:
```toml
[build-system]
requires = ["meson-python", "autopxd2[treesitter]"]
```
and pass `--backend tree-sitter` in your build script (e.g. `meson.build`).

## Development Installation

For development, clone the repository and install with development dependencies:
Expand Down
23 changes: 15 additions & 8 deletions docs/user-guide/backends.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Default: libclang
| `--std` / `--clang-arg` | ✅ | ❌ (rejected with an error) |

Use libclang unless you cannot install LLVM. Use tree-sitter for self-contained
headers in environments where a system `libclang` is unavailable.
headers in environments (such as minimal CI or Docker containers) where a system `libclang` is unavailable.

## libclang

Expand All @@ -34,9 +34,10 @@ Uses LLVM's clang library for parsing. Provides the same parser used by producti
- Full C++ support (classes, templates, namespaces)
- Extracts macros as constants (integers, floats, strings, expressions)
- Automatic system include path detection
- Follows and recursively expands `#include` directives across translation units
- Handles comments and preprocessor directives directly
- Better error messages
- Handles complex headers reliably
- Handles complex and umbrella headers reliably

**Limitations:**

Expand All @@ -55,7 +56,7 @@ autopxd myheader.hpp
## tree-sitter

Uses [tree-sitter](https://tree-sitter.github.io/) grammars for C and C++. It needs no
system LLVM, so it installs and runs anywhere Python does.
system LLVM, so it installs and runs anywhere Python does using pre-built PyPI wheels.

**Install:**

Expand All @@ -74,18 +75,24 @@ autopxd --backend tree-sitter --cpp myclass.hpp

**Limitations:**

- Does not follow `#include` directives. Given `#include "inner.h"`, declarations from
`inner.h` do not appear in the output; only the parsed file's own declarations do.
- Does not extract `#define` macros as constants.
- Does not detect system include paths.
- Rejects `--std` and `--clang-arg`:
The tree-sitter backend is substantially more limited than libclang:

- **No `#include` recursion**: It parses only the file given to it. Given `#include "inner.h"` or `#include <header.h>`, declarations from the included headers do not appear in the output; only the parsed file's own declarations do. If the header relies on types defined in included files, those type definitions will not be present.
- **No `#define` macro extraction**: Constants and macro functions defined via `#define` are not extracted into the `.pxd`.
- **No system include paths**: It does not query or search compiler system include directories.
- **Rejects compiler driver options**: Options that configure the compiler frontend (`--std` and `--clang-arg`) are rejected with an explicit error:

```console
$ autopxd --backend tree-sitter --std c11 myheader.h
Error: --std is only supported by the libclang backend, but the resolved backend is 'tree-sitter'.
Use --backend libclang, or drop the option.
```

**When to use tree-sitter:**
- Single, self-contained C or C++ header files.
- Headless CI pipelines, Docker containers, or minimal environments (e.g. Alpine/musllinux) where installing LLVM/Clang is impractical.
- When you do not need macro constants or included header declarations in the generated `.pxd`.

## System Include Paths

The libclang backend automatically detects system include directories by querying the system clang compiler. This means headers like `<stddef.h>` and `<stdint.h>` work without requiring manual `-I` flags.
Expand Down
13 changes: 13 additions & 0 deletions test/test_cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,19 @@ def test_unknown_backend_not_available(self) -> None:
"""Unknown backends should not be available."""
assert is_backend_available("nonexistent") is False

def test_libclang_unavailable_error_message(self, monkeypatch: pytest.MonkeyPatch, simple_header_file: str) -> None:
"""When libclang is requested but unavailable, print helpful guidance."""
import autopxd

monkeypatch.setattr(autopxd, "is_backend_available", lambda name: False)
runner = CliRunner()
result = runner.invoke(cli, ["--backend", "libclang", simple_header_file])
assert result.exit_code != 0
assert "Error: libclang backend not available" in result.output
assert "apk add clang-dev" in result.output
assert "tree-sitter" in result.output
assert 'pip install "autopxd2[treesitter]"' in result.output


class TestBackendInfo:
"""Tests for backend info retrieval."""
Expand Down
Loading