You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In your GitHub Actions workflow run 39543153124, although the job runs on an ubuntu-latest runner host, pyrime’s pyproject.toml skips manylinux wheels:
As a result, cibuildwheel builds wheels inside an Alpine Linux container (musllinux_1_2_x86_64).
In autopxd2 3.x, the parser was pycparser, which was pure Python and had no external C library dependencies.
In autopxd2 4.0.x, autopxd2 migrated to headerkit. The default backend is libclang, which requires the system libclang.so shared library.
In your Alpine container, before-all only installed apk add librime-dev. Because neither libclang.so was installed nor the optional autopxd2[treesitter] extra was present in the isolated build environment, autopxd exited with status 1 because no parser backend was available.
How to Fix
You have two options depending on which backend you prefer:
Option 1 (recommended): Install clang-dev in the musllinux container (Recommended for full libclang support)
In pyrime's pyproject.toml, update the musllinux cibuildwheel override to install clang-dev:
Option 2: Use the tree-sitter backend (No LLVM / system package required)
rime_api.h parses cleanly with Tree-sitter without requiring system Clang libraries. Note that Tree-sitter operates only on the single input file (it does not follow #include directives or extract #define macros). To use it in headless builds:
In pyrime's pyproject.toml, request the treesitter extra:
A quick clarification on the CI run: apk add clang-dev actually did work on Alpine musl! In that run (39602387878), 20 out of 21 wheel builds succeeded (all x86_64, i686, aarch64, and ppc64le builds across Python 3.11–3.14 passed).
The single failure was on s390x (IBM System/390 Big-Endian architecture running under QEMU software emulation on GitHub Actions). I can look into what is going on with that architecture at some point soon - perhaps some difference in the clang toolchain or capabilities available for that distro on that architecture.
Regarding autopxd <= 3.x vs autopxd now:
autopxd <= 3.x used pycparser, but pycparser cannot parse real system libc headers (it required bundling a fake_libc_include stub directory) and has no support for C++. Additionally, autopxd still had to invoke an external preprocessor (gcc -E or cpp on PATH) to expand macros and resolve includes before feeding source to pycparser. There were often subtle differences in what pycparser could parse and what clang or gcc could parse, and many library headers simply failed to work with that version of autopxd. The project was a pain to maintain.
I wanted to breathe new life into the project. After researching better solutions, libclang was the winner because it provides excellent structured output, can parse more dialects, and is the actual compiler many projects use so compatibility is not an issue. This was essentially going to be a rewrite. Since a lot of the work in writing that new tool would not be specific to output of .pxd files, I decided to make it more general-purpose - and that became headerkit. I also wanted to decouple it from libclang, so it supports pluggable backends. The tree-sitter backend exists primarily as a proof of concept for headerkit's backend pluggability, but is also a perfectly useful alternative to the clang backend in many use cases.
rime_api.h specifically wraps typedefs in a preprocessor macro (RIME_FLAVORED(RimeApi)) so a real C preprocessor of some sort would be required to expand RIME_FLAVORED into RimeApi regardless of using autopxd <= 3 or current autopxd.
If you prefer to invoke headerkit directly instead of autopxd2, that is fully supported, as autopxd2 wraps the more general purpose headerkit tool, but the autopxd project now is intended to provide some degree of backwards compatibility for users who are already using autopxd in their projects. And of course versions of autopxd <= 3.0 are still available on pypi and in the git tree - those versions are just unlikely to receive any bugfixes since that was difficult to maintain and focus is on supporting the headerkit integration which is easier to maintain and more useful.
I agree pycparser has many disadvantages and will switch to headerkit, however, I still hope we can have a different alternative unrelated to headerkit:
for new users who are glad to use headerkit
let them switch to headerkit from autopxd directly, which will reduce troublesome such as bug belonging to headerkit be reported to incorrect repo (autopxd)
provide some degree of backwards compatibility
I think let user switch to headerkit is enough. not need to break original autopxd.
for old users who are insist on using autopxd2
those versions are just unlikely to receive any bugfixes
if someone want to use it, they can take over it, not create a new fork named autopxd3.
Now autopxd is same as headerkit, which defeats the purpose of its existence.
for packagers of many Linux distributions
don't need to rewrite the package scripts of autopxd2, which will reduce the maintenance cost:
if someone want to use it, they can take over it, not create a new fork named autopxd3.
I understand where you're coming from, especially from the perspective of a Linux distribution packager. Major architectural shifts are always an inconvenience for downstream packaging scripts and multi-architecture builds.
However, keeping autopxd2 alive on pycparser was simply no longer viable for me as a maintainer. This project had already been abandoned twice in its history—first by the original author around 2016, and later by the second maintainer—before being transferred to me in 2022. Nobody wanted to maintain the existing codebase because, frankly, it had many structural issues: subtle parsing bugs, inability to parse real libc headers or C++, and fragile bundled fake_libc_include workarounds. Rather than abandon it a third time, I decided to breathe new life into the project by giving it a modern foundation.
That is why this was released as a major version bump (4.0.0). Under Semantic Versioning, major version increments exist specifically to communicate breaking changes and architectural shifts.
For projects, Linux distributions, or environments that specifically require the old pure-Python pycparser implementation without libclang or tree-sitter dependencies:
autopxd2 < 4.0.0 remains permanently available on PyPI and in git tags.
Distro packages and build scripts that rely on the legacy behavior should simply pin autopxd2 < 4.0.0 (or package autopxd2_3 if both need to coexist, as is standard practice for major version breaks).
If community members wish to maintain a 3.x-lts backport branch in this repository for critical fixes, I am totally open to that. Nobody has been contributing to this project much—I am the sole contributor since December 2024.
Going forward, autopxd2 >= 4.0 continues to serve as a dedicated, focused tool for generating Cython .pxd headers—just now backed by a modern engine. Calling headerkit directly is simply a more direct alternative if you want to bypass the autopxd CLI layer or need its broader multi-language capabilities.
I appreciate you taking the time to report the NixOS and external path issues in HeaderKit and I am very happy to continue working through those so it works seamlessly in NixOS and across your downstream packages.
FYI, headerkit 0.47.2 is now live on PyPI with a fix for the s390x / 64-bit big-endian libclang AST traversal callback issue (CPython issue gh-156933) that caused the musllinux_s390x build failure in pyrime's cibuildwheel run. Since autopxd2 depends on headerkit>=0.46.2, any fresh install will automatically pick up this fix.
Activity
elijahr commentedon Oct 21, 2026
Hi @Freed-Wu,
Thanks for reporting this!
Root Cause
In your GitHub Actions workflow run 39543153124, although the job runs on an
ubuntu-latestrunner host,pyrime’spyproject.tomlskipsmanylinuxwheels:As a result,
cibuildwheelbuilds wheels inside an Alpine Linux container (musllinux_1_2_x86_64).pycparser, which was pure Python and had no external C library dependencies.headerkit. The default backend islibclang, which requires the systemlibclang.soshared library.before-allonly installedapk add librime-dev. Because neitherlibclang.sowas installed nor the optionalautopxd2[treesitter]extra was present in the isolated build environment,autopxdexited with status 1 because no parser backend was available.How to Fix
You have two options depending on which backend you prefer:
Option 1 (recommended): Install
clang-devin the musllinux container (Recommended for fulllibclangsupport)In
pyrime'spyproject.toml, update the musllinux cibuildwheel override to installclang-dev:Option 2: Use the
tree-sitterbackend (No LLVM / system package required)rime_api.hparses cleanly with Tree-sitter without requiring system Clang libraries. Note that Tree-sitter operates only on the single input file (it does not follow#includedirectives or extract#definemacros). To use it in headless builds:In
pyrime'spyproject.toml, request thetreesitterextra:In
src/pyrime/meson.build, pass--backend tree-sitter:We have also released autopxd2 v4.0.2, which includes:
apk add clang-dev) and explaining the Tree-sitter fallback and its limitations.cibuildwheel, Docker) and system package managers.Freed-Wu commentedon Oct 21, 2026
https://github.com/rimeinn/pyrime/actions/runs/39602387878/job/119229879488
After
apk add clang-dev, it still failed in:Why not keep original autopxd depending on pycparser?
headerkit XXX.h -wcythonautopxdNow
autopxdis same asheaderkit, which means we don't have any reason not to switchheaderkitand give up this project.elijahr commentedon Oct 21, 2026
Hi @Freed-Wu,
A quick clarification on the CI run:
apk add clang-devactually did work on Alpine musl! In that run (39602387878), 20 out of 21 wheel builds succeeded (all x86_64, i686, aarch64, and ppc64le builds across Python 3.11–3.14 passed).The single failure was on
s390x(IBM System/390 Big-Endian architecture running under QEMU software emulation on GitHub Actions). I can look into what is going on with that architecture at some point soon - perhaps some difference in the clang toolchain or capabilities available for that distro on that architecture.Regarding
autopxd<= 3.x vs autopxd now:autopxd<= 3.x usedpycparser, butpycparsercannot parse real system libc headers (it required bundling afake_libc_includestub directory) and has no support for C++. Additionally,autopxdstill had to invoke an external preprocessor (gcc -Eorcppon PATH) to expand macros and resolve includes before feeding source topycparser. There were often subtle differences in what pycparser could parse and what clang or gcc could parse, and many library headers simply failed to work with that version ofautopxd. The project was a pain to maintain.libclangwas the winner because it provides excellent structured output, can parse more dialects, and is the actual compiler many projects use so compatibility is not an issue. This was essentially going to be a rewrite. Since a lot of the work in writing that new tool would not be specific to output of .pxd files, I decided to make it more general-purpose - and that became headerkit. I also wanted to decouple it from libclang, so it supports pluggable backends. The tree-sitter backend exists primarily as a proof of concept for headerkit's backend pluggability, but is also a perfectly useful alternative to the clang backend in many use cases.rime_api.hspecifically wraps typedefs in a preprocessor macro (RIME_FLAVORED(RimeApi)) so a real C preprocessor of some sort would be required to expandRIME_FLAVOREDintoRimeApiregardless of using autopxd <= 3 or current autopxd.If you prefer to invoke
headerkitdirectly instead ofautopxd2, that is fully supported, asautopxd2wraps the more general purposeheaderkittool, but the autopxd project now is intended to provide some degree of backwards compatibility for users who are already usingautopxdin their projects. And of course versions ofautopxd<= 3.0 are still available on pypi and in the git tree - those versions are just unlikely to receive any bugfixes since that was difficult to maintain and focus is on supporting theheaderkitintegration which is easier to maintain and more useful.Freed-Wu commentedon Oct 21, 2026
I agree pycparser has many disadvantages and will switch to headerkit, however, I still hope we can have a different alternative unrelated to headerkit:
for new users who are glad to use headerkit
let them switch to headerkit from autopxd directly, which will reduce troublesome such as bug belonging to headerkit be reported to incorrect repo (autopxd)
I think let user switch to headerkit is enough. not need to break original autopxd.
for old users who are insist on using autopxd2
if someone want to use it, they can take over it, not create a new fork named autopxd3.
Now autopxd is same as headerkit, which defeats the purpose of its existence.
for packagers of many Linux distributions
don't need to rewrite the package scripts of autopxd2, which will reduce the maintenance cost:
Now autopxd 4.0 introduce breaking changes, we must create two packages named autopxd2_3 and autopxd2 (latest version), like
for the maintainer of headerkit
headerkit is a new project, shouldn't be mixed with autopxd based on pycparser, which will occupy its honor.
elijahr commentedon Oct 21, 2026
I understand where you're coming from, especially from the perspective of a Linux distribution packager. Major architectural shifts are always an inconvenience for downstream packaging scripts and multi-architecture builds.
However, keeping
autopxd2alive onpycparserwas simply no longer viable for me as a maintainer. This project had already been abandoned twice in its history—first by the original author around 2016, and later by the second maintainer—before being transferred to me in 2022. Nobody wanted to maintain the existing codebase because, frankly, it had many structural issues: subtle parsing bugs, inability to parse real libc headers or C++, and fragile bundledfake_libc_includeworkarounds. Rather than abandon it a third time, I decided to breathe new life into the project by giving it a modern foundation.That is why this was released as a major version bump (4.0.0). Under Semantic Versioning, major version increments exist specifically to communicate breaking changes and architectural shifts.
For projects, Linux distributions, or environments that specifically require the old pure-Python
pycparserimplementation withoutlibclangortree-sitterdependencies:autopxd2 < 4.0.0remains permanently available on PyPI and in git tags.autopxd2 < 4.0.0(or packageautopxd2_3if both need to coexist, as is standard practice for major version breaks).3.x-ltsbackport branch in this repository for critical fixes, I am totally open to that. Nobody has been contributing to this project much—I am the sole contributor since December 2024.Going forward,
autopxd2 >= 4.0continues to serve as a dedicated, focused tool for generating Cython.pxdheaders—just now backed by a modern engine. Callingheaderkitdirectly is simply a more direct alternative if you want to bypass theautopxdCLI layer or need its broader multi-language capabilities.I appreciate you taking the time to report the NixOS and external path issues in HeaderKit and I am very happy to continue working through those so it works seamlessly in NixOS and across your downstream packages.
elijahr commentedon Oct 21, 2026
FYI, headerkit 0.47.2 is now live on PyPI with a fix for the s390x / 64-bit big-endian libclang AST traversal callback issue (CPython issue gh-156933) that caused the
musllinux_s390xbuild failure in pyrime's cibuildwheel run. Since autopxd2 depends onheaderkit>=0.46.2, any fresh install will automatically pick up this fix.