Skip to content

Ship COMPAS C++ through PyPI with bundled runtime (pip install compas-popsynth) on Linux + Macs#1473

Merged
avivajpeyi merged 24 commits intoTeamCOMPAS:devfrom
nz-gravity:dev
Apr 9, 2026
Merged

Ship COMPAS C++ through PyPI with bundled runtime (pip install compas-popsynth) on Linux + Macs#1473
avivajpeyi merged 24 commits intoTeamCOMPAS:devfrom
nz-gravity:dev

Conversation

@avivajpeyi
Copy link
Copy Markdown
Collaborator

@avivajpeyi avivajpeyi commented Apr 3, 2026

This PR adds a PyPI install path for COMPAS that bundles the native C++ executable with the Python package, so users can install and run COMPAS via pip (something mike, reinhold and I have been dreaming of since '22!)

See demo on google colab:
Screenshot 2026-04-03 at 2 45 36 am

And demo on OzStar:
Screenshot 2026-04-03 at 12 34 38 pm
Screenshot 2026-04-03 at 12 34 54 pm

And on my macbook:
Screenshot 2026-04-03 at 2 35 51 pm


What does this work on?

Works on x86_64 glibc-based Linux systems supported by the wheel’s manylinux tag (ie normal pip installs on Ubuntu/Debian/RHEL/Fedora-class systems).

Also should work on macOS 14+ for both Apple Silicon and Intel Macs.

Because the wheel bundles the COMPAS binary plus Boost/GSL/HDF5, users should not need to separately install those libraries. That is handled by the bundle/launcher logic in package-linux-bundle.sh and run_compas.sh

ATM It is not targeting: arm64 Linux, Alpine / musl Linux, Windows
Those will need more work...

I wonder if the current platforms will suffice for now.


Main changes:

  • bundled Linux COMPAS executable and runtime dependencies into the Python wheel (cant use image that is present in the main compas CI)
  • added compas_run as the packaged launcher entrypoint (had to change setup.py for this)
  • verified fresh-install wheel execution in CI (as in download the build in separate runner and test if it actually runs withoit installing)
  • started to hack on the mac-builder (but this is tough, maybe it can wait for a future PR?)

Some details on changes:

1. CPP ?= g++ in Makefile:
So CI or callers can override the compiler without editing the file.
for example:
Linux bundle build uses CPP="ccache g++" or CPP=g++
mac scaffolding uses CPP=clang++

2. Setup.py and init
I had to add a CLI entrypoint for the COMPAS exe so once someone installs the compas-python, one can actually run it.
I also had to add an email in init as that is required for python packages.

Finally, we needed some extra lines in the setup.py to link the built Cpp with python.

3. cicd scripts
I had to update the builder for mac and linux, initially made by ADACS teams for the web-compas tool.
I also had to add some additional scripts to actuall "bundle" the COMPAS exes.


How to deploy new python build?

Right now, its still a manual process.
Everytime there is a commit to dev, the linux/mac wheels will be built

But to "ship" the python release associated with new compas CPP, a user will have to bump the python version and click on the pypi-release action.

This is temporary while we are still developing/deciding if we want this feature.

Later, we can "ship" the new python release jointly with every new COMPAS release tag, so the packaged Python release can track the underlying COMPAS version instead of using a separate manually bumped Python version.

Eg when the new tag is made, we will make a new release for the python version as well.

@avivajpeyi avivajpeyi changed the title Ship COMPAS C++ through PyPI with bundled runtime (pip install compas-popsynth) on Linux Ship COMPAS C++ through PyPI with bundled runtime (pip install compas-popsynth) on Linux + Macs Apr 3, 2026
@avivajpeyi avivajpeyi marked this pull request as ready for review April 3, 2026 18:41
@avivajpeyi
Copy link
Copy Markdown
Collaborator Author

One thing to note: right now we have two CI jobs that are doing the same thing (building compas)

This is in my newly created native-linux-bundle builder, and the original main compas compile test

I think itd be better to have one signle linux ci builder.

The CI would then do:

  1. build-native-bundle
  2. verify-native-bundle
  3. run-python-tests with needs: verify-native-bundle
  4. comment-with-plot with needs: run-python-tests

I can either do this now, or in a future PR.

@avivajpeyi
Copy link
Copy Markdown
Collaborator Author

avivajpeyi commented Apr 3, 2026

Im getting PR comment error becuase my fork does not have permissions to write to the main fork...

the 'commenter' job should be running on the teamCompas fork instead, as the teamCompas fork will have permissions to write to the PR.
Screenshot 2026-04-03 at 4 01 23 pm

Copy link
Copy Markdown
Collaborator

@ilyamandel ilyamandel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks amazing, @avivajpeyi !
@jeffriley , would you like to take a look as well?

@jeffriley
Copy link
Copy Markdown
Collaborator

jeffriley commented Apr 4, 2026

@avivajpeyi and I discussed this as Avi was building it. I like it. I think there are some caveats - esp wrt what OS versions we can support, delivering the correct library versions etc. (i.e. if COMPAS requires a specific version of (say) boost, and not the latest version), but as long as those are known and accepted by the user then all good. It adds an alternative to docker that will be useful for some users. Make it clear that this only delivers the executable and whatever dependencies required to run the executable - no source. So fine for users that just want to run COMPAS, not so much if they want to make speculative changes to the physics.

EDIT: architectures too - the executable delivered will contain architecture-specific machine code, so won't be compatible with all systems.

I'm happy for this to be merged - it adds functionality and doesn't subtract what's already available.

@avivajpeyi
Copy link
Copy Markdown
Collaborator Author

avivajpeyi commented Apr 8, 2026

so shall we merge this in? Are there any other things that we need before merging this in?

(one future TODO: we need to make a TeamCOMPAS pypi account, so that others will have access to pypi settings -- right now i have used my personal account, and my personal fork to link to pypi)

@jeffriley
Copy link
Copy Markdown
Collaborator

jeffriley commented Apr 8, 2026 via email

@avivajpeyi avivajpeyi merged commit dc3dc2c into TeamCOMPAS:dev Apr 9, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants