Add xmlsec build to Docker container#25
Conversation
In the Airflow update from 2.9 to 2.10, xmlsec was added as a dependency. xmlsec is a Python package that requires the xmlsec C library installed. This script installs the C library.
I think exit code 139 indicates a Docker container was shut down due to memory constraints. I wonder if we need to give it more memory somehow? It's a little surprising though because I think GHA runners get 7GB by default. 🤔 |
CI/CD environments like GitHub Actions typically run on x86_64/amd64 systems, and when they need to build for arm64, they rely on QEMU for emulation. QEMU emulation has known limitations with certain system calls, and ldconfig is one that commonly has issues in cross-architecture builds. For this situation, it is ok for ldconfig to fail with the arm build because the containers typically run on x86_64/amd64 systems in AWS.
|
It was ldconfig failing. The actions were running on amd, and using emulation to build for arm, and the emulation was an issue for ldconfig to symlink the newly built c library. I added a thing to ignore it. Since we'll be running these on Linux/amd anyway and this is temporary till apt-get updates xmlsec to 1.3.x, it can be ignored when ldconfig fails. |
|
What will happen if someone tries to run it on an ARM machine? (We have users running these images on their Apple Silicon Macs, which is why we publish ARM images.) |
|
I built this and ran Airflow on it, but it did have a flag to use the amd version, so it was emulating amd on my m series (arm) mac. But that's still the amd version, and we're asking about the arm version. If someone pulls the arm image and runs that locally, the xmlsec library still compiles and builds correctly for each architecture, but the symlinking to that, via ldconfig, on arm will not be setup. This would have no impact on existing workflows. Only something that needs xmlsec 1.3.x would need this and currently that's only Airflow, which locally (emulated on Mac) and in prod (linux containers) is setup to run on amd. |
It seems that the original issue was a red herring and not the actual problem. Now updating to run ldconfig normally.
|
It looks like the ldconfig and qemu error was a red herring. This is building correctly now. Nothing seemingly changed, so I don't know why it's now working, but failed the other build. |
In the Airflow update from 2.9 to 2.10, xmlsec was added as a dependency. xmlsec is a Python package that requires the xmlsec C library installed. This script installs the C library. Secondary: xmlsec is available from apt-get, but it is in a 1.2.x version. We need 1.3.x, so we need to specifically build 1.3.x, which is done in this script.