Skip to content

Add xmlsec build to Docker container#25

Merged
8 commits merged into
mainfrom
BI-3349-add-temp-xmlsec-build
May 22, 2025
Merged

Add xmlsec build to Docker container#25
8 commits merged into
mainfrom
BI-3349-add-temp-xmlsec-build

Conversation

@ghost

@ghost ghost commented May 13, 2025

Copy link
Copy Markdown

In the Airflow update from 2.9 to 2.10, xmlsec was added as a dependency. xmlsec is a Python package that requires the xmlsec C library installed. This script installs the C library. Secondary: xmlsec is available from apt-get, but it is in a 1.2.x version. We need 1.3.x, so we need to specifically build 1.3.x, which is done in this script.

In the Airflow update from 2.9 to 2.10, xmlsec was added as a dependency. xmlsec is a Python package that requires the xmlsec C library installed. This script installs the C library.
@ghost ghost changed the title Add xmlsec build to Docker container BI-3349: Add xmlsec build to Docker container May 13, 2025
@ghost ghost marked this pull request as ready for review May 14, 2025 14:09
@ghost ghost self-requested a review as a code owner May 14, 2025 14:09
@bpholt

bpholt commented May 14, 2025

Copy link
Copy Markdown
Member
ERROR: failed to solve: process "/bin/sh -c if [ \"$TARGETPLATFORM\" = \"linux/amd64\" ] || [ \"$TARGETPLATFORM\" = \"linux/arm64\" ]; then     chmod +x /tmp/build_xmlsec.sh && /tmp/build_xmlsec.sh;     fi" did not complete successfully: exit code: 139

I think exit code 139 indicates a Docker container was shut down due to memory constraints. I wonder if we need to give it more memory somehow? It's a little surprising though because I think GHA runners get 7GB by default. 🤔

mforsyth added 2 commits May 14, 2025 10:35
CI/CD environments like GitHub Actions typically run on x86_64/amd64 systems, and when they need to build for arm64, they rely on QEMU for emulation. QEMU emulation has known limitations with certain system calls, and ldconfig is one that commonly has issues in cross-architecture builds. For this situation, it is ok for ldconfig to fail with the arm build because the containers typically run on x86_64/amd64 systems in AWS.
@ghost ghost changed the title BI-3349: Add xmlsec build to Docker container Add xmlsec build to Docker container May 14, 2025
@ghost

ghost commented May 14, 2025

Copy link
Copy Markdown
Author

It was ldconfig failing. The actions were running on amd, and using emulation to build for arm, and the emulation was an issue for ldconfig to symlink the newly built c library. I added a thing to ignore it. Since we'll be running these on Linux/amd anyway and this is temporary till apt-get updates xmlsec to 1.3.x, it can be ignored when ldconfig fails.

@bpholt

bpholt commented May 14, 2025

Copy link
Copy Markdown
Member

What will happen if someone tries to run it on an ARM machine? (We have users running these images on their Apple Silicon Macs, which is why we publish ARM images.)

@ghost

ghost commented May 14, 2025

Copy link
Copy Markdown
Author

I built this and ran Airflow on it, but it did have a flag to use the amd version, so it was emulating amd on my m series (arm) mac. But that's still the amd version, and we're asking about the arm version.

If someone pulls the arm image and runs that locally, the xmlsec library still compiles and builds correctly for each architecture, but the symlinking to that, via ldconfig, on arm will not be setup. This would have no impact on existing workflows. Only something that needs xmlsec 1.3.x would need this and currently that's only Airflow, which locally (emulated on Mac) and in prod (linux containers) is setup to run on amd.

mforsyth added 3 commits May 21, 2025 15:31
It seems that the original issue was a red herring and not the actual problem. Now updating to run ldconfig normally.
@ghost

ghost commented May 22, 2025

Copy link
Copy Markdown
Author

It looks like the ldconfig and qemu error was a red herring. This is building correctly now. Nothing seemingly changed, so I don't know why it's now working, but failed the other build.

@ghost ghost merged commit f38d82f into main May 22, 2025
4 checks passed
@ghost ghost deleted the BI-3349-add-temp-xmlsec-build branch May 22, 2025 17:24
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants