Skip to content

Ship the CUDA delegate as its own library - #21531

Open
shoumikhin wants to merge 14 commits into
gh/shoumikhin/81/headfrom
gh/shoumikhin/83/head
Open

Ship the CUDA delegate as its own library#21531
shoumikhin wants to merge 14 commits into
gh/shoumikhin/81/headfrom
gh/shoumikhin/83/head

Conversation

@shoumikhin

@shoumikhin shoumikhin commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

The CUDA delegate is compiled into whichever component links it, so a C++
application cannot use it without building from source, and the shim layer it
depends on ends up duplicated across several shipped libraries.

Build the delegate as a shared library and ship it in the wheel, so a process
has one copy and both the Python bindings and a C++ application can link the
same one. The CUDA runtime itself is not bundled; it continues to come from the
environment.

Two things were needed to make that actually reduce duplication:

The platform helper resolved the ExecuTorch core statically, which would give
the delegate its own copy of the backend registry. It now resolves the runtime
from the shared runtime library instead.

The shim library force-links its shim objects so their symbols survive, which is
correct, but it did so as a public link option. That propagated to every
consumer, so each one embedded another copy of the same shim code. Making it
private keeps the symbols in the library that owns them while consumers resolve
against it, which takes a representative shim symbol from three definitions down
to one.

Registration still happens through a static initializer, and the delegate is
retained on the link line so that initializer runs even though no symbol from it
is referenced directly.

This is gated on the existing EXECUTORCH_BUILD_SHARED option, so only the
Linux wheel changes; every other build keeps linking static libraries exactly as
before.

Test plan:

The wheel smoke test now asserts that exactly one shipped library defines the
CUDA delegate, alongside the existing backend-registry, thread-pool, CPU kernel,
and XNNPACK assertions. Because the delegate is only present in wheels built
with CUDA, that assertion skips cleanly when it is absent rather than failing.
All three behaviors were confirmed: it passes on a wheel built with this change,
it fails on a wheel built before it (correctly reporting three definers by
name), and it skips on a wheel built without CUDA.

The delegate's own methods are weak symbols, so the assertion checks a strong
symbol from the shim layer instead.

Built the wheel with CUDA enabled from a clean checkout and verified against a
fresh virtual environment with a normal dependency-resolving install:

  • The wheel ships the delegate as its own versioned library next to the runtime,
    the thread pool, the CPU kernels, and the XNNPACK delegate.
  • nm -DC across every shipped shared object shows exactly one definition of a
    representative delegate symbol, down from three.
  • Only the runtime library defines the backend registry, so the delegate does
    not introduce a second one.
  • The thread-pool, CPU kernel, and XNNPACK assertions all still hold with the
    CUDA delegate loaded.
  • With EXECUTORCH_BUILD_SHARED off, the delegate remains a static library and
    no new shared object is produced, so every build that does not opt in is
    unaffected.

Note for anyone building with CUDA: the build needs an explicit
CMAKE_CUDA_ARCHITECTURES for compute capability 6.1 or newer, otherwise an
integer dot-product intrinsic used by one of the kernels does not compile. That
is independent of this change.

@shoumikhin
Update
1ac4281 
[ghstack-poisoned]
@shoumikhin

shoumikhin commented Jul 31, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Stack from ghstack (oldest at bottom):

@pytorch-bot

pytorch-bot Bot commented Jul 31, 2026 •
edited
Loading

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21531

Note: Links to docs will display an error until the docs builds have been completed.

❌ 4 New Failures, 9 Pending, 1 Unrelated Failure, 9 Unclassified Failures

As of commit 7b89a94 with merge base d632341 (image):

NEW FAILURES - The following jobs have failed:

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

FLAKY - The following job failed but was likely due to flakiness present on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 31, 2026
This was referenced Jul 31, 2026
Open
Open
Open
Open
@shoumikhin
Update
9415cc5 
[ghstack-poisoned]
shoumikhin added a commit that referenced this pull request Aug 1, 2026
@shoumikhin
Ship the CUDA delegate as its own library
905a286 
The CUDA delegate is compiled into whichever component links it, so a C++
application cannot use it without building from source, and the shim layer it
depends on ends up duplicated across several shipped libraries.

Build the delegate as a shared library and ship it in the wheel, so a process
has one copy and both the Python bindings and a C++ application can link the
same one. The CUDA runtime itself is not bundled; it continues to come from the
environment.

Two things were needed to make that actually reduce duplication:

The platform helper resolved the ExecuTorch core statically, which would give
the delegate its own copy of the backend registry. It now resolves the runtime
from the shared runtime library instead.

The shim library force-links its shim objects so their symbols survive, which is
correct, but it did so as a public link option. That propagated to every
consumer, so each one embedded another copy of the same shim code. Making it
private keeps the symbols in the library that owns them while consumers resolve
against it, which takes a representative shim symbol from three definitions down
to one.

Registration still happens through a static initializer, and the delegate is
retained on the link line so that initializer runs even though no symbol from it
is referenced directly.

This is gated on the existing `EXECUTORCH_BUILD_SHARED` option, so only the
Linux wheel changes; every other build keeps linking static libraries exactly as
before.

Test plan:

The wheel smoke test now asserts that exactly one shipped library defines the
CUDA delegate, alongside the existing backend-registry, thread-pool, CPU kernel,
and XNNPACK assertions. Because the delegate is only present in wheels built
with CUDA, that assertion skips cleanly when it is absent rather than failing.
All three behaviors were confirmed: it passes on a wheel built with this change,
it fails on a wheel built before it (correctly reporting three definers by
name), and it skips on a wheel built without CUDA.

The delegate's own methods are weak symbols, so the assertion checks a strong
symbol from the shim layer instead.

Built the wheel with CUDA enabled from a clean checkout and verified against a
fresh virtual environment with a normal dependency-resolving install:

- The wheel ships the delegate as its own versioned library next to the runtime,
  the thread pool, the CPU kernels, and the XNNPACK delegate.
- `nm -DC` across every shipped shared object shows exactly one definition of a
  representative delegate symbol, down from three.
- Only the runtime library defines the backend registry, so the delegate does
  not introduce a second one.
- The thread-pool, CPU kernel, and XNNPACK assertions all still hold with the
  CUDA delegate loaded.
- With `EXECUTORCH_BUILD_SHARED` off, the delegate remains a static library and
  no new shared object is produced, so every build that does not opt in is
  unaffected.

Note for anyone building with CUDA: the build needs an explicit
`CMAKE_CUDA_ARCHITECTURES` for compute capability 6.1 or newer, otherwise an
integer dot-product intrinsic used by one of the kernels does not compile. That
is independent of this change.


ghstack-source-id: 964e8af
ghstack-comment-id: 5147959548
Pull-Request: #21531
@shoumikhin
Update
448aec8 
[ghstack-poisoned]
shoumikhin added a commit that referenced this pull request Aug 1, 2026
@shoumikhin
Ship the CUDA delegate as its own library
149715d 
The CUDA delegate is compiled into whichever component links it, so a C++
application cannot use it without building from source, and the shim layer it
depends on ends up duplicated across several shipped libraries.

Build the delegate as a shared library and ship it in the wheel, so a process
has one copy and both the Python bindings and a C++ application can link the
same one. The CUDA runtime itself is not bundled; it continues to come from the
environment.

Two things were needed to make that actually reduce duplication:

The platform helper resolved the ExecuTorch core statically, which would give
the delegate its own copy of the backend registry. It now resolves the runtime
from the shared runtime library instead.

The shim library force-links its shim objects so their symbols survive, which is
correct, but it did so as a public link option. That propagated to every
consumer, so each one embedded another copy of the same shim code. Making it
private keeps the symbols in the library that owns them while consumers resolve
against it, which takes a representative shim symbol from three definitions down
to one.

Registration still happens through a static initializer, and the delegate is
retained on the link line so that initializer runs even though no symbol from it
is referenced directly.

This is gated on the existing `EXECUTORCH_BUILD_SHARED` option, so only the
Linux wheel changes; every other build keeps linking static libraries exactly as
before.

Test plan:

The wheel smoke test now asserts that exactly one shipped library defines the
CUDA delegate, alongside the existing backend-registry, thread-pool, CPU kernel,
and XNNPACK assertions. Because the delegate is only present in wheels built
with CUDA, that assertion skips cleanly when it is absent rather than failing.
All three behaviors were confirmed: it passes on a wheel built with this change,
it fails on a wheel built before it (correctly reporting three definers by
name), and it skips on a wheel built without CUDA.

The delegate's own methods are weak symbols, so the assertion checks a strong
symbol from the shim layer instead.

Built the wheel with CUDA enabled from a clean checkout and verified against a
fresh virtual environment with a normal dependency-resolving install:

- The wheel ships the delegate as its own versioned library next to the runtime,
  the thread pool, the CPU kernels, and the XNNPACK delegate.
- `nm -DC` across every shipped shared object shows exactly one definition of a
  representative delegate symbol, down from three.
- Only the runtime library defines the backend registry, so the delegate does
  not introduce a second one.
- The thread-pool, CPU kernel, and XNNPACK assertions all still hold with the
  CUDA delegate loaded.
- With `EXECUTORCH_BUILD_SHARED` off, the delegate remains a static library and
  no new shared object is produced, so every build that does not opt in is
  unaffected.

Note for anyone building with CUDA: the build needs an explicit
`CMAKE_CUDA_ARCHITECTURES` for compute capability 6.1 or newer, otherwise an
integer dot-product intrinsic used by one of the kernels does not compile. That
is independent of this change.

ghstack-source-id: e5585aa
ghstack-comment-id: 5147959548
Pull-Request: #21531
@shoumikhin
Update
c8cb1fc 
[ghstack-poisoned]
shoumikhin added a commit that referenced this pull request Aug 1, 2026
@shoumikhin
Ship the CUDA delegate as its own library
58307a4 
The CUDA delegate is compiled into whichever component links it, so a C++
application cannot use it without building from source, and the shim layer it
depends on ends up duplicated across several shipped libraries.

Build the delegate as a shared library and ship it in the wheel, so a process
has one copy and both the Python bindings and a C++ application can link the
same one. The CUDA runtime itself is not bundled; it continues to come from the
environment.

Two things were needed to make that actually reduce duplication:

The platform helper resolved the ExecuTorch core statically, which would give
the delegate its own copy of the backend registry. It now resolves the runtime
from the shared runtime library instead.

The shim library force-links its shim objects so their symbols survive, which is
correct, but it did so as a public link option. That propagated to every
consumer, so each one embedded another copy of the same shim code. Making it
private keeps the symbols in the library that owns them while consumers resolve
against it, which takes a representative shim symbol from three definitions down
to one.

Registration still happens through a static initializer, and the delegate is
retained on the link line so that initializer runs even though no symbol from it
is referenced directly.

This is gated on the existing `EXECUTORCH_BUILD_SHARED` option, so only the
Linux wheel changes; every other build keeps linking static libraries exactly as
before.

Test plan:

The wheel smoke test now asserts that exactly one shipped library defines the
CUDA delegate, alongside the existing backend-registry, thread-pool, CPU kernel,
and XNNPACK assertions. Because the delegate is only present in wheels built
with CUDA, that assertion skips cleanly when it is absent rather than failing.
All three behaviors were confirmed: it passes on a wheel built with this change,
it fails on a wheel built before it (correctly reporting three definers by
name), and it skips on a wheel built without CUDA.

The delegate's own methods are weak symbols, so the assertion checks a strong
symbol from the shim layer instead.

Built the wheel with CUDA enabled from a clean checkout and verified against a
fresh virtual environment with a normal dependency-resolving install:

- The wheel ships the delegate as its own versioned library next to the runtime,
  the thread pool, the CPU kernels, and the XNNPACK delegate.
- `nm -DC` across every shipped shared object shows exactly one definition of a
  representative delegate symbol, down from three.
- Only the runtime library defines the backend registry, so the delegate does
  not introduce a second one.
- The thread-pool, CPU kernel, and XNNPACK assertions all still hold with the
  CUDA delegate loaded.
- With `EXECUTORCH_BUILD_SHARED` off, the delegate remains a static library and
  no new shared object is produced, so every build that does not opt in is
  unaffected.

Note for anyone building with CUDA: the build needs an explicit
`CMAKE_CUDA_ARCHITECTURES` for compute capability 6.1 or newer, otherwise an
integer dot-product intrinsic used by one of the kernels does not compile. That
is independent of this change.

ghstack-source-id: 475d170
ghstack-comment-id: 5147959548
Pull-Request: #21531
@shoumikhin
Update
d843d61 
[ghstack-poisoned]
shoumikhin added a commit that referenced this pull request Aug 1, 2026
@shoumikhin
Ship the CUDA delegate as its own library
13486fa 
The CUDA delegate is compiled into whichever component links it, so a C++
application cannot use it without building from source, and the shim layer it
depends on ends up duplicated across several shipped libraries.

Build the delegate as a shared library and ship it in the wheel, so a process
has one copy and both the Python bindings and a C++ application can link the
same one. The CUDA runtime itself is not bundled; it continues to come from the
environment.

Two things were needed to make that actually reduce duplication:

The platform helper resolved the ExecuTorch core statically, which would give
the delegate its own copy of the backend registry. It now resolves the runtime
from the shared runtime library instead.

The shim library force-links its shim objects so their symbols survive, which is
correct, but it did so as a public link option. That propagated to every
consumer, so each one embedded another copy of the same shim code. Making it
private keeps the symbols in the library that owns them while consumers resolve
against it, which takes a representative shim symbol from three definitions down
to one.

Registration still happens through a static initializer, and the delegate is
retained on the link line so that initializer runs even though no symbol from it
is referenced directly.

This is gated on the existing `EXECUTORCH_BUILD_SHARED` option, so only the
Linux wheel changes; every other build keeps linking static libraries exactly as
before.

Test plan:

The wheel smoke test now asserts that exactly one shipped library defines the
CUDA delegate, alongside the existing backend-registry, thread-pool, CPU kernel,
and XNNPACK assertions. Because the delegate is only present in wheels built
with CUDA, that assertion skips cleanly when it is absent rather than failing.
All three behaviors were confirmed: it passes on a wheel built with this change,
it fails on a wheel built before it (correctly reporting three definers by
name), and it skips on a wheel built without CUDA.

The delegate's own methods are weak symbols, so the assertion checks a strong
symbol from the shim layer instead.

Built the wheel with CUDA enabled from a clean checkout and verified against a
fresh virtual environment with a normal dependency-resolving install:

- The wheel ships the delegate as its own versioned library next to the runtime,
  the thread pool, the CPU kernels, and the XNNPACK delegate.
- `nm -DC` across every shipped shared object shows exactly one definition of a
  representative delegate symbol, down from three.
- Only the runtime library defines the backend registry, so the delegate does
  not introduce a second one.
- The thread-pool, CPU kernel, and XNNPACK assertions all still hold with the
  CUDA delegate loaded.
- With `EXECUTORCH_BUILD_SHARED` off, the delegate remains a static library and
  no new shared object is produced, so every build that does not opt in is
  unaffected.

Note for anyone building with CUDA: the build needs an explicit
`CMAKE_CUDA_ARCHITECTURES` for compute capability 6.1 or newer, otherwise an
integer dot-product intrinsic used by one of the kernels does not compile. That
is independent of this change.

ghstack-source-id: 5fb1483
ghstack-comment-id: 5147959548
Pull-Request: #21531
@shoumikhin
Update
ca8f90d 
[ghstack-poisoned]
shoumikhin added a commit that referenced this pull request Aug 1, 2026
@shoumikhin
Ship the CUDA delegate as its own library
15772ec 
The CUDA delegate is compiled into whichever component links it, so a C++
application cannot use it without building from source, and the shim layer it
depends on ends up duplicated across several shipped libraries.

Build the delegate as a shared library and ship it in the wheel, so a process
has one copy and both the Python bindings and a C++ application can link the
same one. The CUDA runtime itself is not bundled; it continues to come from the
environment.

Two things were needed to make that actually reduce duplication:

The platform helper resolved the ExecuTorch core statically, which would give
the delegate its own copy of the backend registry. It now resolves the runtime
from the shared runtime library instead.

The shim library force-links its shim objects so their symbols survive, which is
correct, but it did so as a public link option. That propagated to every
consumer, so each one embedded another copy of the same shim code. Making it
private keeps the symbols in the library that owns them while consumers resolve
against it, which takes a representative shim symbol from three definitions down
to one.

Registration still happens through a static initializer, and the delegate is
retained on the link line so that initializer runs even though no symbol from it
is referenced directly.

This is gated on the existing `EXECUTORCH_BUILD_SHARED` option, so only the
Linux wheel changes; every other build keeps linking static libraries exactly as
before.

Test plan:

The wheel smoke test now asserts that exactly one shipped library defines the
CUDA delegate, alongside the existing backend-registry, thread-pool, CPU kernel,
and XNNPACK assertions. Because the delegate is only present in wheels built
with CUDA, that assertion skips cleanly when it is absent rather than failing.
All three behaviors were confirmed: it passes on a wheel built with this change,
it fails on a wheel built before it (correctly reporting three definers by
name), and it skips on a wheel built without CUDA.

The delegate's own methods are weak symbols, so the assertion checks a strong
symbol from the shim layer instead.

Built the wheel with CUDA enabled from a clean checkout and verified against a
fresh virtual environment with a normal dependency-resolving install:

- The wheel ships the delegate as its own versioned library next to the runtime,
  the thread pool, the CPU kernels, and the XNNPACK delegate.
- `nm -DC` across every shipped shared object shows exactly one definition of a
  representative delegate symbol, down from three.
- Only the runtime library defines the backend registry, so the delegate does
  not introduce a second one.
- The thread-pool, CPU kernel, and XNNPACK assertions all still hold with the
  CUDA delegate loaded.
- With `EXECUTORCH_BUILD_SHARED` off, the delegate remains a static library and
  no new shared object is produced, so every build that does not opt in is
  unaffected.

Note for anyone building with CUDA: the build needs an explicit
`CMAKE_CUDA_ARCHITECTURES` for compute capability 6.1 or newer, otherwise an
integer dot-product intrinsic used by one of the kernels does not compile. That
is independent of this change.

ghstack-source-id: 6d6da98
ghstack-comment-id: 5147959548
Pull-Request: #21531
@shoumikhin
shoumikhin requested a review from Copilot August 1, 2026 05:01
Copilot AI reviewed Aug 1, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR changes the CUDA delegate packaging so it is built as its own shared library (when EXECUTORCH_BUILD_SHARED is enabled) and shipped in the Linux wheel, reducing duplicated shim/runtime code across shipped shared objects and enabling both Python and standalone C++ apps to link the same delegate binary.

Changes:

  • Ship libexecutorch_cuda_backend.so.<major> in the wheel alongside the runtime/XNNPACK/CPU-kernel shared libraries (when CUDA + shared build are enabled).
  • Adjust CUDA backend CMake to build the delegate as SHARED under EXECUTORCH_BUILD_SHARED and set versioning/RPATH suitable for wheel layout.
  • Extend the wheel smoke test to (a) report shipped library composition, (b) verify shipped .so dependencies are satisfiable by the wheel/environment, and (c) assert single-definer behavior for the CUDA delegate when present.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
setup.py Adds wheel installation rule to ship the CUDA delegate shared library when built.
backends/cuda/CMakeLists.txt Builds the CUDA delegate as a shared library under EXECUTORCH_BUILD_SHARED, sets SONAME/version and RPATH for wheel placement.
.ci/scripts/wheel/test_cpp_sdk.py Adds wheel composition/dependency reporting and a single-definer assertion for the CUDA delegate (optional when absent).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread backends/cuda/CMakeLists.txt
Comment thread .ci/scripts/wheel/test_cpp_sdk.py Outdated
@shoumikhin
Update
3c9845d 
[ghstack-poisoned]
@shoumikhin
Update
52b926d 
[ghstack-poisoned]
@shoumikhin
Update
d46bb95 
[ghstack-poisoned]
@shoumikhin
Update
5c03937 
[ghstack-poisoned]
@shoumikhin
Update
7b89a94 
[ghstack-poisoned]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments
@larryliu0820 larryliu0820 Awaiting requested review from larryliu0820 larryliu0820 is a code owner
@kirklandsign kirklandsign Awaiting requested review from kirklandsign kirklandsign is a code owner

Assignees

No one assigned

Labels

ciflow/binaries/all Release PRs with this label will build wheels for all python versions ciflow/binaries ciflow/cuda ciflow/nightly ciflow/periodic ciflow/trunk CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@shoumikhin