[automated runtime tests][8/11] feat: CLI runner for runtime tests, one library at a time#9930
Open
tjzel wants to merge 10 commits into
Open
[automated runtime tests][8/11] feat: CLI runner for runtime tests, one library at a time#9930tjzel wants to merge 10 commits into
tjzel wants to merge 10 commits into
Conversation
tjzel
force-pushed
the
@tjzel/runtime-tests-cli
branch
2 times, most recently
from
July 13, 2026 08:03
d9e850e to
579d7be
Compare
tjzel
changed the base branch from
@tjzel/manual-runtime-tests
to
@tjzel/rejest-reanimated-decoupling
July 13, 2026 08:03
tjzel
force-pushed
the
@tjzel/runtime-tests-cli
branch
from
July 13, 2026 11:05
6048ad5 to
c869aae
Compare
tjzel
force-pushed
the
@tjzel/rejest-reanimated-decoupling
branch
from
July 13, 2026 11:05
cdc743f to
6e44caf
Compare
tjzel
force-pushed
the
@tjzel/runtime-tests-cli
branch
from
July 13, 2026 13:18
c869aae to
f3d22a0
Compare
tjzel
changed the title
tjzel
force-pushed
the
@tjzel/rejest-reanimated-decoupling
branch
from
July 14, 2026 08:29
00a917a to
feb0078
Compare
tjzel
force-pushed
the
@tjzel/runtime-tests-cli
branch
from
July 14, 2026 08:29
f3d22a0 to
f40970b
Compare
tjzel
force-pushed
the
@tjzel/rejest-reanimated-decoupling
branch
from
July 14, 2026 09:25
feb0078 to
6f039a1
Compare
tjzel
force-pushed
the
@tjzel/runtime-tests-cli
branch
2 times, most recently
from
July 14, 2026 10:26
5b7e135 to
38d1959
Compare
tjzel
force-pushed
the
@tjzel/rejest-reanimated-decoupling
branch
from
July 14, 2026 10:26
6f039a1 to
3482314
Compare
tjzel
changed the title
tjzel
force-pushed
the
@tjzel/runtime-tests-cli
branch
from
July 14, 2026 12:04
38d1959 to
2cbd138
Compare
tjzel
force-pushed
the
@tjzel/rejest-reanimated-decoupling
branch
2 times, most recently
from
July 14, 2026 12:29
8466efe to
3820918
Compare
tjzel
force-pushed
the
@tjzel/runtime-tests-cli
branch
from
July 14, 2026 12:29
2cbd138 to
4901bbc
Compare
tjzel
force-pushed
the
@tjzel/rejest-reanimated-decoupling
branch
from
July 14, 2026 13:16
3820918 to
43fa9bf
Compare
tjzel
force-pushed
the
@tjzel/runtime-tests-cli
branch
from
July 14, 2026 13:16
4901bbc to
61a65ab
Compare
tjzel
force-pushed
the
@tjzel/rejest-reanimated-decoupling
branch
from
July 14, 2026 14:03
43fa9bf to
f9d0633
Compare
tjzel
force-pushed
the
@tjzel/runtime-tests-cli
branch
from
July 14, 2026 14:03
61a65ab to
53d3c71
Compare
tjzel
force-pushed
the
@tjzel/rejest-reanimated-decoupling
branch
from
July 14, 2026 14:14
f9d0633 to
635e9d1
Compare
tjzel
force-pushed
the
@tjzel/runtime-tests-cli
branch
2 times, most recently
from
July 15, 2026 10:01
626d86e to
df90778
Compare
tjzel
force-pushed
the
@tjzel/rejest-reanimated-decoupling
branch
from
July 15, 2026 10:01
635e9d1 to
0cf898f
Compare
tjzel
changed the base branch from
@tjzel/rejest-reanimated-decoupling
to
@tjzel/rejest-test-failing
July 15, 2026 10:01
tjzel
changed the title
tjzel
changed the title
tjzel
force-pushed
the
@tjzel/runtime-tests-cli
branch
from
July 16, 2026 15:28
2ca5bd4 to
8eb3b17
Compare
tjzel
force-pushed
the
@tjzel/fix-runtime-tests
branch
from
July 16, 2026 15:28
4d3a6fa to
434f5d1
Compare
tjzel
force-pushed
the
@tjzel/runtime-tests-cli
branch
from
July 16, 2026 15:54
8eb3b17 to
d38c68f
Compare
tjzel
force-pushed
the
@tjzel/fix-runtime-tests
branch
from
July 16, 2026 15:54
434f5d1 to
dcb705b
Compare
tjzel
force-pushed
the
@tjzel/runtime-tests-cli
branch
from
July 16, 2026 20:18
d38c68f to
9a684ba
Compare
tjzel
force-pushed
the
@tjzel/fix-runtime-tests
branch
from
July 17, 2026 07:55
dcb705b to
fe70df9
Compare
tjzel
force-pushed
the
@tjzel/runtime-tests-cli
branch
2 times, most recently
from
July 17, 2026 08:09
17efd27 to
0e8cd1d
Compare
tjzel
force-pushed
the
@tjzel/fix-runtime-tests
branch
2 times, most recently
from
July 17, 2026 09:07
da2cd57 to
6024f18
Compare
tjzel
force-pushed
the
@tjzel/runtime-tests-cli
branch
from
July 17, 2026 09:07
0e8cd1d to
01bbed3
Compare
tjzel
force-pushed
the
@tjzel/fix-runtime-tests
branch
from
July 17, 2026 09:42
6024f18 to
c74c493
Compare
tjzel
force-pushed
the
@tjzel/runtime-tests-cli
branch
from
July 17, 2026 09:42
01bbed3 to
0b12610
Compare
On Android pixel snapping reports a 50dp offset as 49.999…, which Math.floor turned into 49 and failed five of the nine cases on the emulator. Rounding to the nearest integer keeps the assertion exact on iOS and correct on Android.
The app-side selection mirrors the iOS RUNTIME_TESTS compile flag with a dedicated debugRuntimeTests build type (initWith debug) that sets BuildConfig.RUNTIME_TESTS. Under that flag MainActivity registers the FabricExampleRuntimeTests component and the React host resolves index.runtimeTests.<library> as its JS main module. The library is passed per launch as an intent extra (am start --es RUNTIME_TESTS_LIBRARY <library>) rather than a system property: the extra is scoped to a single launch so a stale value can never leak into later runs, it uses only public APIs (SystemProperties requires reflection or a getprop exec), and it works identically on physical devices. MainActivity captures the extra before super.onCreate because ReactActivity's delegate creates the lazy React host inside it. A non-default Metro port has to be baked into the APK at build time through the react_native_dev_server_port resource, since Android resolves the dev server host from resources rather than from a launch environment like the iOS RCT_jsLocation default. The runtime-tests server passes -PreactNativeDevServerPort when it builds and limits the native build to the target device's ABI. The server gains --platform android (iOS stays the default): it reuses the Metro logic, picks a connected device from adb or boots the first AVD, builds assembleDebugRuntimeTests, installs the APK, sets up adb reverse for physical devices (emulators reach the host via 10.0.2.2) and launches the activity with the library extra. The connect timer is still armed only after launch. Release configurations are rejected on Android for now because embedding both per-library bundles in assets needs custom bundling tasks.
tjzel
force-pushed
the
@tjzel/runtime-tests-cli
branch
from
July 17, 2026 10:03
0b12610 to
6f1d31a
Compare
tjzel
force-pushed
the
@tjzel/fix-runtime-tests
branch
from
July 17, 2026 10:03
c74c493 to
fe3f1f3
Compare
pull Bot
pushed a commit
to esinanturan/react-native-reanimated
that referenced
this pull request
Jul 17, 2026
…he Debug/Release × BundleMode/LegacyEval matrix (software-mansion#9974) > [!NOTE] > **This PR description is AI-generated.** ## Summary Running the runtime tests across Debug/Release and Bundle Mode/Legacy Eval Mode — locally and then on CI in software-mansion#9930 — surfaced bugs in the tests themselves that only show under specific configurations or on constrained runners. Assertions depended on mode-specific error messages, exact timings and floored coordinates, `toThrow` missed errors that Worklet Runtimes report asynchronously through `ErrorUtils`, and a ~100M-character string preset OOM-killed default-RAM CI emulators. The COLOR comparator also rejected the `hwb()` colors used by the animation tests: `@react-native/normalize-colors` dropped the legacy comma `hwb()` syntax that Reanimated still parses, and the two parsers round half-value channels to different integers. The comparator now re-parses comma `hwb()` strings in space syntax and compares RGBA channel-wise with a ±1 tolerance, keeping ReJest free of `react-native-reanimated` imports. I fixed the tests ahead of the CLI PR so the automated runs start green. I also swapped the host-object specimens in the memory tests to a pooled `WorkletRuntime`, since `__workletsModuleProxy` stopped being a real `jsi::HostObject` after the `toOptimizedObject` refactor. ## Test plan All three libraries pass across the full configuration matrix locally and on CI in software-mansion#9930 and software-mansion#9932.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
This PR description is AI-generated.
Summary
Since #9929 the runtime tests run one library per JS session, but starting them still takes a human tapping through the app, which rules out CI. I added a host-side CLI:
yarn runtime-tests --library <reanimated|worklets|self-tests> [--platform <ios|android>]starts a WebSocket server (Metro port + 1), provisions the device — an iOS simulator built with theDebugRuntimeTestsscheme, or an Android emulator built withassembleDebugRuntimeTests— and launches FabricExample with the library selection (RUNTIME_TESTS_LIBRARYenv on iOS, an intent extra captured byMainActivityon Android) picking a per-library entry point; the app runs its suites headlessly and streams results back over thehello/start/log/doneprotocol, and the server exits 0/1. Worklets sessions additionally assert that Reanimated was never loaded, which #9936 made possible. I also added.github/workflows/runtime-tests-ios.yml— a reusable workflow with apull_requesttrigger on the runtime-tests paths, running self-tests (shortest, builds the app), a step verifying that theintentional failuressuite fails the runner, then Worklets and Reanimated, each with one retry; a matchingruntime-tests-android.ymlruns both libraries onubuntu-latestwith KVM and an API 34 emulator.Test plan
All three libraries pass locally through the CLI in Bundle Mode and Legacy Mode on iOS, and Reanimated passes end-to-end on an Android emulator; the iOS workflow runs green on this PR.