Skip to content

fix(pyamber): Invalidate cache while loading executor_def. - #7173

Merged
Yicong-Huang merged 1 commit into
apache:mainfrom
Neilk1021:task/fixExecutorManager
Jul 31, 2026
Merged

fix(pyamber): Invalidate cache while loading executor_def.#7173
Yicong-Huang merged 1 commit into
apache:mainfrom
Neilk1021:task/fixExecutorManager

Conversation

@Neilk1021

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

This PR calls importlib.invalidate_caches() prior to importing newly written UDF modules in ExecutorManager.load_executor_definition.

Any related issues, documentation, discussions?

Fixes #7126

How was this PR tested?

Tested with existing Python unit tests:

  • amber/src/test/python/core/architecture/managers/test_executor_manager.py::TestUpdateExecutor::test_repeated_updates_keep_carrying_the_running_state

Verified that rapid sequential executor updates cleanly load without throwing ModuleNotFoundError.

Was this PR authored or co-authored using generative AI tooling?

No.

@Yicong-Huang Yicong-Huang added the release/v1.2 back porting to release/v1.2 label Jul 31, 2026
@github-actions
github-actions Bot requested a review from xuang7 July 31, 2026 18:25
@github-actions

Copy link
Copy Markdown
Contributor

Backport auto-label report

This fix: PR was checked against each actively-supported release branch. release/* labels drive the post-merge backport, so add or remove one to change where this fix lands.

Release branch Analysis
release/v1.2 Change detected on this branch — label added; this fix is queued to backport here. Requested review from @xuang7.

Auto-label run.

@github-actions

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @Ma77Ball
    You can notify them by mentioning @Ma77Ball in a comment.

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 7 worse · ⚪ 8 noise (<±5%) · 0 without baseline

Compared against main 45b7b20 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 378 0.231 25,848/38,451/38,451 us 🔴 +14.4% / 🔴 +137.4%
bs=100 sw=10 sl=64 781 0.477 127,291/154,258/154,258 us ⚪ within ±5% / 🔴 +39.2%
🔴 bs=1000 sw=10 sl=64 912 0.557 1,096,088/1,244,057/1,244,057 us 🔴 +10.7% / 🔴 +17.1%
Baseline details

Latest main 45b7b20 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 378 tuples/sec 407 tuples/sec 765.25 tuples/sec -7.1% -50.6%
bs=10 sw=10 sl=64 MB/s 0.231 MB/s 0.248 MB/s 0.467 MB/s -6.9% -50.5%
bs=10 sw=10 sl=64 p50 25,848 us 22,587 us 12,703 us +14.4% +103.5%
bs=10 sw=10 sl=64 p95 38,451 us 34,502 us 16,199 us +11.4% +137.4%
bs=10 sw=10 sl=64 p99 38,451 us 34,502 us 19,557 us +11.4% +96.6%
bs=100 sw=10 sl=64 throughput 781 tuples/sec 771 tuples/sec 965.22 tuples/sec +1.3% -19.1%
bs=100 sw=10 sl=64 MB/s 0.477 MB/s 0.471 MB/s 0.589 MB/s +1.3% -19.0%
bs=100 sw=10 sl=64 p50 127,291 us 125,972 us 103,830 us +1.0% +22.6%
bs=100 sw=10 sl=64 p95 154,258 us 162,308 us 110,778 us -5.0% +39.2%
bs=100 sw=10 sl=64 p99 154,258 us 162,308 us 118,713 us -5.0% +29.9%
bs=1000 sw=10 sl=64 throughput 912 tuples/sec 927 tuples/sec 992.33 tuples/sec -1.6% -8.1%
bs=1000 sw=10 sl=64 MB/s 0.557 MB/s 0.566 MB/s 0.606 MB/s -1.6% -8.0%
bs=1000 sw=10 sl=64 p50 1,096,088 us 1,077,186 us 1,016,959 us +1.8% +7.8%
bs=1000 sw=10 sl=64 p95 1,244,057 us 1,124,134 us 1,062,501 us +10.7% +17.1%
bs=1000 sw=10 sl=64 p99 1,244,057 us 1,124,134 us 1,082,611 us +10.7% +14.9%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,528.70,200,128000,378,0.231,25848.21,38450.79,38450.79
1,100,10,64,20,2561.40,2000,1280000,781,0.477,127291.16,154257.67,154257.67
2,1000,10,64,20,21923.56,20000,12800000,912,0.557,1096087.94,1244057.08,1244057.08

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.37%. Comparing base (02ae12b) to head (ad0464c).
⚠️ Report is 45 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #7173      +/-   ##
============================================
+ Coverage     79.27%   79.37%   +0.10%     
  Complexity     3787     3787              
============================================
  Files          1160     1160              
  Lines         46124    46125       +1     
  Branches       5114     5114              
============================================
+ Hits          36563    36610      +47     
+ Misses         7942     7896      -46     
  Partials       1619     1619              
Flag Coverage Δ *Carryforward flag
access-control-service 70.00% <ø> (ø) Carriedforward from 02ae12b
agent-service 76.76% <ø> (ø) Carriedforward from 02ae12b
amber 72.60% <ø> (ø) Carriedforward from 02ae12b
computing-unit-managing-service 20.49% <ø> (ø) Carriedforward from 02ae12b
config-service 66.66% <ø> (ø) Carriedforward from 02ae12b
file-service 67.21% <ø> (ø) Carriedforward from 02ae12b
frontend 83.08% <ø> (ø) Carriedforward from 02ae12b
notebook-migration-service 78.94% <ø> (ø) Carriedforward from 02ae12b
pyamber 97.36% <100.00%> (+1.01%) ⬆️
workflow-compiling-service 26.31% <ø> (ø) Carriedforward from 02ae12b

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Yicong-Huang Yicong-Huang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks

@Yicong-Huang
Yicong-Huang enabled auto-merge July 31, 2026 18:33
@Yicong-Huang
Yicong-Huang added this pull request to the merge queue Jul 31, 2026
Merged via the queue into apache:main with commit 24a20c8 Jul 31, 2026
29 of 35 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

Backport PR opened: draft #7174 (#7174) to release/v1.2, assigned to @Neilk1021 — needs manual work because the backported tree failed its pre-merge build.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pyamber release/v1.2 back porting to release/v1.2

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ExecutorManager can fail to import a freshly written UDF module with ModuleNotFoundError because importlib caches are not invalidated

3 participants