Skip to content

Add a queue to shmrpc#2717

Open
Dreamsorcerer wants to merge 4 commits into
mainfrom
sam/fix-shmrpc-concurrency
Open

Add a queue to shmrpc#2717
Dreamsorcerer wants to merge 4 commits into
mainfrom
sam/fix-shmrpc-concurrency

Conversation

@Dreamsorcerer

Copy link
Copy Markdown
Collaborator

I'm not entirely sure if it might be more appropriate to use this everywhere for SHM or not. But, for RPC atleast, this fixes the concurrency tests.

@greptile-apps

greptile-apps Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds a queued shared-memory path for RPC messages. The main changes are:

  • Adds CpuShmQueue as a ring-buffer channel implementation.
  • Lets shared-memory pubsub choose a channel class and layout options per subclass.
  • Switches ShmRPC to the queued channel with a smaller per-message capacity.
  • Adds queue-focused IPC tests and re-enables shared-memory RPC concurrency coverage.

Confidence Score: 5/5

This looks safe to merge.

  • No blocking issues found in the changed code.

Important Files Changed

Filename Overview
dimos/protocol/pubsub/impl/shmpubsub.py Adds channel-class selection and layout-based shared-memory naming for pubsub topics.
dimos/protocol/pubsub/shm/ipc_factory.py Adds the queued CPU shared-memory channel with ring slots, per-segment ownership, publish/read logic, and close handling.
dimos/protocol/rpc/pubsubrpc.py Configures ShmRPC to use the queued shared-memory channel with fixed capacity and slot count.
dimos/protocol/pubsub/shm/test_ipc_factory.py Adds tests for queue delivery, wraparound, attach behavior, and layout mismatch handling.
dimos/protocol/rpc/test_spec.py Expands async RPC coverage across implementations and runs shared-memory concurrent-call tests.

Reviews (4): Last reviewed commit: "Add assert" | Re-trigger Greptile

Comment thread dimos/protocol/pubsub/shm/ipc_factory.py
Comment thread dimos/protocol/pubsub/impl/shmpubsub.py
Comment thread dimos/protocol/pubsub/shm/ipc_factory.py Outdated
Comment thread dimos/protocol/pubsub/shm/ipc_factory.py
@codecov

codecov Bot commented Jul 3, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.07692% with 18 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
dimos/protocol/pubsub/shm/ipc_factory.py 86.66% 11 Missing and 7 partials ⚠️
@@            Coverage Diff             @@
##             main    #2717      +/-   ##
==========================================
+ Coverage   72.35%   72.41%   +0.05%     
==========================================
  Files         902      903       +1     
  Lines       81820    82064     +244     
  Branches     7396     7451      +55     
==========================================
+ Hits        59201    59423     +222     
- Misses      20639    20649      +10     
- Partials     1980     1992      +12     
Flag Coverage Δ
OS-ubuntu-24.04-arm 64.63% <93.07%> (+0.09%) ⬆️
OS-ubuntu-latest 67.26% <93.07%> (+0.07%) ⬆️
Py-3.10 67.24% <93.07%> (+0.06%) ⬆️
Py-3.11 67.25% <93.07%> (+0.07%) ⬆️
Py-3.12 67.24% <93.07%> (+0.07%) ⬆️
Py-3.13 67.25% <93.07%> (+0.07%) ⬆️
Py-3.14 67.27% <93.07%> (+0.08%) ⬆️
Py-3.14t 67.25% <93.07%> (+0.07%) ⬆️
SelfHosted-Large 30.19% <21.92%> (-0.03%) ⬇️
SelfHosted-Linux 37.99% <24.61%> (-0.06%) ⬇️
SelfHosted-macOS 36.32% <21.92%> (-0.05%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
dimos/protocol/pubsub/impl/shmpubsub.py 75.26% <100.00%> (+0.40%) ⬆️
dimos/protocol/pubsub/shm/test_ipc_factory.py 100.00% <100.00%> (ø)
dimos/protocol/rpc/pubsubrpc.py 88.11% <100.00%> (-0.15%) ⬇️
dimos/protocol/rpc/test_spec.py 95.06% <100.00%> (-0.03%) ⬇️
dimos/protocol/pubsub/shm/ipc_factory.py 73.79% <86.66%> (+9.37%) ⬆️

... and 4 files with indirect coverage changes

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions github-actions Bot added the ready-to-merge Required CI checks have passed on this PR label Jul 3, 2026
@github-actions github-actions Bot added ready-to-merge Required CI checks have passed on this PR and removed ready-to-merge Required CI checks have passed on this PR labels Jul 3, 2026
@github-actions github-actions Bot removed the ready-to-merge Required CI checks have passed on this PR label Jul 6, 2026
Comment thread dimos/protocol/pubsub/shm/ipc_factory.py
Comment thread dimos/protocol/pubsub/shm/ipc_factory.py
@github-actions github-actions Bot added the ready-to-merge Required CI checks have passed on this PR label Jul 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-to-merge Required CI checks have passed on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant