[Bug] `remoteUpdateDstAddr_` is per-connection but overwritten by each semaphore

[miss-x-incognita]

IBConnection stores a single remoteUpdateDstAddr_ for IB no-atomic. When multiple semaphores are built on one IBConnection (e.g., src/core/executor/executor.cc), each semaphore constructor calls setRemoteUpdateDstAddr to update the single remoteUpdateDstAddr_. Therefore, only the last semaphore’s GPU address remains in remoteUpdateDstAddr_; earlier semaphores on the same connection no longer receive inbound updates at the correct location.

I would suggest either refactoring IB no-atomic delivery so the destination is not stored once per connection (stateless or per-completion semantics), or formally forbidding multiple semaphores on one connection for IB no-atomic. Let me know which you prefer so that I can raise a PR to fix.

[Binyang2014]

Hi @miss-x-incognita, sorry for reply late. We discussed this internally before. We will formally forbid the multiple semaphore on one connection. We plan to do this for both cudaIPC connection and IB connection. In this case connection and semaphore are 1-1 mapping, so we will merge semaphore into connection entirely to make it simpler.
For this issue, I have a PR for this: #762. If you have other idea, pls let me know.

cc: @chhwang

[miss-x-incognita]

Thanks a lot for the explanation! A 1-1 mapping between connection and semapore sounds great to me.