fix(graph): resolve startup deadlock on liveliness query with many live tokens#226
Open
YuanYuYuan wants to merge 1 commit into
Open
fix(graph): resolve startup deadlock on liveliness query with many live tokens#226YuanYuYuan wants to merge 1 commit into
YuanYuYuan wants to merge 1 commit into
Conversation
zenoh's liveliness_query() synchronously replays every currently-live matching token onto the query's reply channel, on the calling thread, before the call returns. The default handler is a bounded 256-slot flume channel, so once more than 256 tokens match the pattern, the synchronous replay blocks forever waiting for a receiver that cannot run until the call itself returns, deadlocking the caller.
YuanYuYuan
force-pushed
the
dev/liveliness-deadlock
branch
from
July 15, 2026 10:14
9a379de to
084f2d8
Compare
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ZContextBuilder::build(). Fixes this by givingGraph::new_with_pattern's liveliness query a large enough FIFO capacity that the deadlock condition can no longer occur.Root cause
Graph::new_with_pattern(crates/hiroz/src/graph.rs) issues a blocking liveliness query during context startup to discover the existing ROS graph from other peers' liveliness tokens:Zenoh's
Session::liveliness_querysynchronously replays every currently-live token matching the pattern as a reply, on the calling thread, before the call returns. The default handler backing.wait()is a bounded flume channel with capacity 256. Because the replay happens beforeliveliness_query()returns, the caller's own drain loop can't start running until the sender-side burst is already complete — if more than 256 live tokens match the pattern, the synchronous send blocks forever waiting for a receiver that can't run yet. Deterministic self-deadlock once live-token count exceeds capacity, matching the reported ~30% hang rate (live-token count hovering near the 256 threshold across runs).This is a zenoh-core design issue rather than something specific to hiroz's usage; a fix is being pursued upstream at the zenoh level as well.
Fix
Graph::new_with_patternnow passes an explicit, oversizedFifoChannelhandler to the liveliness query instead of relying on zenoh's small default:65536 is a generous ceiling that eliminates the deadlock class at any node/token count realistically seen in a ROS graph.
Test plan
cargo test -p hiroz— full suite green, including graph/action discovery tests exercising the liveliness query path.Breaking Changes
None.