Skip to content

fix(graph): resolve startup deadlock on liveliness query with many live tokens#226

Open
YuanYuYuan wants to merge 1 commit into
mainfrom
dev/liveliness-deadlock
Open

fix(graph): resolve startup deadlock on liveliness query with many live tokens#226
YuanYuYuan wants to merge 1 commit into
mainfrom
dev/liveliness-deadlock

Conversation

@YuanYuYuan

@YuanYuYuan YuanYuYuan commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • New nodes joining a running system with many already-live nodes/tokens could deadlock on startup inside ZContextBuilder::build(). Fixes this by giving Graph::new_with_pattern's liveliness query a large enough FIFO capacity that the deadlock condition can no longer occur.

Root cause

Graph::new_with_pattern (crates/hiroz/src/graph.rs) issues a blocking liveliness query during context startup to discover the existing ROS graph from other peers' liveliness tokens:

let replies = session.liveliness().get(&pattern).timeout(...).wait()?;

Zenoh's Session::liveliness_query synchronously replays every currently-live token matching the pattern as a reply, on the calling thread, before the call returns. The default handler backing .wait() is a bounded flume channel with capacity 256. Because the replay happens before liveliness_query() returns, the caller's own drain loop can't start running until the sender-side burst is already complete — if more than 256 live tokens match the pattern, the synchronous send blocks forever waiting for a receiver that can't run yet. Deterministic self-deadlock once live-token count exceeds capacity, matching the reported ~30% hang rate (live-token count hovering near the 256 threshold across runs).

This is a zenoh-core design issue rather than something specific to hiroz's usage; a fix is being pursued upstream at the zenoh level as well.

Fix

Graph::new_with_pattern now passes an explicit, oversized FifoChannel handler to the liveliness query instead of relying on zenoh's small default:

let replies = session
    .liveliness()
    .get(&pattern)
    .with(zenoh::handlers::FifoChannel::new(65536))
    .timeout(...)
    .wait()?;

65536 is a generous ceiling that eliminates the deadlock class at any node/token count realistically seen in a ROS graph.

Test plan

  • cargo test -p hiroz — full suite green, including graph/action discovery tests exercising the liveliness query path.

Breaking Changes

None.

zenoh's liveliness_query() synchronously replays every currently-live
matching token onto the query's reply channel, on the calling thread,
before the call returns. The default handler is a bounded 256-slot
flume channel, so once more than 256 tokens match the pattern, the
synchronous replay blocks forever waiting for a receiver that cannot
run until the call itself returns, deadlocking the caller.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant