Skip to content

Fix Groot2 publisher thread safety and hook handling#1165

Merged
facontidavide merged 5 commits into
masterfrom
fix/groot2-publisher-thread-safety
Jul 15, 2026
Merged

Fix Groot2 publisher thread safety and hook handling#1165
facontidavide merged 5 commits into
masterfrom
fix/groot2-publisher-thread-safety

Conversation

@facontidavide

Copy link
Copy Markdown
Collaborator

Summary

  • synchronize heartbeat configuration, transition recording, transition serialization, and PUB socket access
  • defer status subscription until publisher state is initialized, then drain status and hook callbacks during teardown
  • use a consistent hook lock order and tombstones so removed or one-shot hooks cannot be re-enabled by stale snapshots
  • correctly install, dump, and remove POST hooks
  • reject malformed JSON requests without terminating the server thread
  • add real ZMQ integration coverage and TSan-only concurrency stress tests

TDD evidence

The new tests were added and run before the implementation changes.

Red phase:

  • TSan reported races in heartbeat delay access and recording/deque access
  • teardown stress reported callback lifetime/vptr races and an unlocked mutex
  • hook stress reported the map/hook lock-order inversion
  • the POST-hook integration test showed the action was replaced before execution and POST removal failed
  • malformed JSON terminated the server process through an uncaught exception

Green phase:

  • all new functional and sanitizer-backed regressions pass after the fixes

Validation

  • pre-commit run --files <changed files>
  • ./run_clang_tidy.sh
  • normal test suite: 495/495 passed
  • ASan + UBSan test suite: 495/495 passed
  • TSan logger/Groot2 slice: 30/30 passed using the repository's existing libzmq suppression
  • final focused normal and ASan/UBSan integration tests: 2/2 passed

Compatibility

This is source-compatible, but it changes the in-memory layout of the public StatusChangeLogger and Monitor::Hook types, so consumers should rebuild against the updated library. The hook tombstone is internal state and is not serialized in the Groot2 JSON protocol.

facontidavide and others added 5 commits July 13, 2026 15:18
Apply code-review cleanups to the thread-safety fix:

- Extract the duplicated callback drain-gate (CallbackState/CallbackGuard
  and HookCallbackState/HookCallbackGuard were byte-identical) into a
  single shared BT::details::CallbackGate in utils/callback_gate.h.
  notify_all() now fires only when a drain is actually waiting.
- Replace the 69-line injected hook lambda and its immediately-invoked
  inner lambda with PImpl::runHook(). The BREAKPOINT_REACHED message is
  now built and sent without holding hook->mutex, and only the ZMQ send
  holds publisher_mutex.
- Drop the Hook::removed tombstone from the public protocol header:
  with the single lock order established (hooks_map_mutex before
  hook->mutex), enableAllHooks() and HOOKS_DUMP traverse the maps under
  hooks_map_mutex, so no stale snapshot can resurrect an erased hook and
  'enabled = false' suffices for stale callback copies.
- GET_TRANSITIONS: swap the transitions buffer out under status_mutex
  and serialize outside the lock the status callback contends on.
- Call unsubscribeFromTreeChanges() in the destructors of the five other
  StatusChangeLogger subclasses; the base-class destructor call runs too
  late to protect derived state, so each dtor must drain callbacks first.
- Deduplicate the hook-JSON literals in the Groot2 tests into
  Groot2Test::makeHook(); use make_unique for PImpl; rely on CTAD for
  lock guards in touched code.

Verified: 495/495 tests (Release), 499/499 under TSan with the CI
suppressions file, zero ThreadSanitizer warnings.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CI builds zmq from source with Conan and links it statically, so the
module-based rule 'race:libzmq.so' does not match and zmq-internal
false positives (encoder_base_t::encode, msg_t::close) fail the
Groot2 hook tests. Match the zmq namespace instead, which works
regardless of how the library is linked.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
StatusChangeLogger is the base class of a public extension point: every
logger, including user-defined ones, embeds its layout. The thread-safety
fix already changes that layout (and, being header-only, the old inlined
implementation survives in consumer binaries even after a library
upgrade), so take the opportunity to pay the ABI break only once:

- all data members move into a PImpl defined in the new
  src/loggers/abstract_logger.cpp
- the member functions are no longer inline, so future behavior fixes in
  this class can ship as an ABI-compatible library update

Adding or changing state in StatusChangeLogger no longer affects the
layout derived classes compile against. Note that changing its virtual
interface would still break ABI.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

@facontidavide facontidavide merged commit 6fe6dcd into master Jul 15, 2026
15 checks passed
@facontidavide facontidavide deleted the fix/groot2-publisher-thread-safety branch July 15, 2026 08:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant