Skip to content

safety(continuous): pool-epoch listener liveness metrics #557

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
julio4 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

safety(continuous): pool-epoch listener liveness metrics #557

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 16 additions & 4 deletions crates/op-rbuilder/src/builder/service.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ use crate::{
traits::{NodeBounds, PoolBounds},
};
use eyre::WrapErr as _;
use futures::FutureExt;
use reth_node_api::NodeTypes;
use reth_node_builder::{BuilderContext, components::PayloadServiceBuilder};
use reth_optimism_evm::OpEvmConfig;
Expand Down Expand Up @@ -121,11 +122,22 @@ impl FlashblocksServiceBuilder {
let mut pending_txs =
pool.pending_transactions_listener_for(TransactionListenerKind::All);
let pool_change_epoch = pool_change_epoch.clone();
let listener_metrics = metrics.clone();
ctx.task_executor().spawn_task(async move {
// Every new pending tx may improve current candidate
while pending_txs.recv().await.is_some() {
pool_change_epoch.fetch_add(1, Ordering::Relaxed);
}
let listen = async {
// Every new pending tx may improve current candidate
while pending_txs.recv().await.is_some() {
let epoch = pool_change_epoch.fetch_add(1, Ordering::Relaxed) + 1;
listener_metrics.pool_change_epoch.set(epoch as f64);
}
};
let _ = std::panic::AssertUnwindSafe(listen).catch_unwind().await;

@akundaz akundaz Jul 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you need to catch panics for this?

@julio4 julio4 Jul 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The counter is an alert signal, without catching panic we would skip it and miss alert.

@akundaz akundaz Jul 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean have you been seeing anything to motivate tracking this? This closure is very simple so I don't see the need to catch a panic here

@julio4 julio4 Jul 17, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to move all this pool logic somewhere else, so from the builder service I consider this closure like a black box even if it is very simple (at the moment).

Now if the pool listener panic for whatever reason we can't tell anything from the builder service, and can't make the difference between an empty pool or a panicked one. But totally agree that in the current shape this seems useless, I can always add this later with more changes that justify this if needed.

listener_metrics.pool_epoch_listener_exit.increment(1);
warn!(
target: "payload_builder",
"Pool epoch listener exited (channel closed or task panicked); \
continuous candidate gating is now frozen and will degrade to idle-backoff"
);
Comment on lines +134 to +140
});
}

Expand Down
8 changes: 8 additions & 0 deletions crates/op-rbuilder/src/metrics.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,14 @@ pub struct OpRBuilderMetrics {
/// candidate loop: best-candidate fees minus first-candidate fees.
/// Recorded once per flashblock interval; 0 if no improvement.
pub continuous_fee_improvement: Histogram,
/// Current value of the continuous-build pool-change epoch counter.
/// A flatlined gauge on a dashboard means the pool listener died, not
/// that the pool is idle.
Comment on lines +201 to +203
pub pool_change_epoch: Gauge,
/// Pool-change epoch listener task exited (pending-tx channel closed or
/// panicked); continuous candidate gating is frozen and degrades to its
/// idle-backoff ceiling until restart.
pub pool_epoch_listener_exit: Counter,
/// Builds completed but not published due to resolved gate
pub flashblock_publish_suppressed_total: Counter,
/// TODO: Use labels for these
Expand Down
Loading