Domain-state Prometheus exporter#272
Draft
x3c41a wants to merge 2 commits into
Draft
Conversation
A PAPI sidecar that polls pallet domain state per finalized block and exports it as Prometheus metrics: provider stake/committed-bytes/balances/pending challenges (per-provider gauges), agreement counts (total/active/expired), pending challenges, checkpoint rewards/pool totals, plus counters accumulated from lifecycle events (challenge created/defended/slashed, checkpoint submitted/miss-penalized, provider registered). Serves /metrics and /health; hand-rolled exposition format, no new deps. The chain nodes already expose standard Substrate metrics; this covers the on-chain domain state nothing else does, so a stress run is observable. Feeds the monitoring stack (follow-up) and is a first consumer of #214. Part of #267.
x3c41a
marked this pull request as draft
July 7, 2026 10:11
Add block_weight_ref_time / block_weight_proof_size gauges (labelled by dispatch class) from System.BlockWeight of the latest finalized block. Substrate's own Prometheus does not expose this; it is the signal that spikes at a challenge-deadline block (the on_finalize slash sweep). Feeds the chain-health dashboard.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third slice of #267: the piece that makes a stress run observable.
The chain nodes already expose standard Substrate Prometheus metrics (block time, weight, txpool, state size). What nothing covers is the pallet's own domain state: provider balances and stake, agreement counts, pending challenges, checkpoint accruals. This adds a small PAPI sidecar that polls that state per finalized block and exports it as Prometheus metrics, so a run of
challenge-saturation.ts(or any activity) shows up on a dashboard. It is the first consumer of the telemetry work in #214 and the domain-specific scrape target themonitoring/stack (next slice) will chart.What it exports
examples/papi/chain-exporter.tsserves/metricsand/health(default port 9700).Gauges, refreshed each finalized block:
MaxPrimaryProviders/MaxMembers): stake, committed bytes, free/reserved balance, pending challenges, lifetime challenges received/failed, deregistering flagCounters, accumulated from lifecycle events (pre-initialized to 0 so
rate()has no startup gap):Design notes: aggregate counts are exported as totals, not per-entry, so a saturation run doesn't blow up label cardinality. Gauges track the finalized view (so they lag best block by the finalization gap, by design); event counters catch up over every block since the last scrape so a finalization jump doesn't drop events. Values are float64, so balances above 2^53 lose low-order precision. Hand-rolled exposition format, no new dependencies.
Tested
Live on local zombienet + provider node. Ran
just demoagainst it and watched the metrics track:providers_total0→1 on registration,buckets_total/agreements_total0→1 (lagging best by the finalization gap, as expected for a finalized-view exporter), per-provider stake/committed-bytes/balances populated, andchallenge_created_total/challenge_defended_totalincrement on the real events. No scrape errors.tsc --noEmitpasses.Follow-up
The
monitoring/stack (Prometheus + Grafana + docker-compose, with provisioned dashboards for chain health / economics / protocol activity) will scrape this exporter alongside the omni-node's built-in metrics. Provider-node/metricsgets added there once #214 lands.Independent of #269 / #271; based directly on
dev.