Add probing service by randomlogin · Pull Request #815 · lightningdevkit/ldk-node

randomlogin · 2026-03-01T11:20:36Z

Added a probing service which is used to send probes to estimate channels' capacities.

Related issue: #765.

Probing is intended to be used in two ways:

on a 'normal' node that allocates some liquidity to probe channels;
on a probing node which doesn't make any real payments, just observes the network.

For probing a new abstraction Prober is defined and is (optionally) created during node building.
Prober periodically sends probes to feed the data to the scorer.
Prober sends probes using a ProbingStrategy.

ProbingStrategy trait has only one method: fn next_probe(&self) -> Option<Probe>; every tick it generates a probe, where Probe represents how to send a probe.

To accommodate two different ways the probing is used, we either construct a probing route manually (Probe::PrebuiltRoute) or rely on the router/scorer (Probe::Destination).

Prober tracks how much liquidity is locked in-flight in probes, prevents the new probes from firing if the cap is reached.

There are two probing strategies implemented:

Random probing strategy, it picks a random route from the current node, the route is probed via send_probe, thus ignores scoring parameters (what hops to pick), it also ignores liquidity_limit_multiplier which prohibits taking a hop if its capacity is too small. It is a true random route.
High degree probing strategy, it examines the graph and finds the nodes with the biggest number of (public) channels and probes routes to them using send_spontaneous_preflight_probes which uses the current router/scorer.

The former is meant to be used on payment nodes, while the latter on probing nodes. For the HighDegreeStrategy to work it is recommended to set probing_diversity_penalty_msat to some nonzero value to prevent routes reuse, however it may fail to find any available routes.

There are three tests added:

check the probing locked amount increases/decreases
check that the new probes are not fired if the current locked amount cap is reached
performance testing which sets up a network of nodes and 4 observing nodes

Example output (runs for ~1 minute, needs --nocapture flag):

SCID            Direction         Probing Random               Probing HiDeg                Probing HiDeg+P              Probing nostrat              Real outbound msat
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
114349209354241 C→Random          [0, 90 100 235]              unknown                      unknown                      unknown                      0
114349209354241 Random→C          [9 899 765, 100 000 000]     unknown                      unknown                      unknown                      917 184 187
114349209419777 D→HiDeg+P         unknown                      unknown                      [0, 91 127 725]              unknown                      0
114349209419777 HiDeg+P→D         unknown                      unknown                      [8 872 275, 100 000 000]     unknown                      905 313 697
114349209485312 B→F               [0, 3 418 855]               unknown                      unknown                      unknown                      160 288 291
114349209485312 F→B               [96 581 145, 100 000 000]    unknown                      unknown                      unknown                      819 711 709
114349209550848 B→D               [0, 6 141 420]               [0, 2 988 399]               [0, 91 127 725]              unknown                      30 052 300
114349209550848 D→B               [93 858 580, 100 000 000]    [97 011 601, 100 000 000]    [8 872 275, 100 000 000]     unknown                      874 555 512
114349209616384 E→HiDeg           [0, 2 097 950]               [0, 90 455 967]              unknown                      unknown                      0
114349209616384 HiDeg→E           [97 902 050, 100 000 000]    [9 544 033, 100 000 000]     unknown                      unknown                      913 366 836
114349209681920 B→C               [98 411 698, 100 000 000]    unknown                      unknown                      unknown                      86 511 108
114349209681920 C→B               [0, 1 588 302]               unknown                      unknown                      unknown                      812 521 355
114349209747457 B→E               [0, 6 625 595]               [0, 90 455 967]              [0, 4 795 311]               unknown                      0
114349209747457 E→B               [93 374 405, 100 000 000]    [9 544 033, 100 000 000]     [95 204 689, 100 000 000]    unknown                      907 339 924
114349209812993 C→nostrat         [0, 1 113 591]               unknown                      unknown                      unknown                      0
114349209812993 nostrat→C         [98 886 409, 100 000 000]    unknown                      unknown                      unknown                      990 000 000
114349209878528 C→E               [2 097 950, 5 335 740]       [0, 90 808 929]              unknown                      unknown                      55 569 306
114349209878528 E→C               [94 664 260, 97 902 050]     [9 191 071, 100 000 000]     unknown                      unknown                      878 423 826
114349209944065 B→C               [97 243 524, 100 000 000]    unknown                      [1 698 550, 6 086 572]       unknown                      938 012 264
114349209944065 C→B               [0, 2 756 476]               unknown                      [93 913 428, 98 301 450]     unknown                      34 367 637
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Known directions                   18/20                        8/20                         8/20                         0/20

For performance testing I had to expose the scoring data (scorer_channel_liquidity).
Also exposed scoring_fee_params: ProbabilisticScoringFeeParameters to Config.

TODOs:

adjust default parameters
improve HighDegree strategy to take into account channel capacities
improve HighDegree strategy to cache results
improve performance test to better estimate real channel capacity

ldk-reviews-bot · 2026-03-01T11:20:38Z

👋 Thanks for assigning @tnull as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

ldk-reviews-bot · 2026-03-04T07:57:35Z

🔔 1st Reminder

Hey @tnull! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot · 2026-03-07T00:00:29Z

🔔 2nd Reminder

Hey @tnull! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot · 2026-03-09T00:01:28Z

🔔 3rd Reminder

Hey @tnull! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot · 2026-03-11T00:02:29Z

🔔 4th Reminder

Hey @tnull! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot · 2026-03-14T00:01:07Z

🔔 5th Reminder

Hey @tnull! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot · 2026-03-16T00:01:29Z

🔔 6th Reminder

Hey @tnull! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

enigbe

Hi @randomlogin, thanks for the work on this! I've reviewed the first two commits:

7f3ce11: "Create probing service" and
6574bf9: "Add probing tests"

I've left a bunch of inline comments addressing configuration and public API, commit hygiene, testing infrastructure, and test flakiness.

In summary:

A couple of items are exposed publicly that seem like they should be scoped to probing or gated for tests only (see scoring_fee_params in Config and scorer_channel_liquidity on Node).
The probing tests duplicate existing test helpers (setup_node, MockLogFacadeLogger). Reusing and extending what's already in tests/common/ would reduce duplication and keep the test file focused on the tests themselves.
test_probe_budget_blocks_when_node_offline has a race condition where the prober dispatches probes before the baseline capacity is measured, causing the assertion between the baseline and stuck capacities to fail. Details in the inline comment.
A few nits about commit hygiene, import structure, and suggestions for renaming stuff.

Also needs to be rebased.

src/builder.rs

src/probing.rs

enigbe · 2026-03-13T21:39:58Z

src/probing.rs

+pub struct HighDegreeStrategy {
+    network_graph: Arc<Graph>,
+    /// How many of the highest-degree nodes to cycle through.
+    pub top_n: usize,


Could top_n be renamed to num_top_nodes? The latter reads less generic to me but up to you to modify or not.

I'd leave it as is (maybe top_k, as somehow it is more common in algorithms to describe the number of samplings).

What about top_node_count?

^{Personally I don't like 'num' as a short for 'number'}

src/probing.rs

src/builder.rs

tests/probing_tests.rs

ldk-reviews-bot · 2026-03-18T00:01:49Z

🔔 7th Reminder

Hey @tnull! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

randomlogin · 2026-03-18T03:18:49Z

@enigbe, thanks for a review, the updates are incoming soon.

ldk-reviews-bot · 2026-03-21T00:01:02Z

🔔 8th Reminder

Hey @tnull @enigbe! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot · 2026-03-21T02:11:16Z

🔔 1st Reminder

Hey @tnull @enigbe! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot · 2026-03-23T00:01:52Z

🔔 9th Reminder

Hey @tnull @enigbe! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot · 2026-03-23T02:12:07Z

🔔 2nd Reminder

Hey @tnull @enigbe! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot · 2026-03-25T00:02:08Z

🔔 10th Reminder

Hey @tnull @enigbe! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot · 2026-03-25T02:13:01Z

🔔 3rd Reminder

Hey @tnull @enigbe! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

Introduce a background probing service that periodically dispatches probes to improve the scorer's liquidity estimates. Includes two built-in strategies.

tnull

Thanks for taking this on and excuse the delay here!

Did a first review pass and this already looks great! Here are some relatively minor comments, mostly concerning the API design.

src/builder.rs

tnull · 2026-03-26T10:47:32Z

src/builder.rs

 	}

-	let scoring_fee_params = ProbabilisticScoringFeeParameters::default();
+	let mut scoring_fee_params = ProbabilisticScoringFeeParameters::default();


I do wonder if we should allow the user to set the entire ProbabilisticScoringFeeParameters and ProbabilisticScoringDecayParameters via the ProbingConfigBuilder mentioned above? Do you see any reason where that would conflict with other API design decisions?

I think we should expose these settings (in NodeBuilder, not probing builder).
However these are for fine tuning and should be used only by advanced users. Also I would expose UserConfig, as for example user cannot decide what features do advertise.

I can add builder methods for scoring parameters, though maybe it should be in another PR aimed on exposing settings for an advanced user?

Also I would expose UserConfig, as for example user cannot decide what features do advertise.

Yes, this is very intentional, as we want to provide a sane/safe API. For example letting user freely choose to set certain parameters if we don't implement them properly will just lead to a lot of footguns, in some circumstances even with the potential for money loss.

In that case I think we don't need to expose ProbabilisticScoringFeeParameters and ProbabilisticScoringFeeParameters by the very reason you mentioned.

src/probing.rs

tests/common/mod.rs

tests/probing_tests.rs

ldk-reviews-bot · 2026-03-28T00:00:26Z

🔔 4th Reminder

Hey @enigbe! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

Change cursor of top nodes from HighDegreeStrategy to use cac: Create src/util.rs Add probe HTLC maximal lower bound Fix styling (config argument order), explicit Arc::clone instead of .clone() Change tests open_channel to reuse existing code

ldk-reviews-bot · 2026-03-30T00:01:07Z

🔔 5th Reminder

Hey @tnull @enigbe! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

tnull

Seems tests are failing right now:

thread 'exhausted_probe_budget_blocks_new_probes' (167312) panicked at tests/probing_tests.rs:381:5:
no probe dispatched within 15 s


failures:
    exhausted_probe_budget_blocks_new_probes
    probe_budget_increments_and_decrements

randomlogin · 2026-03-31T15:55:02Z

2 of macOs builds has hanged, but not on probing tests; they have passed in another macOs build.

ldk-reviews-bot · 2026-04-01T00:01:59Z

🔔 6th Reminder

Hey @tnull @enigbe! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

tnull · 2026-04-02T12:40:20Z

src/builder.rs

 use crate::message_handler::NodeCustomMessageHandler;
 use crate::payment::asynchronous::om_mailbox::OnionMessageMailbox;
 use crate::peer_store::PeerStore;
+use crate::probing;


nit: Please import the objects you're using directly, also avoiding the probing:: prefix in docs.

tnull · 2026-04-02T12:44:37Z

src/builder.rs

+			logger: Arc::clone(&logger),
+			strategy,
+			interval: probing_cfg.interval,
+			liquidity_limit_multiplier: Some(config.probing_liquidity_limit_multiplier),


I do wonder if we should move the probing_liquidity_limit_multiplier config field to ProbingConfig now?

tnull · 2026-04-02T12:45:35Z

src/event.rs

 		keys_manager: Arc<KeysManager>, static_invoice_store: Option<StaticInvoiceStore>,
 		onion_messenger: Arc<OnionMessenger>, om_mailbox: Option<Arc<OnionMessageMailbox>>,
-		runtime: Arc<Runtime>, logger: L, config: Arc<Config>,
+		runtime: Arc<Runtime>, logger: L, config: Arc<Config>, prober: Option<Arc<Prober>>,


Please move the prober after om_mailbox here, as runtime, logger, and config are usually the trailing fields in most constructors, etc.

tnull · 2026-04-02T12:47:34Z

tests/probing_tests.rs

+#[tokio::test(flavor = "multi_thread")]
+async fn probe_budget_increments_and_decrements() {
+	let (bitcoind, electrsd) = setup_bitcoind_and_electrsd();
+	let chain_source = TestChainSource::Electrum(&electrsd);


Please use random_chain_source() for all tests that aren't chainsource specific.

tnull · 2026-04-02T12:54:36Z

src/lib.rs

 mod message_handler;
 pub mod payment;
 mod peer_store;
+mod probing;


Let's make this a pub mod and expose Prober and related objects under crate::probing, not at the root level. Note that for uniffi to still build you might need to add corresponding re-exports to the list in the beginning of ffi/types.rs, as uniffi indeed expects all objects to live under the crate root.

tnull

Also asked Claude to do a review, some of this might be worth addressing:

 1. Bug: locked_msat accounting mismatch for Probe::Destination (most significant)

  In run_prober (src/probing.rs:593-609), for a Destination probe, amount_msat is added to locked_msat once. But send_spontaneous_preflight_probes
  can dispatch multiple probe paths. Each resolved path triggers a separate ProbeSuccessful/ProbeFailed event, and
  handle_probe_successful/handle_probe_failed each subtract path.hops.iter().map(|h| h.fee_msat).sum() — which is the amount plus routing fees for
  that path.

  Two problems compound:
  - Multiple paths: added once, subtracted N times (one per path event). With saturating_sub this silently clamps to 0, but the budget will appear
  to clear prematurely.
  - Fee mismatch: even for a single path, the subtracted amount includes routing fees (amount_msat + fees), while only amount_msat was added. So
  the subtraction is always ≥ what was added.

  The PrebuiltRoute variant is fine — it consistently uses path.hops.iter().map(|h| h.fee_msat).sum() on both sides.

  2. Bug: Fee calculation in RandomStrategy::try_build_path uses final amount instead of forwarded amount

  At src/probing.rs:462:
  let fee = update.fees.base_msat as u64
      + (amount_msat * update.fees.proportional_millionths as u64 / 1_000_000);

  This computes each intermediate hop's fee based on the delivery amount_msat, but each hop actually charges fees on the amount it forwards
  (delivery + all downstream fees). For routes > 2 hops, this underestimates intermediate fees. The total_outgoing check at the end could then let
  through routes that are actually too large for the first-hop HTLC limit. This is minor for probing purposes since probes fail deliberately, but
  it's still incorrect.

  3. Doc inconsistency on HighDegreeStrategy

  The struct-level doc comment says "Returns None (skips the tick) if all top nodes are on cooldown." But the implementation at line 276-286 does
  the opposite — when all top nodes are on cooldown, it clears the cooldown map and returns the first node, never returning None on cooldown
  exhaustion. The None return only happens when the graph is empty.

  4. random_range overflow edge case

  In src/util.rs:14, let range = max - min + 1 overflows to 0 when max - min == u64::MAX, causing a division-by-zero panic at u64::MAX % range.
  This can't happen with current callers, but the function has no guard against it.

  5. Race condition in budget check (minor/theoretical)

  In run_prober, the budget check (locked_msat.load() + amount > max_locked_msat) and the subsequent fetch_add are not atomic. Since the prober
  loop is single-threaded and the event handler only decreases locked_msat, this can't cause the budget to be exceeded — the worst case is a
  slightly stale read that rejects a probe that would have fit. So this is benign, but worth noting.

  6. scorer_channel_liquidity is very expensive

  src/lib.rs:1090-1108 serializes the entire scorer to a Vec<u8> and then deserializes it as a ProbabilisticScorer just to call
  estimated_channel_liquidity_range. This is O(scorer size) per call. It's only used in tests currently, but it's a public API — worth a doc
  warning or a #[cfg(test)] gate if it's not intended for production use.

  ---
  Issues #1 and #2 are real bugs. #1 in particular means the budget tracking for Destination probes is systematically wrong.

randomlogin marked this pull request as ready for review March 1, 2026 12:10

tnull mentioned this pull request Mar 2, 2026

Bot seems to be broken lightningdevkit/ldk-bot#8

Closed

tnull self-requested a review March 2, 2026 07:56

enigbe suggested changes Mar 16, 2026

View reviewed changes

randomlogin force-pushed the add-probing-service branch 3 times, most recently from 436e4a3 to 07dfde4 Compare March 19, 2026 01:24

randomlogin requested a review from enigbe March 19, 2026 02:10

randomlogin added 3 commits March 26, 2026 02:26

Add probing service

963aa9d

Introduce a background probing service that periodically dispatches probes to improve the scorer's liquidity estimates. Includes two built-in strategies.

Fix uniffi and docs

3571c5e

Add short descriptions to probing tests

c31f1ce

randomlogin force-pushed the add-probing-service branch from ff741c2 to c31f1ce Compare March 26, 2026 01:27

Add uniffi support of probing

200de80

tnull reviewed Mar 26, 2026

View reviewed changes

randomlogin added 2 commits March 28, 2026 16:13

Add dedicated probing builder

83ae595

Change cursor of top nodes from HighDegreeStrategy to use cac: Create src/util.rs Add probe HTLC maximal lower bound Fix styling (config argument order), explicit Arc::clone instead of .clone() Change tests open_channel to reuse existing code

Fix formatting

ebb6227

randomlogin requested a review from tnull March 28, 2026 15:19

tnull reviewed Mar 30, 2026

View reviewed changes

Fix probing tests

1e73e6e

randomlogin force-pushed the add-probing-service branch from f99786b to 1e73e6e Compare March 31, 2026 12:51

Increase probing test timeout

c470da6

randomlogin force-pushed the add-probing-service branch from 6841a0c to c470da6 Compare March 31, 2026 14:25

randomlogin requested a review from tnull March 31, 2026 15:54

tnull reviewed Apr 2, 2026

View reviewed changes

Conversation

randomlogin commented Mar 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ldk-reviews-bot commented Mar 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ldk-reviews-bot commented Mar 4, 2026

Uh oh!

ldk-reviews-bot commented Mar 7, 2026

Uh oh!

ldk-reviews-bot commented Mar 9, 2026

Uh oh!

ldk-reviews-bot commented Mar 11, 2026

Uh oh!

ldk-reviews-bot commented Mar 14, 2026

Uh oh!

ldk-reviews-bot commented Mar 16, 2026

Uh oh!

enigbe left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ldk-reviews-bot commented Mar 18, 2026

Uh oh!

randomlogin commented Mar 18, 2026

Uh oh!

ldk-reviews-bot commented Mar 21, 2026

Uh oh!

ldk-reviews-bot commented Mar 21, 2026

Uh oh!

ldk-reviews-bot commented Mar 23, 2026

Uh oh!

ldk-reviews-bot commented Mar 23, 2026

Uh oh!

ldk-reviews-bot commented Mar 25, 2026

Uh oh!

ldk-reviews-bot commented Mar 25, 2026

Uh oh!

tnull left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ldk-reviews-bot commented Mar 28, 2026

Uh oh!

ldk-reviews-bot commented Mar 30, 2026

Uh oh!

tnull left a comment

Choose a reason for hiding this comment

Uh oh!

randomlogin commented Mar 31, 2026

randomlogin commented Mar 1, 2026 •

edited

Loading

ldk-reviews-bot commented Mar 1, 2026 •

edited

Loading

enigbe left a comment •

edited

Loading