TxCollection with background coin prefetch - #122
Open
Sjors wants to merge 8 commits into
Open
Conversation
Add the TxCollection interface and the node::TxCollection class that backs it. The constructor looks up each requested wtxid in the mempool and keeps a reference to any transaction that is already present, so later commits can report which requested transactions are still missing and let the client fill them in.
Now that the collection can be mutated after construction, guard the collected transactions with a mutex: IPC clients may call TxCollection methods concurrently from different threads.
Move the coinbase construction out of CreateNewBlock() into a reusable BlockAssembler::CreateCoinbaseTx() helper, so a later commit can build a coinbase when assembling a template from externally collected transactions. This does not change behavior. Review with --color-moved=dimmed-zebra.
Add TxCollection::makeTemplate(), which assembles the collected transactions, in the requested order, into a block that builds on the given prevhash, and validates it. The block is validated by the same TestBlockValidity() call used by checkBlock(), with the proof-of-work and merkle-root checks disabled. A dummy coinbase is added only so the transactions can be validated as part of a block. Its output pays no fees (fees=0), because the total fee amount is not tracked here. Clients construct the real coinbase themselves, which is why the next commit disables getCoinbaseTx() for externally generated templates. When the requested prevhash does not match the active tip, we mirror the getblocktemplate proposal reasons: 'stale-prevblk' when prevhash is an ancestor of the current tip, and 'inconclusive-not-best-prevblk' when it is unknown or on a fork.
Disable methods that are not expected to be used in the externally provided template use case. getTxFees() and getTxSigops() could be implemented, but it's not worth doing since the external software that constructed the template is expected to know this. getCoinbaseTx() is disabled because the dummy coinbase is only added to allow validating the collected transactions as a block. It pays no fees (fees=0) and should not be used by clients, which construct the real coinbase themselves.
Give TxCollection a background thread that loads the input coins of collected transactions into the active chainstate's coins cache, so that by the time the client calls makeTemplate() the TestBlockValidity() check does not need to read them from disk. Work is queued by the constructor and by addMissingTxs(), both of which return without waiting for it: the coins load while the (Stratum v2 Job Declarator) client learns which transactions are missing and transfers them, taking the coin database reads off the template critical path. The thread warms the cache through CCoinsViewCache::HaveCoin() in batches of 128 outpoints per cs_main acquisition, so it never holds up validation for a full block's worth of disk reads. Coins fetched this way are clean cache entries: they do not grow the next flush and can be evicted at any time, and they remain useful when the actual block is connected. Inputs spending other transactions in the collection are skipped, since they have no coin in the UTXO set. Suggested by Sjors Provoost in bitcoin#35751 (comment) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Measure TxCollection::MakeTemplate() for a 1000-transaction collection spending 2000 confirmed coins, with a cold coins cache versus after the background prefetch has completed, plus the cost of the prefetch itself (which in real usage overlaps with the client providing missing transactions). Example results (macOS, Apple Silicon, regtest chainstate): | ns/template | template/s | err% | total | benchmark |--------------------:|--------------------:|--------:|----------:|:---------- | 2,658,500.00 | 376.15 | 3.0% | 0.04 | TxCollectionTemplateColdCoins | 1,506,459.00 | 663.81 | 3.9% | 0.03 | TxCollectionTemplatePrefetchedCoins | ns/collection | collection/s | err% | total | benchmark |--------------------:|--------------------:|--------:|----------:|:---------- | 746,209.00 | 1,340.11 | 2.2% | 0.01 | TxCollectionPrefetch Note that the regtest coins database is tiny and fully resident in the OS page cache; on a mainnet-sized UTXO set with real disk reads the difference is expected to be larger. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sjors
force-pushed
the
2026/03/collect-txs
branch
from
July 24, 2026 11:23
af9a115 to
d0c2077
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Have the
TxCollectionconstructor spin up a background thread to fetch coins from the UTXO set. Meanwhile the Job Declarator Server (IPC client) can queryunknownTxPos()and fetch missing transactions from the miner.This saves a little bit of time when
makeTemplate()validates the template.TxCollectionTemplateColdCoinsTxCollectionTemplatePrefetchedCoinsTxCollectionPrefetch(the part hidden in the client round-trip)A benchmark on mainnet would be more useful, in order to compare this to typical network roundtrip times when fetching missing transactions.
Builds on:
(fully vibe coded proof-of-concept)