Skip to content

chore: bound node transaction-event ledger growth#1406

Merged
MicBun merged 5 commits into
mainfrom
feat/orderbook-tx-events
Jul 13, 2026
Merged

chore: bound node transaction-event ledger growth#1406
MicBun merged 5 commits into
mainfrom
feat/orderbook-tx-events

Conversation

@MicBun

@MicBun MicBun commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

resolves: https://github.com/trufnetwork/trufscan/issues/184

The node's main.transaction_events fee ledger — and its CASCADE child transaction_event_distributions — accumulates a row per fee event forever, in consensus state replicated to every validator and captured in snapshots/statesync. There is no retention today. The dominant writer is insert_records (flat 1 TRUF per write, method 2), so the table grows unbounded.

Change

Adds a leader-scheduled retention trim, mirroring trim_order_events / the tn_digest scheduler:

  • trim_transaction_events($preserve_blocks, $delete_cap) (migration 052) — a deterministic (block-height cutoff, never wall-clock), delete-capped prune of the high-volume method-2 write-fee rows only. Low-volume, semantically-rich classes (deployStream, setTaxonomies, requestAttestation, setMetadata, createMarket) are left untouched so their fee/distribution history is preserved. The CASCADE removes each pruned row's distribution child.
  • Wired into the tn_digest scheduler drain loop (after trim_order_events), gated by TrimTxEventsEnabled — which ships false so a binary rollout is decoupled from actually pruning.

Why method-2 only

insert_records charges a flat 1 TRUF, so its rows are uniform and the transaction stays permanently resolvable off-chain via the Trufscan chain indexer. Keeping the low-volume classes preserves the only node-computed fee/distribution data the indexer does not have.

Verification

  • kwil-cli utils parse on 052.
  • go build ./..., go vet ./extensions/tn_digest/..., gofmt clean.
  • TestTrimTransactionEvents (selectivity, cascade, delete-cap drain, young-chain no-op) passes; existing TestTransactionEventsLedger / TestTransactionIDTracking and tn_digest engine-op tests still pass.

Rollout

  • Consensus-affecting: a live network needs 052 applied and the binary rolled out in lockstep.
  • Ships inert. Activate (TrimTxEventsEnabled = true) only after the Trufscan indexer fallback is live in prod, so a pruned tx still resolves on the explorer /tx page.

Related

Summary by CodeRabbit

  • New Features

    • Added an automated trim_transaction_events retention action to prune eligible high-volume write-fee transaction-event records while keeping recent activity.
    • Introduced configurable trim parameters (preserve window, delete cap, max runs) with rollout gated enablement (disabled by default).
    • Scheduler now runs this trim step after existing order-event draining, in bounded batches with retries and early stop when finished.
  • Tests

    • Added unit tests for log parsing, integration tests verifying transaction build/sign and database effects, and end-to-end tests covering selective pruning, cascade cleanup, batching limits, and no-op behavior.

@holdex

holdex Bot commented Jul 13, 2026

Copy link
Copy Markdown

Time Submission Status

Member # Time Running Total Status Last Update
MicBun 4h ✅ Submitted Jul 13, 2026, 11:45 AM

Submit or update total time with:

@holdex pr submit-time 2h

Add time on top of previous submission with:

@holdex pr add-time 1h30m

See available commands to help comply with our Guidelines.

@coderabbitai

coderabbitai Bot commented Jul 13, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

Adds a trim_transaction_events schema action, engine broadcaster and result parser, scheduler integration with rollout controls, and coverage for selective deletion, cascades, caps, cutoff handling, transaction construction, and database effects.

Changes

Transaction event retention trimming

Layer / File(s) Summary
Retention action contract and deletion logic
internal/migrations/052-transaction-events-retention.sql
Defines cutoff-based, capped deletion for targeted transaction events and emits deletion results through a NOTICE.
Engine broadcast and result parsing
extensions/tn_digest/internal/engine_ops.go, extensions/tn_digest/internal/engine_ops_test.go
Builds, signs, retries, and broadcasts trim transactions, then parses deletion results with unit coverage for successful, empty, and missing log entries.
Scheduler integration and rollout controls
extensions/tn_digest/scheduler/constants.go, extensions/tn_digest/scheduler/scheduler.go
Adds configurable retention constants and a disabled-by-default bounded trim loop after digest draining.
Transaction and retention validation
extensions/tn_digest/engine_ops_integration_test.go, tests/streams/transaction_events_ledger_test.go
Verifies transaction construction, engine execution, selective pruning, cascade deletion, delete caps, cutoff no-ops, and end-to-end ledger trimming.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant DigestScheduler
  participant EngineOperations
  participant trim_transaction_events
  participant transaction_events
  DigestScheduler->>EngineOperations: Request bounded trim
  EngineOperations->>EngineOperations: Read fresh nonce, build, and sign transaction
  EngineOperations->>trim_transaction_events: Broadcast preserve_blocks and delete_cap
  trim_transaction_events->>transaction_events: Delete capped rows below cutoff
  trim_transaction_events-->>EngineOperations: Return NOTICE summary
  EngineOperations-->>DigestScheduler: Return deletion result
Loading

Possibly related PRs

Suggested reviewers: outerlook, pr-time-tracker

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 45.45% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title is concise and accurately describes the main change: adding retention trimming to bound transaction-event ledger growth.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/orderbook-tx-events

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@internal/migrations/032-order-book-actions.prod.sql`:
- Around line 335-337: Restore the missing place_sell_order action in the
production migration alongside the existing buy and split-limit paths, matching
the implementation from 032-order-book-actions.sql. Ensure it includes the
explicit method-10 transaction ledger entry via record_transaction_event,
preserving the zero protocol-fee values and sell-order behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 84448196-846c-4c8f-af10-21f20609f04e

📥 Commits

Reviewing files that changed from the base of the PR and between d5762f3 and ffb8f4a.

📒 Files selected for processing (3)
  • internal/migrations/030-order-book-schema.sql
  • internal/migrations/032-order-book-actions.prod.sql
  • internal/migrations/032-order-book-actions.sql

Comment thread internal/migrations/032-order-book-actions.prod.sql Outdated
@MicBun MicBun self-assigned this Jul 13, 2026
@MicBun MicBun marked this pull request as draft July 13, 2026 09:43
@MicBun MicBun force-pushed the feat/orderbook-tx-events branch from 7b36160 to 15c2cb8 Compare July 13, 2026 10:51
@MicBun MicBun changed the title feat: record order-book placements in the transaction ledger feat: bound node transaction-event ledger growth Jul 13, 2026
@MicBun MicBun marked this pull request as ready for review July 13, 2026 11:44
@MicBun

MicBun commented Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

@holdex pr submit-time 4h

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
extensions/tn_digest/internal/engine_ops.go (1)

615-761: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Near-duplicate of the order-events trim plumbing. The transaction-events broadcast/retry/parse functions and the scheduler drain loop mirror the order-events equivalents almost verbatim; extracting a shared generic helper (parameterized by action name, arg encoders, and NOTICE parser) would avoid this duplication growing with each future trim_X_events addition.

  • extensions/tn_digest/internal/engine_ops.go#L615-L761: factor BroadcastTrimTransactionEventsWithRetry, broadcastTrimTransactionEventsOnce, and parseTxEventsTrimResultFromTxLog into a shared generic/parameterized implementation with the order-events equivalents.
  • extensions/tn_digest/scheduler/scheduler.go#L276-L333: factor trimTransactionEvents's drain loop into a shared helper with trimOrderEvents (227-274), parameterized by the trim function and constants.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@extensions/tn_digest/internal/engine_ops.go` around lines 615 - 761, The trim
transaction-events implementation duplicates the order-events plumbing and
scheduler drain logic. In extensions/tn_digest/internal/engine_ops.go:615-761,
refactor BroadcastTrimTransactionEventsWithRetry,
broadcastTrimTransactionEventsOnce, and parseTxEventsTrimResultFromTxLog into
shared parameterized helpers reused by the order-events equivalents, covering
action names, argument encoding, and NOTICE parsing. In
extensions/tn_digest/scheduler/scheduler.go:276-333, extract the
trimTransactionEvents drain loop into a shared helper with trimOrderEvents,
parameterized by the trim function and relevant constants.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@internal/migrations/052-transaction-events-retention.sql`:
- Around line 47-57: Update the cutoff<=0 branch in the trim_transaction_events
migration to emit the parser-compatible NOTICE format, using deleted=0,
remaining=0, and has_more=false rather than the cutoff message. Preserve the
existing early return so parseTxEventsTrimResultFromTxLog and
broadcastTrimTransactionEventsOnce recognize the successful no-op without
retries.

---

Nitpick comments:
In `@extensions/tn_digest/internal/engine_ops.go`:
- Around line 615-761: The trim transaction-events implementation duplicates the
order-events plumbing and scheduler drain logic. In
extensions/tn_digest/internal/engine_ops.go:615-761, refactor
BroadcastTrimTransactionEventsWithRetry, broadcastTrimTransactionEventsOnce, and
parseTxEventsTrimResultFromTxLog into shared parameterized helpers reused by the
order-events equivalents, covering action names, argument encoding, and NOTICE
parsing. In extensions/tn_digest/scheduler/scheduler.go:276-333, extract the
trimTransactionEvents drain loop into a shared helper with trimOrderEvents,
parameterized by the trim function and relevant constants.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 16261731-eef7-438e-8743-e52beea258a1

📥 Commits

Reviewing files that changed from the base of the PR and between ffb8f4a and be6372a.

📒 Files selected for processing (7)
  • extensions/tn_digest/engine_ops_integration_test.go
  • extensions/tn_digest/internal/engine_ops.go
  • extensions/tn_digest/internal/engine_ops_test.go
  • extensions/tn_digest/scheduler/constants.go
  • extensions/tn_digest/scheduler/scheduler.go
  • internal/migrations/052-transaction-events-retention.sql
  • tests/streams/transaction_events_ledger_test.go

Comment thread internal/migrations/052-transaction-events-retention.sql
@MicBun MicBun changed the title feat: bound node transaction-event ledger growth chore:: bound node transaction-event ledger growth Jul 13, 2026
@MicBun MicBun changed the title chore:: bound node transaction-event ledger growth chore: bound node transaction-event ledger growth Jul 13, 2026
@MicBun MicBun merged commit b548cf0 into main Jul 13, 2026
9 of 10 checks passed
@MicBun MicBun deleted the feat/orderbook-tx-events branch July 13, 2026 19:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant