feat: Indexer reorg / gap detection & automatic reconciliation

[joelpeace48-cell]

Background & current state

The indexer (NEW-024) ingests events from Soroban RPC, but transient RPC errors, endpoint switches (rpcPool.js), or rollbacks can produce gaps or duplicate/inconsistent rows. Without reconciliation, downstream leaderboards/analytics drift from chain truth.

Goal

Add reorg/gap detection and automatic reconciliation so the indexed state self-heals after RPC hiccups or ledger inconsistencies.

Technical design

• Track per-ledger ingestion with contiguous-range invariants; on detecting a gap (missing ledger range) or a hash/cursor mismatch, re-fetch and reconcile the affected range idempotently.
• Quarantine unparseable events into a dead_events table for inspection rather than crashing.
• Periodic consistency sweep comparing indexed counts vs. on-chain getters (e.g., get_participant_count).

Edge cases

• Large gap after long downtime → bounded backfill (NEW-028) instead of infinite loop.
• Duplicate delivery → idempotent upserts make reconciliation safe.
• Persistent parse failure → quarantine + alert, keep indexing.
• RPC returns inconsistent data across endpoints → prefer canonical + log divergence.

Task breakdown

• Gap/mismatch detection over the cursor stream.
• Reconcile-range routine (idempotent re-fetch).
• dead_events quarantine + metrics/alerts.
• Consistency sweep vs. on-chain getters.
• Fault-injection tests.

Acceptance criteria

• An injected gap/duplicate is reconciled automatically with no manual intervention.
• Unparseable events are quarantined, not fatal.

Testing & verification

• Fault-injection: drop/duplicate/corrupt ledger ranges; assert eventual consistency.

Out of scope

• Initial indexer build (NEW-024); rollups (NEW-026).

Dependencies / related

• Builds on NEW-024; uses NEW-028 for large gaps.

Difficulty: medium · Effort: M · indexer

[otobongdev]

@otobongdev has applied to work on this issue as part of the Stellar Wave Program's 6th wave.

i want to handle this issue

ℹ️ Repo Maintainers: To accept this application, review their application or assign @otobongdev to this issue.

[Mathew2k-hash]

@Mathew2k-hash has applied to work on this issue as part of the Stellar Wave Program's 6th wave.

I’d like to work on this issue because data correctness and resilience are critical for blockchain indexers, especially when downstream analytics and leaderboards depend on accurate chain state. I have experience building fault-tolerant backend systems, event-processing pipelines, reconciliation workflows, and idempotent data ingestion mechanisms. I can implement robust gap and reorg detection, safe reconciliation routines, dead-letter event handling, and consistency validation against on-chain data while ensuring the indexer continues operating during failures. I’ll also provide comprehensive fault-injection tests to verify eventual consistency and self-healing behavior under real-world failure scenarios.

ℹ️ Repo Maintainers: To accept this application, review their application or assign @Mathew2k-hash to this issue.

[theo372001]

@theo372001 has applied to work on this issue as part of the Stellar Wave Program's 6th wave.

I’d like to work on this issue. Having deeply analyzed the current state of our indexer (NEW-024), I have a clear vision for how to implement this self-healing architecture effectively.

ℹ️ Repo Maintainers: To accept this application, review their application or assign @theo372001 to this issue.

[devjayy43]

@devjayy43 has applied to work on this issue as part of the Stellar Wave Program's 6th wave.

I would like to work on this. please assign

ℹ️ Repo Maintainers: To accept this application, review their application or assign @devjayy43 to this issue.

[osasfaith]

@osasfaith has applied to work on this issue as part of the Stellar Wave Program's 6th wave.

I would love to work on this issue.

ℹ️ Repo Maintainers: To accept this application, review their application or assign @osasfaith to this issue.

[drips-wave]

Congratulations, @devjayy43! 🎉 Your application was accepted by the repo's maintainers, and the issue is due on June 30, 2026.

🧑‍💻 @devjayy43: Please resolve the issue such that the repo's maintainers have enough time to review your contribution before the due date. You'll earn Points for completing the issue on-time, which will make you eligible for a share of the Stellar Wave Program's reward pool.

Warning

When opening a PR, please link it to this issue to ensure it gets tracked accurately. Points are awarded when this issue is marked as completed by the maintainer.

🤠 Repo maintainers: Please keep an eye on the contributor's progress and review their work before the due date. You can manage this issue, including adjusting its complexity and points, here.

🌊 Happy Wave 🌊