Add Prometheus alerting rule for agent loop consecutive failures

[robertocarlous]

Summary

The agent loop can silently fail repeatedly without triggering any alert. An alert rule on consecutive agent loop errors would page on-call before user funds are affected.

Proposed Solution

Add to deploy/monitoring/prometheus/alert-rules.yaml:

- alert: AgentLoopConsecutiveFailures
  expr: increase(agent_loop_errors_total[5m]) > 3
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "Agent loop has failed {{ $value }} times in 5 minutes"

Also add:

• AgentLoopStalled: no successful loop tick in > 10 minutes
• DLQDepthHigh: DLQ depth > 50 unprocessed events

Acceptance Criteria

• Alert rules added and validated with promtool check rules
• Alerts fire correctly in a local Prometheus instance
• Runbook URL added to each alert annotation
• CI step added to validate alert rules in k8s-validate.yml

[Johnpii1]

@Johnpii1 has applied to work on this issue as part of the Stellar Wave Program's 6th wave.

Hello Sir,

I came across your project and I’m really interested in contributing.

I have experience with React for building responsive, user-friendly interfaces, and I’m also familiar with smart contracts and blockchain basics. I focus on writing clean, scalable code and improving user experience.

I’m a fast learner and ready to get started immediately. Please let me know the next steps.

Thank you.

Best regards,
johnpaul

ℹ️ Repo Maintainers: To accept this application, review their application or assign @Johnpii1 to this issue.

[oyinade247]

@oyinade247 has applied to work on this issue as part of the Stellar Wave Program's 6th wave.

Allow me fix this ASAP

ℹ️ Repo Maintainers: To accept this application, review their application or assign @oyinade247 to this issue.

[drips-wave]

Congratulations, @oyinade247! 🎉 Your application was accepted by the repo's maintainers, and the issue is due on June 30, 2026.

🧑‍💻 @oyinade247: Please resolve the issue such that the repo's maintainers have enough time to review your contribution before the due date. You'll earn Points for completing the issue on-time, which will make you eligible for a share of the Stellar Wave Program's reward pool.

Warning

When opening a PR, please link it to this issue to ensure it gets tracked accurately. Points are awarded when this issue is marked as completed by the maintainer.

🤠 Repo maintainers: Please keep an eye on the contributor's progress and review their work before the due date. You can manage this issue, including adjusting its complexity and points, here.

🌊 Happy Wave 🌊