Skip to content

Installed gateway daemons ignore the fatal-config exit code and crash-loop on bad config #3160

Description

@MervinPraison

Summary

The gateway process is careful to signal its restart intent to a supervisor: it exits 78 (EX_CONFIG) for a fatal/unfixable configuration error ("stop restarting me") and 75 (EX_TEMPFAIL) for a transient failure ("please restart me"). But the daemon units the SDK installs — systemd, launchd and the Windows scheduled task — all restart unconditionally, so the 78 signal is discarded and a genuinely broken config produces an infinite crash-loop (a fresh process every ~5 seconds) instead of a clean stop. This defeats the restart-intent contract the runtime already implements and turns an operator typo into a log-flooding, API-hammering loop that only stops when a human notices.

Current behaviour

The runtime emits the intent-carrying exit codes correctly:

# src/praisonai-bot/praisonai_bot/cli/features/gateway.py  (~L126, L194, L256)
#   * 75 (EX_TEMPFAIL) — transient failure; ask the supervisor to restart
#   * 78 (EX_CONFIG)   — fatal configuration error (do not restart)
...
        return GATEWAY_FATAL_CONFIG_EXIT_CODE       # 78
...
        except FatalConfigError as e:
            ...

Core even ships the classifier and a restart-loop guard for exactly this purpose (GATEWAY_FATAL_CONFIG_EXIT_CODE, GATEWAY_RESTART_EXIT_CODE, classify_exit_reason, RestartLoopGuard in praisonaiagents/gateway/protocols.py).

But the generated supervisor units throw the signal away:

# src/praisonai-bot/praisonai_bot/daemon/systemd.py  (~L44)
ExecStart={python} -m praisonai_bot gateway start --config {abs_config}
Restart=always      # ← ignores exit code; restarts even on 78 (fatal config)
RestartSec=5
<!-- src/praisonai-bot/praisonai_bot/daemon/launchd.py  (~L63) -->
<key>KeepAlive</key>
<true/>              <!-- ← always relaunch, regardless of exit status -->

The Windows backend (src/praisonai-bot/praisonai_bot/daemon/windows.py, scheduled task PraisonAIGateway) likewise has no exit-code-aware stop condition.

Net effect: Restart=always / KeepAlive=true restart the gateway even when it just told the OS "my config is fatally broken, do not restart me". The operator sees a process that flaps forever, each boot re-parsing the same broken config and (depending on where it fails) re-hitting provider/channel APIs.

Desired behaviour

The installed daemons should honour the exit-code contract the runtime already speaks:

  • Exit 78 (fatal config) ⇒ the supervisor stops and does not restart, leaving a clear "stopped: fatal config" status for the operator to fix.
  • Exit 75 (transient) and unexpected crashes ⇒ restart, ideally with backoff.
  • Clean exit 0 ⇒ stay stopped.
  • Restart storms are bounded (tie in the existing RestartLoopGuard) so even a mis-mapped case cannot hammer the machine.

Layer placement

  • Primary layer: wrapper (praisonai-bot) — the daemon unit generators own the OS supervision policy and are where the exit-code mapping must be written.
  • Why not core: core already provides the contract (GATEWAY_FATAL_CONFIG_EXIT_CODE, classify_exit_reason, RestartLoopGuard); nothing is missing there — this is purely about the wrapper's generated units consuming it.
  • Why not tools: daemon/service management is not an agent-callable integration.
  • Why not plugins: OS service installation is core wrapper operator tooling, not an opt-in lifecycle hook.
  • Secondary touch: none required (core contract already exists).
  • 3-way surface (CLI + YAML + Python): no — this is service-install output, surfaced via praisonai gateway install.

Proposed approach

  • Extension point: the unit templates in daemon/{systemd,launchd,windows}.py, mapping the existing core exit codes to per-OS restart directives.
# systemd (sketch)
Restart=on-failure
RestartSec=5
SuccessExitStatus=0
RestartPreventExitStatus=78    # GATEWAY_FATAL_CONFIG_EXIT_CODE — do not restart on fatal config
StartLimitIntervalSec=60
StartLimitBurst=5              # bounded restarts, mirrors RestartLoopGuard
<!-- launchd (sketch): crash-only relaunch, and stop on fatal config -->
<key>KeepAlive</key>
<dict>
  <key>SuccessfulExit</key><false/>
  <key>Crashed</key><true/>
</dict>
<!-- plus a wrapper that maps exit 78 to a non-relaunch stop -->

Resolution sketch

# Before (today): fatal config → infinite crash-loop
$ praisonai gateway install && journalctl --user -u praisonai-bot -f
gateway: EX_CONFIG (78) fatal configuration error
<5s later> gateway: EX_CONFIG (78) fatal configuration error
<5s later> gateway: EX_CONFIG (78) fatal configuration error   # forever

# After (proposed): fatal config → stop, wait for the operator
$ praisonai gateway install && journalctl --user -u praisonai-bot -f
gateway: EX_CONFIG (78) fatal configuration error
systemd: praisonai-bot.service: start request repeated too quickly / not restarting (RestartPreventExitStatus=78)
$ praisonai gateway status
  daemon: stopped (fatal config — fix gateway.yaml and re-start)

Severity

High — an unattended 24/7 daemon is the intended production deployment. A crash-looping gateway floods logs, wastes CPU, and can repeatedly hit LLM/channel APIs on each boot attempt, all from a single bad edit — while the runtime is already signalling the correct "stop" intent that the units silently ignore.

Validation

  • src/praisonai-bot/praisonai_bot/cli/features/gateway.py — returns GATEWAY_FATAL_CONFIG_EXIT_CODE (78) on FatalConfigError and documents 75/78 semantics (~L126, L194, L256).
  • src/praisonai-bot/praisonai_bot/daemon/systemd.py:45Restart=always, no RestartPreventExitStatus.
  • src/praisonai-bot/praisonai_bot/daemon/launchd.py:63KeepAlive is <true/> (unconditional relaunch).
  • src/praisonai-bot/praisonai_bot/daemon/windows.py — scheduled task with no exit-code-aware stop.
  • praisonaiagents/gateway/protocols.pyGATEWAY_FATAL_CONFIG_EXIT_CODE, classify_exit_reason, RestartLoopGuard already exist and are unused by the unit generators.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingclaudeAuto-trigger Claude analysisenhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions