fix(inventory): isolate per-adapter, per-bridge, and per-chain failures in inventory accounting

[droplet-rl]

Summary

A transient outage in one adapter/bridge/chain's read API used to propagate up through the inventory update chain and crash the relayer/rebalancer process. This PR isolates failures at three layers so a single venue's read-side outage no longer gates the entire fill loop, while preserving accounting safety.

Motivation

Two recent incidents on the same theme:

1. zion-across-relayer-primary crashed on a 502 from Hyperliquid:

HttpRequestError: 502 Bad Gateway - nginx/1.22.1
  at HyperliquidStablecoinSwapAdapter._getInitiatedWithdrawalsFromHypercore
  at HyperliquidStablecoinSwapAdapter.getPendingRebalances
  at BaseRebalancerClient.getPendingRebalances
  at InventoryClient.update

2. zion-across-fast-relayer-rebalancer crashed on a 503 from the shared BridgeApi:

HttpError: HTTP 503: Service Unavailable
  at BridgeApiClient.getWithRetry
  at BridgeApiClient.getAllTransfersInRange
  at BridgeApi.queryL1BridgeInitiationEvents
  at TokenSplitterBridge.queryL1BridgeInitiationEvents
  at BaseChainAdapter (getOutstandingCrossChainTransfers)

Both bottom out in InventoryClient.update, which had no catch around either the rebalancer-client path or the cross-chain-transfer-accounting path. Treating any one venue's read API as a hard precondition for the whole relay loop is the root problem.

Change

Three layers of isolation, finest-grain inward:

1. BaseRebalancerClient.getPendingRebalances (rebalancer-adapter level)

Wraps each adapter.getPendingRebalances(account) call in try/catch. On failure: log at error (at: \"BaseRebalancerClient#getPendingRebalances\", includes adapter name), skip that adapter for the cycle. Aggregates whatever the healthy adapters returned.

2. BaseChainAdapter.getOutstandingCrossChainTransfers (per-bridge level)

Wraps the (l1Token, monitoredAddress) block in try/catch. The inner Promise.all([queryL1BridgeInitiationEvents, queryL2BridgeFinalizationEvents]) stays all-or-nothing on purpose — partial success there would corrupt outstanding = deposited − finalized accounting — so the catch sits at the (token, address) granularity. On failure: log at error (at: \"<adapterName>#getOutstandingCrossChainTransfers\"), and the affected pair is simply absent from the cycle's result.

3. CrossChainTransferClient.update (per-chain safety net)

Promise.all → Promise.allSettled across chains. Successful chains overwrite with fresh state; failed chains preserve their previously-recorded state rather than being blanked. Stale state biases the InventoryClient toward under-rebalancing (safer) instead of duplicate-bridging (worse). Per-chain failures log at error (at: \"CrossChainTransferClient\"). This is defense-in-depth for any uncaught path inside BaseChainAdapter.

Behavioral effect

• The HL 502 from HyperliquidStablecoinSwapAdapter.getPendingRebalances now logs and skips; pendingRebalances is partial (HL contribution absent) for that update cycle.
• The 503 from TokenSplitterBridge → BridgeApi now logs and skips at the per-bridge level; outstanding-transfer entries for the affected (token, address) pair are absent for the cycle.
• If a whole chain's accounting somehow rejects, CrossChainTransferClient.update preserves the prior cycle's state for that chain rather than blanking it.
• Next cycle retries in all cases.

For a sustained HL outage, the relayer will still attempt to initiate new HL swap rebalances since rebalanceInventory doesn't gate on read-side health. Out of scope here; will likely follow up with read-side health gating on HL rebalance initiation in a separate PR.

Ops note

Silent-by-default is by design (relayer-up beats accounting-perfect), but operators should add Datadog alerts on the new error logs so a persistent outage doesn't quietly degrade inventory accounting:

• at: \"BaseRebalancerClient#getPendingRebalances\"
• at: \"<adapterName>#getOutstandingCrossChainTransfers\" (substring #getOutstandingCrossChainTransfers)
• at: \"CrossChainTransferClient\" with Failed to fetch outstanding cross chain transfers

Test plan

• tsc --noEmit clean
• eslint + prettier clean on touched files
• hardhat test test/RebalancerClient.cumulativeRebalancing.ts test/CrossChainTransferClient.update.ts test/generic-adapters/BaseChainAdapter.isolation.ts test/generic-adapters/SplitBridgeTracking.ts — 25/25 passing
• hardhat test test/InventoryClient.InventoryRebalance.ts test/InventoryClient.RefundChain.ts test/Monitor.ts — 66/66 passing, no regressions
• New unit tests:
  • RebalancerClient.cumulativeRebalancing.ts — \"Isolates per-adapter failures when aggregating pending rebalances\"
  • CrossChainTransferClient.update.ts — three cases: per-chain isolation + stale preservation, fresh replace on success, all-chains-fail does not throw
  • BaseChainAdapter.isolation.ts — \"Isolates a failing bridge so other bridges' outstanding transfers still surface\"
• Operator: confirm the three error logs surface in Datadog under the expected at: values and wire alerts.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f43cf50182

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Document partial pending-rebalance reads

This changes getPendingRebalances from fail-fast aggregation to returning a partial view when any adapter read fails, but the rebalancer README still describes the client as aggregating the account's pending state without noting this failure mode. The repo's AGENTS.md requires relevant README/AGENTS updates whenever behavior or runtime flow changes; without that update, operators and consumers of ReadOnlyRebalancerClient can assume the returned accounting is complete during an adapter outage when it is intentionally incomplete.

Useful? React with 👍 / 👎.

[droplet-rl]

@codex thanks — fair point. The repo's AGENTS.md (line 15) explicitly requires README.md/AGENTS.md updates when runtime behavior changes, and the new partial-aggregation semantics weren't reflected in src/rebalancer/README.md.

Pushed a0d60ba adding a "Partial-view semantics on adapter failure" subsection under Read-only mode: ReadOnlyRebalancerClient. It records:

• BaseRebalancerClient.getPendingRebalances now isolates per-adapter failures, logs at error with at: "BaseRebalancerClient#getPendingRebalances" + adapter name, and returns a partial aggregate from the remaining adapters.
• ReadOnlyRebalancerClient consumers (InventoryClient.update, Monitor) see an intentionally partial view during an adapter outage and operators should alert on the new error log so persistent outages don't quietly degrade inventory accounting.
• Direct adapter callers that bypass BaseRebalancerClient (notably the Binance finalizer in src/finalizer/utils/binance.ts, which calls the Binance adapter directly) are unaffected — their errors still propagate, so the existing "errors surface normally" contract for that path is preserved.

Please re-review.

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Delightful!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[droplet-rl]

Acknowledged — Codex re-review on a0d60ba came back clean. No further changes.

[pxrl]

Is it correct to return undefined here? Should we instead be returning 0 or something, in case the value is dereferenced?

[droplet-rl]

Good question — it's safe: the return; exits only the per-iteration forEachAsync callback, not getPendingRebalances itself.

• forEachAsync in the SDK is await mapAsync(array, callback) which is Promise.all(array.map(callback)) — the callback's return value is discarded, so returning early just skips aggregation for that adapter.
• The outer getPendingRebalances always returns pendingRebalances, which is initialized to {} on entry and only ever gets keys added via pendingRebalances[chainId][token] = ... for adapters whose read succeeded. It is never undefined, and for the skipped adapter the corresponding chainId/token entry is simply absent (rather than zero) — i.e. a partial map.
• Consumers handle that shape naturally:
  • InventoryClient (src/clients/InventoryClient.ts:307-317) reads via pendingRebalances[chainId]?.[l1TokenSymbol] with isDefined guards, so a missing chain/token entry contributes 0 to the balance computation — exactly the same effect a literal bnZero would have, without forcing us to enumerate which chain/token pairs the failed adapter would have reported.
  • Monitor similarly indexes via pendingRebalances[chainId]?.[token] with optional chaining.

So from a caller's perspective the choice is between "missing key" and "key present with value 0" — both are dereferenced safely, and "missing key" has the advantage that we don't have to guess the set of chain/token pairs the failed adapter would have populated (it differs per adapter and per route config). Happy to switch to explicit zero-fill if you'd rather have the dictionary shape be uniform across success/failure cases, but the current form keeps the failure semantics minimal — "this adapter contributed nothing" rather than "this adapter contributed zero to chains X, Y, Z."

[droplet-rl]

Thanks for the review @pxrl — replied to your inline question on the return;. TL;DR: it exits only the per-iteration forEachAsync callback (the SDK's forEachAsync discards callback return values), so the outer getPendingRebalances always returns the initialized pendingRebalances map. Skipped adapters leave their chain/token keys absent rather than zero, and existing consumers (InventoryClient, Monitor) already index with isDefined/optional chaining so a missing entry contributes 0. Happy to switch to explicit zero-fill if you'd prefer uniform shape — flagged that as the alternative in the inline reply.