Skip to content

fix: unify simulated cache handling and expose usage to proxy clients#648

Open
zhoudashuaibi wants to merge 2 commits into
fawney19:mainfrom
zhoudashuaibi:pr-fix-cache-and-quota
Open

fix: unify simulated cache handling and expose usage to proxy clients#648
zhoudashuaibi wants to merge 2 commits into
fawney19:mainfrom
zhoudashuaibi:pr-fix-cache-and-quota

Conversation

@zhoudashuaibi

Copy link
Copy Markdown

This PR addresses two related issues we hit while running Aether in production:

  1. Unlimited wallet balance was still rejected after the wallet state was updated
    When a key's wallet was initially cached as BalanceDenied and then changed to unlimited balance mode, the gateway kept returning the stale cached rejection. This change makes the refresh logic re-check wallet rejections so that unlimited wallets are allowed through instead of being blocked by a stale BalanceDenied state.

  2. Simulated cache was not applied consistently and was invisible to proxy clients
    The existing kiro simulated cache only covered a subset of providers. We added a provider-level simulated_cache configuration that applies to all providers and overrides any native cache reporting when enabled. We also wired the simulated cache hit/miss counts through to proxy clients via response metadata.

What's changed

  • apps/aether-gateway/src/control/auth/resolution.rs

    • Re-validate wallet rejections during auth context refresh so unlimited wallets can recover from a previous BalanceDenied state.
    • Add test coverage for the unlimited-wallet recovery path.
  • apps/aether-gateway/src/execution_runtime/kiro_cache.rs

    • Add SimulatedCacheConfig and provider-level simulated_cache_enabled / simulated_cache_min_percent / simulated_cache_max_percent context fields.
  • apps/aether-gateway/src/execution_runtime/simulated_cache.rs (new)

    • Centralized simulated cache implementation that forces cache overrides for providers regardless of native cache support.
  • apps/aether-gateway/src/execution_runtime/stream/execution.rs & sync/execution.rs

    • Use simulated cache uniformly and expose cache usage back to proxy clients.
  • crates/aether-usage-runtime/src/request_metadata.rs & write.rs

    • Carry simulated cache usage through usage tracking so clients can see it.
  • Admin provider write/normalize paths and frontend provider form

    • Allow configuring simulated cache parameters on each provider.
  • Usage repositories (memory / postgres / sqlite)

    • Store simulated cache read/write tokens and add test coverage.

Notes for reviewers

  • This PR intentionally combines the unlimited-wallet fix and the simulated-cache unification because the second commit builds on the usage-tracking changes introduced in the first one.
  • We have been running this internally for a few days and it resolves both the "insufficient balance" false positives for unlimited wallets and the inconsistent cache reporting across providers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant