Skip to content

sec(auth): retry failed key loads, fail closed without a limiter pool, key rate limits on the client address - #1294

Draft
IAmJSD wants to merge 2 commits into
LeanerCloud:mainfrom
IAmJSD:fix/issue-104
Draft

IAmJSD wants to merge 2 commits into
LeanerCloud:mainfrom
IAmJSD:fix/issue-104

Conversation

@IAmJSD

@IAmJSD IAmJSD commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator

First slice of #104 (medium/low findings from the 2026-07-28 auth review). I re-checked every finding against current main (a14dc8f2). This PR fixes three of them. The rest are either deferred to an in-flight branch that edits the same code, or left for a later slice, with the reason given for each in the table below.

Refs #104

Per-finding status

# Finding (severity) Status on main a14dc8f2 This PR
1 KMS signers cache the public key forever, so rotation never reaches the JWKS (MEDIUM) Still present: resolveOnce/sync.Once in internal/oidc/aws_signer.go and azure_signer.go Deferred to open PR #1152. It replaces resolveOnce with a publicKeyCache in the same lines. Rotation (a TTL, or keying on the resolved version) should be built on that cache once #1152 merges.
1a A03-014: one failed GetPublicKey poisons the signer, GCP included (medium) Still present Deferred to #1152, which fixes this ("sticky KMS signer errors").
2 Invited users get nothing from Forgot Password for the 7-day invite window (MEDIUM) Still present: resetRecentlyIssued, service_password.go:298 Deferred to #1152, which edits the same function (A03-009).
3 Rate-limit client IP: rightmost XFF with no trusted-proxy count, and the ip:port RemoteAddr fallback (MEDIUM) Still present: httpToLambdaRequest, internal/server/http.go Partly fixed. The port is dropped from RemoteAddr (and from an ip:port rightmost XFF entry), and an unparseable rightmost entry falls back to the peer address. Deferred: a configurable trusted-proxy hop count. It needs a per-deployment setting (ingress topology per cloud, plus Terraform/CFN wiring), so the owner should decide the setting's name and default.
3a A02-008: the Lambda function-URL path keys on the CloudFront edge address when enable_cdn = true (medium) Still present Deferred with the hop-count item above; it needs the same trusted-proxy configuration.
4 httpclient.New() drops proxy support (MEDIUM), plus A08b-003 (other DefaultTransport defaults lost) Still present, but the code no longer lives in this repo: pkg/httpclient moved to LeanerCloud/cloud-commitments-go (pkg/httpclient/httpclient.go:169 on its main still builds a bare &http.Transport{}). internal/secrets/azure_resolver.go now just calls httpclient.New(). Deferred, out of repo. The fix belongs in cloud-commitments-go, followed by a go.mod bump here.
5 LoadKey memoizes a transient secret-fetch failure for the process lifetime (LOW) Still present: internal/credentials/cipher.go sync.Once cached cachedKeyErr. This is reachable: ensureDB retries reinitializeAfterConnect on the next request, but that retry got the cached error back. Fixed.
6 Email identity is case-sensitive end to end (LOW), plus A03-019 (RFC-5322 display-name form stored verbatim) Still present Deferred. It needs a migration (a lower(email) unique index or citext) plus a backfill, and the owner has to decide what happens to existing case-variant duplicate rows. The service-layer normalization would also touch service_user.go lines that fork branches fix/issue-313 and fix/issue-312 rewrite.
7 checkRateLimitStrict fails open when the limiter has no pool (latent, LOW) Still present Partly fixed. DBRateLimiter.Allow now returns ErrRateLimiterUnavailable with no pool (or a nil receiver), so the strict path answers 503. Deferred: a nil Handler.rateLimiter treated as 503. I tried it, and it breaks 34 existing handler tests that build &Handler{} with no limiter (login, setup-admin, reset, change-password, MFA, router coverage). Each would need a permissive limiter fixture, which is a test sweep larger than this slice. Production always installs a limiter (app.go:523, :834).
8 One goroutine per API-key-authenticated request (LOW) Changed but still present. Validation no longer books usage. RecordUsageAsync (service_apikeys.go:466) runs once per request, but it still starts a goroutine per request before the singleflight dedupe. Deferred to a later slice. Fork branch fix/issue-313 rewrites the adjacent code in service_apikeys.go, and the fix (start a flusher only when none is in flight for the key) changes concurrency-sensitive accounting, so it deserves its own PR.
9 A02-017: awsCfgOnce loads the AWS config with the request's context (low) Not reproducible with the default loader. awsconfig.LoadDefaultConfig on an already-canceled context returns nil error (checked with a throwaway probe under AWS_CONFIG_FILE=/dev/null), because credentials and region resolve lazily. Deferred. No reachable failure was found. Changing the memoization also means rewriting the cache seeding in 7 test files, including handler_accounts_hostcheck_test.go, which fork branch fix/issue-559 also edits.

#313 correction to #104's "Verified sound" section: #313 (A03-020) points out that the post-fetch subtle.ConstantTimeCompare calls listed as a positive in #104 are inert. Each one compares a row's hash with the same hash the row was selected by. So that bullet should be read as "store-integrity check", not "closes a timing oracle". Fork branch fix/issue-313 rewrites those comments. This PR leaves them alone.

Root cause and fix

  • credentials.LoadKey (internal/credentials/cipher.go): sync.Once stored the error alongside the key. Now a mutex-guarded cache memoizes only a successful load. A failed load returns its error, and the next call retries. The mutex is held across the load, so concurrent callers queue behind one fetch instead of all hitting Secrets Manager / Key Vault at once (no retry stampede). One caller's canceled context fails only that caller's load.
  • DBRateLimiter.Allow (internal/api/db_rate_limiter.go): a nil pool or nil receiver returned (true, nil). It now returns (false, ErrRateLimiterUnavailable). checkRateLimitStrict turns that into 503 on login, setup-admin, reset-password, change-password and MFA. The ordinary checkRateLimit keeps its documented fail-open behavior and logs a warning.
  • Client IP for rate limiting (new internal/server/source_ip.go, called from httpToLambdaRequest): the port is stripped from RemoteAddr and from the rightmost XFF entry (some proxies, for example Azure Application Gateway, append ip:port). If the rightmost entry is not an IP after that, the peer address is used instead of an arbitrary string. The helper lives in its own file because http.go is already over the 500-line guideline. SourceIP is only consumed by the rate limiter and a request log line, so dropping the port changes nothing else.

Verification

All evidence below is unit-level (mocks, fakes and httptest). Nothing here is DB-level: the limiter change only affects the no-pool branch, so I wrote no Postgres integration test. The concurrent case is the LoadKey stampede test, run under -race.

Environment for all runs: AWS_CONFIG_FILE=/dev/null AWS_SHARED_CREDENTIALS_FILE=/dev/null AWS_EC2_METADATA_DISABLED=true, Go 1.26.9.

Fail-before: each new test was run against the unfixed code first.

  • go test -count=1 -run 'TestLoadKey_FailedLoadIsRetried|TestLoadKey_ConcurrentCallers' ./internal/credentials/ failed both tests. The second call got the cached context canceled. With 32 concurrent callers, all 32 failed (expected 1), and the resolver was called once (expected 2).
  • go test -count=1 -run 'TestDBRateLimiter|TestHandler_checkRateLimitStrict' ./internal/api/ failed TestDBRateLimiter_Allow_NilLimiter, all 4 subtests of TestDBRateLimiter_NilPool_ReportsUnavailable, and both subtests of TestHandler_checkRateLimitStrict_NilPoolFailsClosed, because Allow returned (true, nil) and the strict check let the request through. Note: the old *_NilPool tests pinned the fail-open behavior. They are replaced, not deleted.
  • go test -count=1 -run TestHTTPToLambdaRequest_SourceIP ./internal/server/ failed 7 of 10 subcases (203.0.113.7:51234, [2001:db8::1]:443, the XFF ip:port forms, and the unparseable/empty rightmost entry). The 3 cases that already behaved correctly passed.

Pass-after, on head 29abc168:

  • go build ./..., go vet ./..., go vet -tags integration ./...: clean.
  • go test -count=1 on ./internal/credentials/, ./internal/server/, ./internal/api/, ./internal/mocks/ and ./cmd/rekey/: all ok.
  • go test -race -count=1 -run 'TestLoadKey|TestDBRateLimiter|TestHandler_checkRateLimit' ./internal/credentials/ ./internal/api/: ok.
  • golangci-lint v2.10.1 run --allow-parallel-runners on ./internal/credentials/... ./internal/server/ ./internal/api/: 0 issues.
  • gocyclo -over 10 -ignore "_test\.go" on the changed non-test files: no output.

Combined-tree verification. This branch was tested together with the other queued branches on a local, never-pushed tree: queue/platform @ 936996ad = origin/main a14dc8f2 plus these heads, merged in this order:

Branch Tested head
fix/issue-337 ff3420c
fix/issue-169 7d75eb5
fix/issue-342 66b221f
fix/issue-310 5e11c21
fix/issue-326 5f0ea59
fix/issue-243 365e959
fix/issue-343 6c94036
fix/issue-312 d2848a8
fix/issue-316 acd60dd
fix/issue-339 75ea9b9
fix/issue-339-commitment-type 73bdb19
fix/issue-353 5d25fcd
fix/issue-313 3156bc4
fix/issue-357 53d9e58
fix/issue-368 0ba727d
fix/issue-419 61648c9
fix/issue-111 b04c372
fix/issue-686 0ad90b7
fix/issue-674 793095d
fix/issue-1055 4ec96f1
fix/issue-559 dfa9525
fix/issue-558 034eed4
fix/issue-874 1720539
fix/issue-666 1e120da
fix/issue-668 fb26345
fix/issue-568 ce04173
fix/issue-311 e54f250
fix/issue-177 0cbb88a
fix/issue-151 c21377c
fix/issue-166 41fb77b
fix/issue-27 2d67622
fix/issue-323 69288ae
fix/issue-59 36463e8
fix/issue-43 35e4069
fix/issue-158 6f76f8a
fix/issue-104 bb00491
fix/issue-55 971ef1b
fix/issue-361 89ac220
fix/issue-120 b458719
fix/issue-36 42bac6e
fix/issue-525 57c97dd
fix/issue-712 e5bcda0
fix/issue-72 6cc63ed

fix/issue-72 and fix/issue-312 conflict on one line in internal/api/handler_dashboard.go (both change the calculateCommitmentMetrics call). The tree used #72's side with #312's validated provider variable; whichever PR merges second takes that same resolution.

Linux, Go 1.26.9, AWS_CONFIG_FILE=/dev/null AWS_SHARED_CREDENTIALS_FILE=/dev/null AWS_EC2_METADATA_DISABLED=true; Docker Postgres via testcontainers:

  • Passed at 936996ad: go build ./..., go vet ./..., go vet -tags integration ./... and the full go test -race -short -count=1 ./....
  • Static checks at 936996ad: golangci-lint v2.10.1 found 0 issues on every changed package, with and without --build-tags integration. tests/e2e is a separate module, so it was vetted under its own e2e tag. gocyclo -over 10 is silent on the changed non-test files.
  • Integration tests, every changed package, one package at a time (-tags integration -count=1, 60-minute local timeout because the VM was shared):
    • At 936996ad, these pass: internal/server, internal/credentials, internal/scheduler, internal/database/postgres/migrations, internal/purchase and internal/api.
    • internal/config passes except TestPostgresStoreDB_ResolveStaleRIExchange_ValidatesCloudCheck, which fails the same way on unmodified main (fixed by fix(config): compare resolve checked_at at the database's microsecond precision #1252).
    • These were run on the first build of this tree (26795376, same branches at earlier heads): internal/analytics, internal/auth, internal/commitmentopts, internal/database and cmd/lambda/clear-rate-limit all pass. The later fixes don't touch those packages.
  • What the combined run caught, each fixed on its own branch before this PR opened:
    • a compile break between fix/issue-104's test and fix/issue-326's httpToLambdaRequest signature change;
    • fix/issue-316 rejecting legacy-keyed plans;
    • fix/issue-120 versus main's global-filter test.
  • Secrets: git-secrets, using the repo's scripts/setup-git-secrets.sh patterns in a fresh repo with an isolated HOME, found nothing in the changed files, and a planted fake key was flagged. scripts/test-git-secrets-allowlist.sh: 137 PASS, 0 FAIL.
  • Nothing was applied to any cloud.

Out of scope / follow-ups

  • Everything marked Deferred in the table, in particular the trusted-proxy hop count (owner decision on the setting), email normalization (migration plus a decision on duplicate rows), the nil-Handler.rateLimiter test sweep, the per-request goroutine in RecordUsageAsync, and the httpclient transport fix in cloud-commitments-go.
  • internal/server/http.go is 530 lines after this change (was 537), still over the 500-line guideline.
  • httpToLambdaRequest reads only the first X-Forwarded-For header line (Header.Get). Whether to join repeated header lines depends on how each ingress appends, so I left it unchanged here and it should be handled with the hop-count work.

Could a maintainer mirror the issue's triage labels onto this PR: effort/l, triaged, priority/p3, severity/low, urgency/eventually, impact/many, type/security.

🤖 Generated with Claude Code

IAmJSD and others added 2 commits October 23, 2026 07:33
…, key rate limits on the client address

First slice of the 2026-07-28 auth review's medium/low findings:

- credentials.LoadKey memoized a failed secret fetch for the process
  lifetime via sync.Once, so one throttled fetch (or a first caller with
  a canceled context) broke credential decryption until the container
  recycled, even though ensureDB retries. Only a successful load is
  memoized now; loads are serialized so concurrent callers queue behind
  one fetch.
- DBRateLimiter.Allow returned (true, nil) with no pool, silently
  disabling the strict credential-endpoint limits. It now returns
  ErrRateLimiterUnavailable, so checkRateLimitStrict answers 503.
- The HTTP adapter keyed rate limits on net/http's "ip:port" RemoteAddr
  when no X-Forwarded-For was present, giving every connection a fresh
  bucket. The port is now dropped (also from an "ip:port" rightmost XFF
  entry), and an unparseable rightmost entry falls back to the peer.

Refs LeanerCloud#104

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The combined-tree run with fix/issue-326 (which changes httpToLambdaRequest
to take a ResponseWriter and return an error) failed to compile this test,
which called httpToLambdaRequest(req). Test requestSourceIP directly and pin
that httpToLambdaRequest derives SourceIP from it with a go/parser check, so
the test builds whichever of the two branches merges first. Removing the
wiring still fails the new check.

Refs LeanerCloud#104

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@IAmJSD IAmJSD added effort/l Weeks triaged Item has been triaged priority/p3 Polish / idea / may never ship severity/low Minor harm urgency/eventually No deadline impact/many Affects most users type/security Security finding labels Oct 23, 2026
@coderabbitai

coderabbitai Bot commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/l Weeks impact/many Affects most users priority/p3 Polish / idea / may never ship severity/low Minor harm triaged Item has been triaged type/security Security finding urgency/eventually No deadline

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant