Skip to content

fix(api): prune the stale team-index entry when the expiration sweep finds an orphan (rebase of #3567) - #3735

Open
AdaAibaby wants to merge 1 commit into
e2b-dev:mainfrom
AdaAibaby:fix/redis-team-index-orphan-prune
Open

AdaAibaby wants to merge 1 commit into
e2b-dev:mainfrom
AdaAibaby:fix/redis-team-index-orphan-prune

Conversation

@AdaAibaby

Copy link
Copy Markdown
Contributor

Rebased follow-up to #3567 (closed inactive). The leak is still present on main, and this fixes a real race the original approach missed.

Bug

When ExpiredItems finds an orphaned expiration ZSET member (sandbox key already gone, confirmed by a nil MGET) it removes the ZSET member but leaves the sandbox ID in the per-team SET index. Those entries accumulate forever: TeamItems silently skips the nil MGET, the healer only restores missing ZSET members (it never repairs the team index), and Remove only cleans the team index on the normal path. The leftover comes from paths that delete the sandbox key without going through Remove — e.g. a pause (finishTransition DELs the key but does not SREM the index).

Fix — and why a plain SREM is unsafe

Sandbox IDs are reused across executions (a resume keeps the ID, changes the execution ID). A plain SREM would race a concurrent resume that re-creates the key and re-adds the ID between the MGET and the SREM. Because the healer does not repair the team index, that wrong removal would be permanent — the sandbox would vanish from TeamItems until it is next removed.

So the prune is an atomic EXISTS-guarded SREM (pruneOrphanIndexScript):

if redis.call('EXISTS', KEYS[1]) == 0 then
    return redis.call('SREM', KEYS[2], ARGV[1])
end
return 0

It removes the ID only if the sandbox key is still absent at that instant; a resume that re-created the key makes EXISTS != 0 and the SREM is skipped. Both keys share the {teamID} hash tag, so it is Redis-Cluster-safe. The orphan handler reads teamID from the batch it already iterates (no struct change); the ZSET member removal is unchanged.

Scope

This fixes only the orphan-index leak. The original #3567 also replaced the add/remove Lua scripts with TxPipelined — an unrelated refactor of the atomicity primitives in this concurrency-critical path — which is dropped here. (Notably this fix needs a Lua script for its atomic check-then-SREM, so removing Lua elsewhere was not the right direction.)

Tests

  • orphan sandbox ID is pruned from the team index;
  • a live sandbox key is never unindexed by the sweep (EXISTS guard), even when a dead execution of the same reused ID left an expired member.

go build ./... and go vet pass; the redis tests pass against a real Redis (REDIS_TEST_DATABASE_URL).

Closes #3566. Supersedes #3567.

…finds an orphan

When ExpiredItems finds an orphaned expiration ZSET member (the sandbox key is
already gone, confirmed by a nil MGET) it removed the ZSET member but left the
sandbox ID sitting in the per-team SET index. Those entries accumulate forever:
TeamItems silently skips the nil MGET, the healer only restores missing ZSET
members (it never repairs the team index), and Remove only cleans the team
index on the normal removal path. The leftover comes from paths that delete the
sandbox key without going through Remove (e.g. a pause via finishTransition,
which DELs the key but does not SREM the index).

Prune the team-index entry in the same sweep. Because sandbox IDs are reused
across executions (a resume keeps the ID, changes the execution ID), a plain
SREM would race a concurrent resume that re-creates the key and re-adds the ID
between the MGET and the SREM, and - since the healer does not repair the team
index - that wrong removal would be permanent (the sandbox would vanish from
TeamItems until it is next removed). So the prune is an atomic EXISTS-guarded
SREM (pruneOrphanIndexScript): it removes the ID only if the sandbox key is
still absent at that instant; a resume that re-created the key makes EXISTS != 0
and the SREM is skipped. Both keys share the {teamID} hash tag, so the script is
Redis-Cluster-safe.

The orphan handler reads teamID from the batch it is already iterating, so no
struct change is needed. The ZSET member removal is unchanged.

Rebased onto current main. Scoped to this one fix - the original e2b-dev#3567 also
replaced the add/remove Lua scripts with TxPipelined, an unrelated refactor of
the atomicity primitives in this concurrency-critical path; that is dropped here
(and, notably, this fix needs a Lua script for its atomic check-then-SREM, so
removing Lua elsewhere was not the right direction). Adds a test that the orphan
index entry is pruned, and a test that a live sandbox key is never unindexed by
the sweep.

Closes e2b-dev#3566. Supersedes e2b-dev#3567.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sandbox/storage: stale team-index SET entries accumulate when sandbox keys are lost without going through Remove()

2 participants