Skip to content

feat(query): match roles on reverse-index postings - #108

Merged
venkat1701 merged 8 commits into
mainfrom
feature/role-filtered-postings
Oct 27, 2026
Merged

venkat1701 merged 8 commits into
mainfrom
feature/role-filtered-postings

Conversation

@venkat1701

@venkat1701 venkat1701 commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator

First step of the retrieval work (role-constrained retrieval). No storage format change.

What changes

MATCH EDGE e WHERE e HAS @a AS role used to read candidates from @a's reverse-index postings, then re-read the hyperedge for every candidate to check the role, and several HAS conjuncts could not be intersected because the role check made them inexact. But every reverse-index posting already stores the role set of its incidence (IndexValues.Incident(roleSet, locator)), and the derivation that maintains the index keeps it equal to the incidence's role set. So:

  • The planner merges every CONTAINS and HAS conjunct on the match variable into one Incident path with one leg per atom. A leg requires every role any of its conjuncts names. The path is exact, so no Verify step remains for these conjuncts.
  • New engine/.../index/PostingCursor: reads inline and promoted posting lists in place (no temporary tree for inline lists) and intersects them with a leapfrog. seek moves on keys only. Role sets are checked only on keys where every leg agrees, so a hub with a million postings and a rare role, intersected with a small leg, costs a role check per shared key, never a scan of the hub.
  • Dictionary.roleSetsWithAll keeps a role to role-set index and reads only symbols added since the last call (and nothing at all if no commit happened), instead of scanning the symbols slot on every role lookup. Role sets are interned on main through system transactions before any posting uses them, so the index is complete for older generations, branches and open transactions.

Correctness and tenant isolation

  • RoleRetrievalTest.roleFilteredMatchesAgreeWithMembers: 600 random edges with random role sets (three hub atoms with promoted postings), 250 random WHERE clauses mixing CONTAINS, HAS, HAS ... AS, OR and NOT, results and count(*) compared with a brute-force check over MEMBERS. Disabling the role filter makes it fail.
  • Merged legs, unknown roles, open transactions, AT GENERATION with a role set interned later, and a restart are covered.
  • SecurityTest.roleFilteredMatchesStayInsideTheTenant: a posting that names another tenant's hyperedge never reaches the result; anchors from another tenant give no rows.
  • PostingCursorTest: random multi-leg intersections against a naive one, forward-only seeks, open transactions, the incremental role-set index, and a test that a rare role on a 3000-posting hub costs at most two role checks.

Performance (median ms, 50k Review edges, same harness on main and this branch, three interleaved runs each)

Query main branch
HAS hub AS actor 8.43 1.48
HAS hub AS actor AND HAS mid AS reviewer 0.83 0.12
HAS hub AS actor OR HAS hub2 AS actor 22.9 2.75
HAS mid AS target LIMIT 10 0.094 0.074
CONTAINS (hub, mid) 1.14 0.78
CONTAINS (leaf) 0.26 0.23
CONTAINS (hub, hub2) 1.02 1.14
HAS hub 4.01 4.32

The last two rows are within run-to-run spread on a busy machine (branch runs 0.94 to 1.27 and 3.78 to 4.39). Timing only the intersection in one JVM, the new cursor is faster in both cases (one leg 0.15 vs 0.28 ms, two legs 0.28 vs 0.31 ms).

Full build: engine 257, database 96, server 28, all passing.

Every CONTAINS and HAS conjunct on the match variable now becomes one
incident path with a leg per atom. Each reverse-index posting already
carries the role set of its incidence, so HAS ... AS role is checked on
the posting instead of re-reading the edge, and the path is exact.

- PostingCursor reads inline and promoted posting lists in place and
  intersects them with a leapfrog. Seeks move on keys only; role sets
  are checked only on keys where every leg agrees, so a hub with a rare
  role never costs more than the old scan-and-verify plan.
- Dictionary keeps a role to role-set index and reads only symbols
  added since the last call, instead of scanning the symbols slot on
  every role lookup.
@venkat1701
venkat1701 changed the base branch from main to fix/serializable-predicate-reads October 25, 2026 16:24
The first role query after every commit took a lock and rescanned new
symbols. Role-set symbols are only created through Dictionary.intern,
so the map is loaded once and updated when a role set is interned;
queries read it without locking.
A thread could intern a role set another thread was still committing,
before the map existed; the first load read the published generation
and missed it. Role sets remembered before the load are now kept and
added to it.
@venkat1701
venkat1701 changed the base branch from fix/serializable-predicate-reads to main October 27, 2026 12:26
@venkat1701
venkat1701 merged commit a7fa755 into main Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant