Repository navigation
feat(query): match roles on reverse-index postings - #108
Merged
Merged
Conversation
Every CONTAINS and HAS conjunct on the match variable now becomes one incident path with a leg per atom. Each reverse-index posting already carries the role set of its incidence, so HAS ... AS role is checked on the posting instead of re-reading the edge, and the path is exact. - PostingCursor reads inline and promoted posting lists in place and intersects them with a leapfrog. Seeks move on keys only; role sets are checked only on keys where every leg agrees, so a hub with a rare role never costs more than the old scan-and-verify plan. - Dictionary keeps a role to role-set index and reads only symbols added since the last call, instead of scanning the symbols slot on every role lookup.
This was referenced Oct 25, 2026
venkat1701
changed the base branch from
main
to
fix/serializable-predicate-reads
October 25, 2026 16:24
The first role query after every commit took a lock and rescanned new symbols. Role-set symbols are only created through Dictionary.intern, so the map is loaded once and updated when a role set is interned; queries read it without locking.
A thread could intern a role set another thread was still committing, before the map existed; the first load read the published generation and missed it. Role sets remembered before the load are now kept and added to it.
venkat1701
changed the base branch from
fix/serializable-predicate-reads
to
main
October 27, 2026 12:26
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First step of the retrieval work (role-constrained retrieval). No storage format change.
What changes
MATCH EDGE e WHERE e HAS @a AS roleused to read candidates from@a's reverse-index postings, then re-read the hyperedge for every candidate to check the role, and severalHASconjuncts could not be intersected because the role check made them inexact. But every reverse-index posting already stores the role set of its incidence (IndexValues.Incident(roleSet, locator)), and the derivation that maintains the index keeps it equal to the incidence's role set. So:CONTAINSandHASconjunct on the match variable into oneIncidentpath with one leg per atom. A leg requires every role any of its conjuncts names. The path is exact, so noVerifystep remains for these conjuncts.engine/.../index/PostingCursor: reads inline and promoted posting lists in place (no temporary tree for inline lists) and intersects them with a leapfrog.seekmoves on keys only. Role sets are checked only on keys where every leg agrees, so a hub with a million postings and a rare role, intersected with a small leg, costs a role check per shared key, never a scan of the hub.Dictionary.roleSetsWithAllkeeps a role to role-set index and reads only symbols added since the last call (and nothing at all if no commit happened), instead of scanning the symbols slot on every role lookup. Role sets are interned on main through system transactions before any posting uses them, so the index is complete for older generations, branches and open transactions.Correctness and tenant isolation
RoleRetrievalTest.roleFilteredMatchesAgreeWithMembers: 600 random edges with random role sets (three hub atoms with promoted postings), 250 randomWHEREclauses mixingCONTAINS,HAS,HAS ... AS,ORandNOT, results andcount(*)compared with a brute-force check overMEMBERS. Disabling the role filter makes it fail.AT GENERATIONwith a role set interned later, and a restart are covered.SecurityTest.roleFilteredMatchesStayInsideTheTenant: a posting that names another tenant's hyperedge never reaches the result; anchors from another tenant give no rows.PostingCursorTest: random multi-leg intersections against a naive one, forward-only seeks, open transactions, the incremental role-set index, and a test that a rare role on a 3000-posting hub costs at most two role checks.Performance (median ms, 50k
Reviewedges, same harness onmainand this branch, three interleaved runs each)HAS hub AS actorHAS hub AS actor AND HAS mid AS reviewerHAS hub AS actor OR HAS hub2 AS actorHAS mid AS target LIMIT 10CONTAINS (hub, mid)CONTAINS (leaf)CONTAINS (hub, hub2)HAS hubThe last two rows are within run-to-run spread on a busy machine (branch runs 0.94 to 1.27 and 3.78 to 4.39). Timing only the intersection in one JVM, the new cursor is faster in both cases (one leg 0.15 vs 0.28 ms, two legs 0.28 vs 0.31 ms).
Full build: engine 257, database 96, server 28, all passing.