[venkat1701]

Why

At scale 4 with 64 MiB caches, HyperGraphDB enumerated a hyperedge's members about 8 times faster than HStore (0.13x), and two-hop neighbourhoods about 5 times faster (0.19x). At scale 1, HStore led on both.

Profiling the scale 4 case showed where the time went:

• A member lookup missed the node cache on its catalog leaf about half the time.
• On a miss, NodeCodec.decode decoded all of the leaf's ~83 records, each with its inline member list, to return one record. That decode accounted for 70% of the time spent reading members.
• The decoded leaf took about 17 KiB of heap against a page image of about 1.5 KiB, so only about half of the 7,253 catalog leaves fit in 64 MiB.
• With a 256 MiB cache there were no misses, and member reads ran at about 80% of HyperGraphDB's rate.

Smaller pages don't fix it. With 4 or 8 KiB pages, a lookup visits about 10 pages instead of 3.

What changes

• Records framed by length. Codecs whose values are written one after another now say so with ValueCodec.independentValues(): the catalog (AtomCodec), posting directories (PostingsCodec) and ValueCodec.rows. In their leaves, NodeCodec writes each value's length in front of it. This costs about one byte per record, and the leaf budget counts it.
• Packed leaves. A framed leaf read from a page keeps the page image, its keys and the offset of every record. It decodes a record the first time it is read.
  • The decoded record is kept in the leaf and published with a compare-and-set.
  • The record's heap estimate is charged to the leaf's node cache slot, so the cache still evicts down to its budget.
  • Charges for a slot that was evicted, replaced or cleared are dropped.
• Sizes measured on demand. A read-only leaf measures its encoded size only when a write needs it. Before, every decoded inline member list was measured.
• Verification. Loading a packed leaf can't compare the header's refers-to-pages flag with its records without decoding them all. TreeVerifier (hstore check) now does that comparison instead.
• Storage format 8. Format 7 directories are refused with the usual message.

Numbers

Probe runs at scale 4, alternating main and this branch, two runs each. The miss counts are exact; throughput moved by up to 3x between runs on this machine.

Scale 4	main	this branch
member lookup, 64 MiB cache	164k/s, 0.53 leaf misses per lookup	815k/s, 0 misses
two-hop, 64 MiB cache	9.4k/s, 5.1 misses per query	102k/s, 0.2 misses
member lookup, 128 MiB cache	478k/s	1.17M/s
two-hop, 128 MiB cache	19k/s	97k/s

Scale 1 is unchanged: 2.3M against 2.4M member lookups per second, and 230k against 224k two-hop queries per second.

The comparison harness at scale 4 (async, 64 MiB caches), with 3 HStore runs on this branch, set against the main and HyperGraphDB runs from the benchmark report (3 runs each, earlier the same day):

Scale 4	HStore main	HStore this branch	HyperGraphDB	branch vs HyperGraphDB
read.members	218k/s	1.72M/s	1.72M/s	1.00x
read.twohop	15.0k/s	175k/s	78.1k/s	2.2x
read.incidence.cold	346k/s	1.32M/s	11.8k/s	112x
read.incidence	2.66M/s	3.72M/s	1.01M/s	3.7x

At scale 1, main and this branch ran back to back, alternating, 3 runs each. No workload moved by more than the spread between runs, except two:

• read.incidence.parallel is 1.28x faster on this branch;
• the files are about 1% larger after loading, from the length bytes (medians of 91.3 and 92.2 MiB).

read.members came out at 0.80x by median, but this branch's runs (2.59M, 2.35M and 1.48M per second) overlap main's (3.06M, 2.31M and 2.93M). Runs on this laptop vary that much. I'd still rerun scale 1 on a quiet machine before updating the published numbers.

Testing

• Full suite: 225 tests pass, several times over the last commits.
• New PackedLeafTest:
  • a packed catalog leaf returns every record and decodes each one only once;
  • eight threads reading one leaf see the same records, and each record is charged once;
  • the cache counts records as they unpack, stays within its budget, and ignores charges for evicted or cleared leaves;
  • a wrong length frame is reported as corruption;
  • hstore check catches a header whose page flag disagrees with the records.
  • a writer copying a packed leaf neither keeps nor charges its records.
• Breaking it on purpose: each of these makes the tests fail:
  • dropping the cache charge;
  • dropping the check for evicted slots;
  • dropping the verifier's flag check;
  • letting a copy keep its records.
• Updated tests:
  • The catalog heap estimate test reads the records before checking the estimate.
  • The history compaction test sat right at the 50% live ratio, and the extra byte per record pushed its early segment to 51.8% live, so it now compacts below 75%.
• Native image:
  • The example loads in format 8, its queries return the same rows as before, and hstore check verifies the trees and reference counts.
  • After a SIGKILL partway through a 20,000-insert load, reopening replays 1,089 commits and the counts match. COMPACT and check pass afterwards.