Variable-length grouped traversal is too expensive on a large on-disk graph and gets killed

[tae898]

Ladybug version

v0.16.1

What operating system are you using?

Ubuntu 22.04

What happened?

Summary

This query is too expensive to execute reliably on a large on-disk Ladybug database, and the process gets killed during execution:

MATCH (a:NodeType01)-[:EdgeType01*0..3]->(b:NodeType01)
RETURN b.active AS active, count(b) AS total, avg(b.score) AS avg_score
ORDER BY total DESC, active

Environment

• LadybugDB 0.16.1
• Python 3.12.13
• Linux x86_64
• on-disk database

Dataset size

• 10,000,000 nodes
• 77,790,000 edges
• 10 node labels
• 10 relationship types

Observed behavior

On this dataset size, running the query causes the process to be terminated with exit code 137.

Expected behavior

The query should either complete successfully or fail gracefully with a normal handled error.

Actual behavior

The whole process is killed while executing this query on the large dataset.

Are there known steps to reproduce?

No response

[tae898]

I killed the other experiments and only ran the ladybug db benchmark to have more memory, and I managed to run this query! :)

I didn't measure the actual memory consumption of this, but only just saw the ballpark numbers on htop, and it seems that this query consumes like 50GB of memory, maybe even more, which is far higher than the other graph dbs where I run the same query.

[tae898]

So currently I'm benchmarking several graph dbs. The numbers are in https://github.com/humemai/cypherglot/blob/main/scripts/benchmarks/results/runtime-results-all-Ryzen-9-7950X.md. They are being updated, as I speak. Once I'm done with it, I'll report more concise version to you. I wanna run the ladybug benchmark two more times to report the mean and std. As you know, I also posted the high ingestion time issue #474, and because of this the ladybug runs are a bit slow at this moment, compared to the other dbs.

[adsharma]

The large RSS while running benchmarks could be related to memory management issues in python bindings. Some of them of resolved in (unreleased) main branch. One quick experiment to try is to close the connection after some number of iterations to release the memory. and open a fresh one.

The correct fix of course is to build memory management around the QueryResult abstraction. When you close the QueryResult memory gets released even if Connection and Database are still open.