Proposal: Tool-Memory-Evidence benchmark for agentic retrieval with FastEmbed + Qdrant

[MahdiNavaei]

Hi Qdrant/FastEmbed team,

I built a small independent benchmark for evaluating retrieval quality in agentic workflows using Qdrant and FastEmbed:

https://github.com/MahdiNavaei/agentic-retrieval-evals-qdrant

The motivation came from a gap I often see in classic RAG evaluation: most examples measure whether the system retrieved the right document. In agentic workflows, retrieval often has to cover a broader surface before the system can act correctly:

• the right tool
• the right memory or policy
• the right grounding evidence

This benchmark makes those three retrieval targets measurable in a small, local-first setup.

What it measures

Each task has a query and three expected targets:

• expected_tool
• expected_memory
• expected_evidence

The benchmark reports:

• Tool@1
• Tool@3
• Memory@1
• Evidence@3
• mean latency
• P95 latency
• quality/latency trade-offs versus dense retrieval

Strategies compared

The current version compares:

• dense: FastEmbed dense embeddings searched in Qdrant
• sparse: FastEmbed sparse BM25-style vectors searched in Qdrant
• dense-rerank: dense candidates reranked with a FastEmbed cross-encoder
• hybrid-rerank: dense and sparse candidates merged, deduplicated, then reranked

It also uses Qdrant local-memory mode by default, so it can run without Docker or a Qdrant server. Server mode is still supported.

Current verified results

These results are from a small synthetic dataset, so I am not presenting them as a scientific benchmark or leaderboard. The useful part is the evaluation shape and the quality/latency trade-off visibility.

Strategy	Tool@1	Tool@3	Memory@1	Evidence@3	Mean ms	P95 ms
dense	0.700	0.900	0.650	0.550	61.46	83.96
dense-rerank	0.750	0.850	0.700	0.750	1123.66	1235.73
sparse	0.550	0.800	0.600	0.750	1.52	2.05
hybrid-rerank	0.750	0.850	0.700	0.750	1099.49	1322.92

A few observations:

• Dense retrieval worked well as a semantic baseline and had the best Tool@3.
• Sparse retrieval was very fast and improved Evidence@3, but performed worse on Tool@1.
• Reranking improved top-rank quality for tool, memory, and evidence retrieval, but added significant latency.
• Evidence retrieval remained below the original demonstration target, which I treated as a useful failure-mode signal rather than tuning the labels to improve the score.

Question

Would something like this be useful as a FastEmbed/Qdrant docs example, runnable example, or experimental benchmark?

If yes, where would you prefer a minimal upstream version to live?

Possible options:

docs/examples/tool_memory_evidence_benchmark.md
docs/examples/tool_memory_evidence_benchmark.py

or:

experiments/agentic_retrieval_benchmark/

I would be happy to trim the scope based on maintainer preference.