Handly evaluation of the matchverifier#24
Merged
Conversation
Adds a cadrage document (project reminder, stakes, positioning, current state, weekly roadmap through end of July) and a first pass at detailed framework documentation, per this week's objective of laying the groundwork for docs that can later be shared as a DMCSI working paper.
…rift and LoRA leads Folds in additional workstreams flagged by the team: repo cleanup, embedding-space diagnostics, Wasserstein drift detection for monitoring, an Argo Workflows evaluation pipeline prototype, and LoRA/QLoRA as a longer-horizon lead for lighter-weight agentic inference.
…rogress prompt_builder.py's nested-quote f-strings are valid PEP 701 syntax (Python >=3.12); the earlier "broken" claim came from testing against Python 3.11 instead of the project's pinned 3.13. Only agentic_rag.py is actually broken. Also records the cleanup already carried out this week (chore/clean-main branch, stale branches flagged for deletion).
- Fix the syntax error in AgenticRAGClassifier (misplaced closing paren in the MatchVerificationInput construction). - Wire AgenticRAGClassifier into the CLI: --agentic-rag now runs the real top-k retrieval + CodeChooser pipeline instead of a stub; candidate count configurable via AGENTIC_RAG_TOP_K. - Add --verify: chain any classifier prediction into the MatchVerifier for an independent verdict, in single and batch modes (groundwork for the production-monitoring use case). - Add src/evaluation/: dependency-free hierarchical metrics (leaf and per-level accuracy, failure rate), reproducible stratified eval-set builder, and an evaluation campaign harness. - Add a minimal CI workflow (ruff, compileall, pytest) plus unit tests for the metrics; drop unused imports flagged by ruff in navigator.py. Integration against Neo4j/LLM still needs to be validated on Datalab; docs updated accordingly.
Updated project description and objectives in cadrage document. Corrected broken module references and initiated documentation for the framework.
…ction axes in the roadmap Expand §3.3-B with statistical rigor (confidence intervals, significance testing, per-segment error analysis) and a Navigator/Agentic-RAG/supervised design-tradeoff writeup; expand §3.3-C's drift detection into a concrete plan comparing Wasserstein distance, PSI and KS-test with threshold calibration and a simulated-drift validation step. Reflected in the S3/S4 weekly objectives.
§2.3 (Nettoyage) was removed upstream, leaving a stray §2.4; renumber to §2.3. §3.1 lost its second numbered item, so switch the remaining one from a numbered list to a flat one and adjust the intro sentence.
Add evaluation framework and fix agentic RAG classifier
- Add SupervisedClassifier: wraps the production torchTextClassifiers model (loaded via MLflow pyfunc, MLFLOW_TRACKING_URI/MLFLOW_MODEL_URI) behind the same MatchVerificationInput contract as the agentic classifiers, wired into the CLI (--supervised) and run_eval.py's METHODS, so it can serve as the reference baseline for the Navigator/Agentic-RAG/supervised comparison called for in the cadrage's design-tradeoff note (S3). Output parsing handles several plausible pyfunc output shapes but is unverified against the real model (no MLflow access in this sandbox) — flagged accordingly. - Add src/evaluation/drift.py: Wasserstein distance, PSI (continuous and categorical) and KS-test drift detection, with alert thresholds calibrated empirically from the reference sample (resampling under the no-drift null) rather than arbitrary constants. Covered by unit tests on synthetic data, including injected/rolling drift. - Add an MLflow connectivity smoke test alongside the existing ones. - Remove the duplicate eval_set parquet left over from before src/evaluation/config.py existed. Docs updated to reflect what's built vs. still needing validation against real Neo4j/LLM/MLflow (none of which is reachable here).
…_URI Fail with a clear message instead of an opaque MLflow error when MLFLOW_MODEL_URI is an http(s) link to the MLflow UI rather than a models:/ URI, and document the related tracking-server-vs-model-registry mismatch pitfall.
…ep loop A single model-driven Runner.run loop can't reliably both use tools and know when to stop: the SDK only resets tool_choice on "any tool used", with no concept of the domain stop condition (is_final=1). Testing showed every configuration of that single loop failing (0/8 on an 8-query benchmark): free tool_choice skips tools and guesses, forcing it for turn 1 only produces a wrong non-final answer, and forcing it permanently navigates correctly but can never emit the final structured answer (MaxTurnsExceeded every time). BaseClassifier now drives two Agent variants (via Agent.clone()) itself: an exploration agent (tools forced, no output_type) and a finalize agent (tool_choice="none", tools kept declared to avoid a chat-template hang), switching between them based on Navigator.is_current_final() - ground truth read from the graph, never the model's self-reported completion - checked only right after an actual move so a wrong RAG-seeded position can't be "verified" and returned unchanged. Also includes a repeat-call breaker for the occasional exact-repeat loop seen under greedy decoding, and go_to_child/ go_to_parent now return the destination's children directly (fewer turns). Verified 8/8 on both --navigator and --agentic-rag across a diverse query set, --verify chaining, and no change to CodeChooser/MatchVerifier/ SupervisedClassifier behavior.
A single Runner.run cannot both use tools reliably and know when to stop, so BaseClassifier now drives an explicit two-phase step loop (explore then finalize). When the finalize call itself errors, return a MatchVerificationInput at the last graph position reached (confidence 0.0) via _fallback_output instead of letting the exception propagate and losing the row entirely. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…eval harness Closes the statistical-rigor gap flagged in the cadrage doc (§3.3-B): the eval set's rare-code oversampling meant unweighted accuracy didn't represent real traffic, and there was no way to tell if a delta between two methods was significant. - build_eval_set.py stores ipw_weight (inverse-probability-of-selection) and eval_stratum per row. - metrics.py: accuracy_at_depth/evaluate take optional weights (emitting *_weighted keys alongside, never replacing the unweighted ones) and a new low_confidence_rate, distinguishing "agent gave up" (confidence 0 fallback) from a genuinely wrong answer. - bootstrap.py: stratum-aware cluster bootstrap for confidence intervals. - compare.py: paired bootstrap + McNemar's test to compare two methods run on the same eval set, since independent-CI-overlap is invalid for paired data. - run_eval.py wires weights/confidences into evaluate() and adds --bootstrap. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Re-enable @observe on classify_navigator so its trace has a parent span. - Attach --experiment-name to the Langfuse trace (name/tag/metadata) in all four entry points instead of only logging it. - Flag finalize-call failures in _run_navigator_loop as ERROR spans instead of silently succeeding with confidence 0.0. - Fix CodeChooser returning the raw agents-SDK RunResult instead of its final_output. - Let MatchVerifier judge raw ground-truth labels (no proposed_explanation/ confidence) and surface the official code definition in its prompt. - Add Graph.get_{code_information,children,descendants,siblings,parent} wrappers with a shared dotted/undotted code retry, used by both the Navigator tools and graph.py's own tool set.
Reconciles main's local-only "Ruff liner" commit (perf/correctness fixes made before the supervised classifier and Navigator-warm-start AgenticRAGClassifier redesign landed on the session branch) with the session branch's subsequent work: - src/main.py: kept main's "build NavigatorAgenticClassifier once outside the query loop" fix; dropped main's asyncio.gather/semaphore concurrency for classify_agentic_rag since it targeted the old Graph+top_k+CodeChooser design that the session branch's Navigator-warm-start redesign (ebb941a) superseded - the new AgenticRAGClassifier holds per-call Navigator state and isn't safely concurrent the same way. - src/neo4j_graph/graph.py: kept main's removal of the silent try/except around embedding-model init (let connection failures raise loudly) plus the session branch's check_embedding_ctx_length=False fix.
- cadrage_2026-07.md and framework.md §6.1 both still described the Langfuse tracing gaps (audit du 8/07) as pending code fixes; the fixes themselves landed today (@observe on classify_navigator, --experiment-name attached to the trace, ERROR span on finalize failure). Mark them done, keep the still-open langfuse.flush()/shutdown() item. - cadrage §2.2's Agentic RAG bullet still described the pre-8/07 design (embedding search + CodeChooser arbitration); the classifier was redesigned that day to use the closest code as a Navigator warm-start instead. Note the redesign explicitly.
…riants; polish evaluate_train_labels.py - .gitignore: evaluate_train_labels.ipynb (jupytext-rendered from the .py source) and data/eval/train_verification*/ (covers train_verification_fixed/ alongside train_verification/) are generated outputs, not source. - evaluate_train_labels.py: document the actual jupytext -> Quarto render path (Quarto can't run a .py directly without a project kernel attached), and cast summary stats to float() so the JSON-facing values aren't numpy scalars.
Eval harness completion, Langfuse tracing fixes, and doc corrections
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Langfuse tracing fixes: re-enable @observe on classify_navigator, attach --experiment-name to traces (name/tag/metadata) in all entry points, flag Navigator finalization failures as ERROR spans instead of silently succeeding.
Bug fixes: CodeChooser was returning the raw agent RunResult instead of .final_output; MatchVerifier now handles raw ground-truth labels and surfaces the official NACE code definition; Graph gains a shared dotted/undotted code-retry helper used by both its own tools and the Navigator's.
New evaluation tooling: evaluate_embeddings.py (embedding-space k-NN diagnostic — found a concerning 0% accuracy@1 with the currently configured embedding model), evaluate_train_labels.py + verify_train_labels.py + human_review_app.py (train-label quality audit with human-in-the-loop review).
Cleanup: removed a broken scratch duplicate, gitignored generated eval run artifacts so only source and small versioned eval-set samples stay tracked.