Fix/agentic rag navigator robustness#27
Merged
Merged
Conversation
Replaces three near-duplicate s3fs.S3FileSystem constructions (notice_manager.py, build_eval_set.py, convert_to_parquet.py), each reading a slightly different combination of AWS_* env vars, with a single src/utils/storage.get_file_system(). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Removes src/evaluation/drift.py (Wasserstein/PSI/KS drift monitoring, abandoned per docs/cadrage_2026-07.md — not just its tests) along with evaluation/evaluation.py, explorations.py and src/test.py, and the tests covering the deleted code (test_bootstrap.py, test_build_eval_set.py, test_compare.py, test_drift.py). Updates README's quickstart command and pyproject.toml's ruff exclude list to match. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- BaseClassifier: when exploration stalls on a non-final node (step budget exhausted), force descent to a real leaf via a child-only tool choice instead of letting finalize accept a category as the answer; add a deterministic (non-LLM) first-leaf walk as the last-resort fallback target, and a safety net that rejects any finalize output landing on a non-final code. Broaden the try/except around the loop to cover exploration/retry/ forced-descent as well as finalize, so every failure mode (e.g. an LLM endpoint timeout) degrades the same way. - base_agent.py: cap the OpenAI client at a 60s timeout instead of the SDK's 10 min default, so a stuck request fails fast rather than hanging. - match_verifier.py: prompt the explanation to cite a concrete matching/ conflicting element of the code's definition, not a restatement of intent. - test_connections.py: swap the MLflow smoke test to a metadata-only check (loading the registered pyfunc model raises ModuleNotFoundError outside the training repo) and test SupervisedClassifier's real HTTP path (codif-ape-API) instead of a local NaiveCode2Text LLM check. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New third classification approach alongside Navigator and Agentic RAG: the model gets the full NACE hierarchy (code + name, flattened text) upfront in its system prompt, rather than navigating node-by-node or starting from an embedding-similarity seed, and freely chooses which codes/tools to query and when to stop (single unforced Runner.run, no Python-owned step loop). - src/neo4j_graph/build_nace_summary.py — offline builder that flattens the hierarchy via the new Graph.get_summary_tree() into an indented text file (data/nace_summary.txt, gitignored as regenerable from Neo4j). - src/agents/Text2Code/classifiers/summary_classifier.py — the classifier itself, using Graph's stateless lookup tools (get_code_information, get_children) rather than the Navigator's current-position tools. - Wired into the CLI (--summary) and run_eval.py (method="summary"). - graph.py also fixes get_code_information's children query to filter out the null entry OPTIONAL MATCH produces when a node has no children. Also: verify_train_labels.py now asks SummaryAgenticClassifier for a second opinion on every train-label row MatchVerifier flags as a mismatch, computed once and stored in the output parquet (summary_code/summary_confidence/ summary_explanation) rather than recomputed on demand; human_review_app.py reads those columns directly instead of calling the classifier itself. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…enders Adds compute_hierarchical_accuracy (accuracy@1 at each NAF hierarchy depth, not just the leaf) and compute_worst_offenders (labels the k-NN retrieval gets most wrong) to evaluate_embeddings.py, with matching Plotly figures, and mirrors the new analysis into presentation/embeddings.qmd. evaluate_train_labels.py gets a trailing jupytext cell marker. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
_preview/ is Quarto's rendered output (cf. .gitignore comments), refreshed by a local `quarto render`/`quarto preview` run: updated search index, content-hashed syntax-highlighting CSS, and bundled site_libs (bootstrap, quarto-html, quarto-nav, quarto-search, mermaid). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… S3 storage - Drops the §5.1 drift-detection section (src/evaluation/drift.py removed) and the "document en cours de construction" status banner, renumbering the embedding-diagnostics section to §5.1. - Documents build_nace_summary.py (§2.3), SummaryAgenticClassifier (§3.4), and the --summary CLI flag (§4). - Adds §6.1 on the shared src/utils/storage.py S3 helper and the S3 sync of previously local-only eval/review/diagnostic data. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
SummaryAgenticClassifier has no Python-owned guard rail like BaseClassifier's, so an unhandled error (e.g. an LLM endpoint timeout) could still abort a whole evaluation campaign on one label. run_eval.py now records such failures as a failed prediction instead. Also brings framework.md/cadrage_2026-07.md up to date with the 15/07 Navigator loop hardening (forced descent, non-final safety net, client timeout) that had landed in code but not yet in the docs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
framework.md/cadrage_2026-07.md still cited an exploratory accuracy@1 = recall@5 = 0% result from before evaluate_embeddings.py was committed. The committed, reproducible diagnostic (extended today with hierarchical accuracy and full-eval-set metrics, cf. presentation/embeddings.qmd's cached results) shows accuracy@1 = 48.1%, recall@5 = 77.5% on the full 5,181-label eval set instead — the warm start is usable, just not reliable enough on its own without the Navigator's follow-up check. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…bels.py run_eval.py now times each classification call and reports total/mean duration alongside accuracy, needed to compare methods on cost as well as correctness (cadrage §3.3-B). verify_train_labels.py retries MatchVerifier/SummaryAgenticClassifier once on transient failure (LLM timeout) before giving up on a row rather than crashing the whole run, and streams each verified row to a checkpoint file as soon as it's computed so an interruption only loses unflushed work - same durability pattern as run_eval.py's own prediction checkpoint. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…te_train_labels.py evaluate_eval_set_multi_method.py runs all 4 classifiers (Navigator, Agentic-RAG, Summary, supervised) plus MatchVerifier and CodeChooser arbitration concurrently on a sample of the eval set, timing every call. Its companion multi_method_review_app.py lets a human pick which candidate code is actually correct, to score CodeChooser and each method against human judgment rather than a training label of unknown quality. Moved human_review_app.py alongside it under the new src/evaluation/apps/ package, as a single home for the review UIs, and updated references accordingly (docs/framework.md, verify_train_labels.py). evaluate_train_labels.py is removed: its label-verification step was already extracted into verify_train_labels.py, and its reclassify+arbitrate steps are superseded by evaluate_eval_set_multi_method.py's concurrent, checkpointed batch equivalent. Its pyproject.toml ruff exclude (needed for its top-level awaits) is removed along with it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ntation multi_method_eval.qmd writes up a completed run of evaluate_eval_set_multi_method.py (60 rows, seed=123, concurrency=6): timing per method, the robustness gap between Navigator/Agentic-RAG (covered by BaseClassifier's stall/non-final guard) and Summary/CodeChooser (bare Runner.run, no guard, high timeout failure rate under concurrency), accuracy per method, and how often the ground truth itself holds up under MatchVerifier + CodeChooser arbitration. index.qmd adds a landing page linking the three site pages (présentation, diagnostic embeddings, éval. multi-méthode); presentation.qmd no longer claims the index.html output slot now that index.qmd owns it. _quarto.yaml wires both new pages into the site's render list and navbar. _preview/site_libs and the new _freeze/site_libs/revealjs assets are the regenerated Quarto build output for these pages, cf. the prior "Regenerate Quarto preview site assets" commit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.