fix(build): preserve leaked biomedical category instead of discarding it#1944
Open
wonhurk wants to merge 1 commit into
Open
fix(build): preserve leaked biomedical category instead of discarding it#1944wonhurk wants to merge 1 commit into
wonhurk wants to merge 1 commit into
Conversation
Biomedical semantic subagents are instructed to record a node's entity category (drug/gene/disease/protein/technology/peptide/institution/ company) in `entry_type`. Models sometimes also write the category into `file_type`, which is not a valid file_type. The existing canonicalizer collapses any unknown file_type to `concept` via _FILE_TYPE_SYNONYMS, which silences the validator warning but silently discards the category and mislabels the node as a `concept`. Recover a leaked biomedical category into `entry_type` (without overwriting an existing one) and coerce `file_type` to `document`, which matches how these patent/paper record files are actually typed. The ambiguous `technology` token (already a code-corpus synonym -> concept) is only treated as biomedical when the node carries a biomedical signal (an existing `entry_type`), so non-biomedical corpora are unaffected. Adds tests covering the recovery, the ambiguous-technology guard, and the unknown-junk fallback to `concept`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(build): preserve leaked biomedical category instead of discarding it
Problem
Biomedical semantic subagents are instructed (in the
/graphifyextractionrules) to record a node's entity category —
drug/gene/disease/protein/technology/peptide/institution/company— in thenode's
entry_typeattribute.file_typeis a separate axis with a smallclosed set of valid values (
code,document,paper,image,rationale,concept).In practice, models frequently also write the category into
file_type,e.g. a patent-record node comes out as
file_type: "technology". That is not avalid file_type. The current canonicalizer in
build_from_jsonhandles this bycollapsing any unknown file_type to
conceptthrough_FILE_TYPE_SYNONYMS:This silences the invalid-file_type validator warning, but it has two downsides
for biomedical corpora:
technology→concept;peptide,drug,gene, etc. aren't in the synonym map at all and also fall toconcept. The information the model extracted is lost.conceptdocument rather than thedocument(patent/paper record) it actually is.Real-world example (patent records): nodes like
mitocatch,phlip,bispecific-nanobody,erythrocyte-mito-capsulecame through with the categoryduplicated into
file_type— and in the observed data these nodes alreadycarried the correct
entry_type, so the category was recoverable but wasbeing thrown away.
Fix
When a node's
file_typeis a leaked biomedical category, recover it intoentry_type(without overwriting an existing one) and coercefile_typetodocument:Ambiguity handling for
technologytechnologyis intentionally also a code-corpus synonym (→concept), so itis ambiguous. To avoid changing behavior for non-biomedical corpora, a bare
technologyfile_type with no biomedical signal keeps the existingconceptmapping; it is only recovered as biomedical when the node alreadyshows a biomedical signal (an existing
entry_type). The conditionft not in _FILE_TYPE_SYNONYMSmeans the biomedical-exclusive categories(
peptide,drug,gene,disease,protein,institution,company) arealways recovered, since none of them appear in the synonym map.
Behavior matrix
file_typeentry_typefile_typeentry_typepeptidedocumentpeptidetechnologytechnologydocumenttechnologytechnologyconcept(unchanged)gizmo(junk)concept(unchanged)paper(valid)paper(unchanged)No invalid-file_type validator warning is emitted in any case.
Tests
Adds three tests to
tests/test_build.py:test_leaked_biomedical_category_recovered_into_entry_typetest_ambiguous_technology_without_signal_stays_code_synonymtest_unknown_invalid_file_type_still_falls_back_to_conceptAll existing
test_build.pytests continue to pass.Scope
Both
build_from_jsonand the multi-extractionbuild()(which delegates tobuild_from_json) are covered by the single change. No public API changes.