fix(tasks): standardize tokenizer to Bio_ClinicalBERT across all task classes

[Rian354]

PR: Standardize Tokenizer to Bio_ClinicalBERT Across All Task Classes

Problem

PR #28 introduced NotesLabsMIMIC4 w/ the correct tokenizer ("emilyalsentzer/Bio_ClinicalBERT"), but the older task classes retained bert-base-uncased in their input_schema:

Class	Field	Tokenizer (before)
ClinicalNotesMIMIC4	discharge_note_times	bert-base-uncased
ClinicalNotesMIMIC4	radiology_note_times	bert-base-uncased
ClinicalNotesICDLabsMIMIC4	discharge_note_times	bert-base-uncased
ClinicalNotesICDLabsMIMIC4	radiology_note_times	bert-base-uncased
NotesLabsMIMIC4	admission_note_times	emilyalsentzer/Bio_ClinicalBERT (correct)

This is a silent correctness bug. The preprocessing schema tokenizes text using bert-base-uncased vocabulary IDs, but the runtime TextEmbeddingModel calls AutoModel.from_pretrained(processor.tokenizer_model). If tokenizer_model = "bert-base-uncased", it loads a general-domain BERT. The two tokenizers share most of BERT's 30,522-token vocab but differ for clinical and chemical terms.

This was identified as one of the three root causes of the 0.07 PR-AUC failure in PR #28. The fix was applied to NotesLabsMIMIC4 but not backported to the older classes.

Why Standardize Now

• ClinicalNotesMIMIC4 and ClinicalNotesICDLabsMIMIC4 are used in the legacy comparison baseline (--task clinical_notes_icd_labs). Running them w/ bert-base-uncased makes comparisons unfair.
• Bio_ClinicalBERT is already loaded for NotesLabsMIMIC4. Using the same model for all tasks avoids loading two BERT models simultaneously.
• One tokenizer across the codebase is easier to audit and swap.

Full-Scale Tokenizer Comparison (full MIMIC-IV, MLP + Transformer, 50 epochs, seed=42)

Model	HuggingFace ID	MLP PR-AUC	Xfmr PR-AUC	MLP ROC	Xfmr ROC	MLP VRAM	Xfmr VRAM	MLP Ep h	Xfmr Ep h
Bio_ClinicalBERT (default)	emilyalsentzer/Bio_ClinicalBERT	0.3206	0.1795	0.869	0.749	28.0 GB	14.7 GB	0.62h	0.87h
PubMedBERT	microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext	0.3224	0.1889	0.876	0.764	28.0 GB	14.7 GB	0.89h	0.83h
ClinicalBERT	medicalai/ClinicalBERT	0.3033	0.3024	0.864	0.860	15.2 GB	8.4 GB	0.42h	0.65h
Longformer-Clinical	yikuan8/Clinical-Longformer	0.2850	0.2089†	0.859	0.779	70.6 GB	70.6 GB	4.19h	4.22h

italic = still running (ep16, best ep16, still improving). † = converged (peaked ep11, stale 7 epochs).
Bio_ClinicalBERT and PubMedBERT transformers peaked at ep1 and degraded. ClinicalBERT transformer held 0.302 through ep11.

medicalai/ClinicalBERT is the most stable encoder across architectures. It is the only encoder where the transformer does not collapse (0.302 vs 0.180 for Bio_ClinicalBERT transformer). Recommendation: switch default to medicalai/ClinicalBERT.

To swap the tokenizer for an experiment, set tokenizer_model in the task's input_schema:

task = NotesLabsMIMIC4(window_hours=24)
task.input_schema["admission_note_times"][1]["tokenizer_model"] = "medicalai/ClinicalBERT"

This changes the task cache UUID, so caches for different tokenizers are isolated.

Changes

pyhealth/tasks/multimodal_mimic4.py

Replaced all 4 occurrences of "bert-base-uncased" w/ "emilyalsentzer/Bio_ClinicalBERT" in ClinicalNotesMIMIC4 and ClinicalNotesICDLabsMIMIC4 input schemas.

Cache impact: the task cache UUID for these classes changes (tokenizer_model is in input_schema which is in vars(task) via set_task). Any existing task-level caches built w/ bert-base-uncased will not be reused. The base global_event_df.parquet is unaffected.

Files Changed

File	Lines	Change
pyhealth/tasks/multimodal_mimic4.py	4 lines	bert-base-uncased -> Bio_ClinicalBERT

[will-pang]

lgtm!

[will-pang]

I resolved some merge conflicts @Rian354, can you double check and merge to main if it looks good to you?

[Rian354]

Works, shouldn't break anything