Clinical Triage Assistant at Centre Hospitalier Saint-Aurélien

Context. Centre Hospitalier Saint-Aurélien is a hospital whose emergency department faced understaffed triage at peak hours. I worked as sole engineer on a proof of concept for a triage assistant, from corpus construction through to a running inference service, in July 2026. Five deliverables were requested and all five were completed: a bilingual dataset, a specialized model, a demonstration endpoint, a CI pipeline, and a technical report.
Problem. The risk specific to the triage step is a critical case not identified early enough. A language model can read a patient complaint and propose an urgency level and a department, but the useful question is not whether it scores well in isolation. The client supplied a randomized study of 50 physicians in which the model used alone scored 15 points above practitioners, while the same model placed in physicians' hands produced a gain of 1.6 points, not statistically significant. Raw performance does not convert mechanically into clinical value: integration into real work decides that.
Constraints. The system had to be assistance subject to clinician validation, never autonomous decision, which follows from the study above rather than from caution. A medical corpus made GDPR anonymization harder than usual, since clinical vocabulary defeats off-the-shelf detection settings. Access had to be confined with no exposed ports, every interaction auditable, every data transformation traceable. Training ran on a single L4 GPU, whose memory dictated two settings rather than the other way around.
Approach. Six stages: corpus aggregation, anonymization, supervised fine-tuning, preference alignment, adapter merge, deployment. I selected Qwen3-1.7B-Base rather than the instruct variant, so that my fine-tuning would install the triage behaviour directly instead of having to correct a pre-existing generic alignment. Four sources across two languages, each with licence and access date recorded: three feed the supervised corpus, and UltraMedical-Preference is reserved for alignment and never enters supervised training. A clinical evaluation set is held outside the train, validation and test split. Because the three supervised sources are heterogeneous and none is natively structured for triage, I delegated normalization to a lightweight model, Mistral Small, rewriting every answer into a uniform shape of urgency level, department and explanation. Anonymization rested on four documented trade-offs against Presidio defaults: organization entities excluded, since clinical vocabulary triggered detections in bulk without identifying anyone; location retained at country granularity, which carries clinical meaning in travel medicine without identifying an individual; a confidence threshold of 0.5; and medical eponyms handled by exception list and context rules. The measured residual identifier rate is 0.9%. Supervised fine-tuning used LoRA at rank 16, alpha 32, with gradient checkpointing and a 512-token cap, running 26 minutes on an L4. DPO alignment was applied to the SFT model rather than the base model, to stay inside the distribution of the tuned model, and chosen over RLHF because it needs no intermediate reward model and one training run instead of two: with LoRA, disabling the adapter yields the reference model with no copy in memory. The service runs as a vLLM server behind a containerized API, with GitHub Actions running the API tests and building the image when they pass.
Result. The training indicators were good: the model ranked preference pairs correctly in three cases out of four with a mean margin of 0.84, no divergence, no loss collapse. The legitimate conclusion is that alignment converged on the expected signal. It says nothing about clinical correctness, and five clinical vignettes showed why. Alignment improved the form of the answers, their structure and their placement on the urgency scale, but its effect on diagnostic accuracy was unstable: three degradations, one neutral case, one improvement. The two worst involve conditions whose prognosis depends directly on time to treatment. On chest pain, the supervised model identified an infarction and assigned maximum urgency while the aligned model substituted a panic attack; on sudden weakness with speech disturbance, the aligned model drifted to deep vein thrombosis and masked the stroke. Four distinct failure modes emerged, and the most severe is fabrication: the model invents a clinical sign or a test that was never performed in order to support its conclusion, and a fabricated history stated as fact carries no marker distinguishing it from data actually supplied. My metrics could not see any of this, because preference accuracy and margin measure a relative ranking between two candidate answers and never tie either one back to the content of the prompt. Fabrication is therefore unpenalized and plausibly rewarded, since an answer that supplies absent elements reads as more complete than one that stays within available data. Averaging compounds it: these failures are rare and severe, and a mean does not surface that kind of distribution tail. Latency averaged 1.57 seconds, between 1.17 and 2.28, compatible with interactive use. The prototype establishes the technical feasibility of the chain. It does not establish clinical reliability, and the fact that the training indicators reported nothing is itself the result: validation of a system of this kind cannot rest on the metrics of its own training.
Limits. My evaluation covers 7 cases scored by a single non-clinician evaluator, with no blind scoring and no inter-rater agreement, which establishes the existence of the failure modes but not their frequency. There is no load testing, and robustness holds in the sense of availability rather than relevance: an empty input still returns a response formatted as a valid one. The pipeline performs continuous integration but not continuous deployment, so putting the service on the machine remains manual; automating it would require an image registry and secrets management for the target environment. MedQuAD declares no licence on the repository used, a low exposure on generic public question-answer material but one to clear before any production use. Finally, scaling to a larger model would not remove the fabrication mode: a larger model phrases invented content more confidently, which displaces the risk rather than treating it.
Repository. github.com/Jojo4911/medical-triage-llm