Generalist AI Models Ace Medical Exams but Fail at Running Hospitals. A New Specialized Model Changes That.
Large language models might pass medical licensing exams with flying colors, but they often struggle to answer the mundane yet critical questions that keep a hospital running: Will this patient be readmitted in thirty days? Will insurance deny this claim?
CDS PhD student Lavender Jiang and her colleagues address this gap in their new paper, “Generalist Foundation Models Are Not Clinical Enough for Hospital Operations.” Working with CDS PhD alum Angelica Chen, CDS Professor of Computer Science and Data Science Kyunghyun Cho, and CDS-affiliated Associate Professor of Neurosurgery Eric Oermann, Jiang demonstrated that smaller, specialized models trained on internal hospital data significantly outperform massive generalist models on operational tasks.
For years, the excitement in medical AI focused on diagnostic reasoning. However, physicians spend a significant portion of their time on documentation and logistics rather than direct patient care. Operational decisions — such as predicting length of stay or mortality risk — directly impact resource allocation and patient outcomes. Generalist models trained on the open internet simply lack the specific context required for these predictions.
“The internet probably does not contain sufficient knowledge for the generalist foundation model to be good at this kind of task,” Jiang said.
To solve this, the researchers developed LANG1, a family of models ranging from 100 million to 7 billion parameters. They pretrained these models on a specialized corpus blending 80 billion clinical tokens from NYU Langone Health’s electronic health records (EHRs) with general web text.
The team tested LANG1 against “frontier” generalist models, including GPT-4 and Llama-3–70B, using a new benchmark they created called REMEDE. The results were stark. In zero-shot settings, general-purpose models failed to provide reliable predictions for tasks like insurance denial or comorbidity coding. However, after fine-tuning, the 1 billion parameter version of LANG1 outperformed generalist models that were up to 70 times larger.
“This kind of clinical ability […] does not emerge from pretraining alone,” Jiang said. “The mapping from the clinical notes to the clinical outcome is very complex.”
One of the most significant barriers to deploying hospital-specific models is the cost and scarcity of labeled data. A key finding of the paper was that LANG1 could transfer knowledge between different tasks. For example, training the model on readmission prediction improved its ability to predict mortality, even without specific labels for the latter.
Furthermore, the team addressed the challenge of portability. Hospitals often have different demographics and documentation styles, making it difficult for a model trained in New York to work elsewhere. One obvious solution would be to pool data from hospitals nationwide to create a single, universal model. However, in a separate paper titled “Paradox of De-identification,” Jiang and her colleagues demonstrated that large language models can often reverse-engineer patient identities from “anonymized” records, rendering widespread data sharing risky.
This privacy deadlock makes LANG1’s ability to “transfer” knowledge — learning from one hospital and applying it to another without sharing raw patient data — a critical breakthrough. To test this, the researchers evaluated LANG1 on the MIMIC-III dataset from Beth Israel Hospital in Boston. The model successfully generalized to the new environment, suggesting that institutions could potentially share or license base models without compromising patient privacy and without needing to train from scratch.
This research builds on Jiang’s previous work with NYUTron, a large language model deployed at NYU Langone Health. While NYUTron showed clinical utility, LANG1 takes the next step by proving that instruction fine-tuning can overcome data limitations and that smaller, domain-specific models offer a more efficient path for healthcare systems than relying on massive commercial AIs.
This work has also been written about in NYU Langone Health’s NewsHub.
By Stephen Thomas
