Head CT Foundation Model Outperforms Commercial Alternatives, Detects Diseases Beyond Hemorrhage
The cheap, fast scans that emergency rooms use to check for strokes and brain bleeds may also detect Alzheimer’s disease and related dementias — conditions traditionally diagnosed through expensive MRIs that many elderly patients never receive. A new foundation model trained on 361,663 head CT scans from NYU Langone shows that these widely available scans can identify cognitive impairment with strong accuracy, opening a path to earlier dementia detection in emergency rooms and underserved communities where MRI access remains limited.
The model, called FM-HCT and described in the paper “3D Foundation Model for Generalizable Disease Detection in Head Computed Tomography,” published in Nature Biomedical Engineering, was developed by a team including CDS PhD student Haoxu (Howard) Huang and CDS PhD alum Weicheng (Jack) Zhu (who share first authorship), CDS PhD student Long Chen, CDS PhD alum Boyang Yu, CDS MS alum Huanze Tang, and CDS-affiliated Associate Professor Narges Razavian.
FM-HCT outperformed several existing CT foundation models, including Google’s CT Foundation model, Merlin, and CT-FM, across ten different disease detection tasks. When fine-tuned on labeled data, the model substantially improved on models trained from scratch using internal NYU Langone data, with similar gains on external datasets from NYU Long Island and the Radiological Society of North America.
The decision to build the model came from a clinical collaboration. Jennifer Frontera, a neurologist, and Kara Melmed, a neurointensivist, both at NYU Langone and co-authors on the paper, approached Razavian with an important stroke outcome prediction task they could not solve with conventional methods because for that task there were relatively few labeled samples available.
“I couldn’t build powerful models with those labeled samples,” Razavian said. “I couldn’t answer their question unless I built a whole foundation model and fine-tuned it for this specific task.”
Huang, who joined the project alongside Zhu, wanted to test whether a simple architecture could scale on three-dimensional medical scans the way large language models scale on text. The team used a plain Vision Transformer with two standard self-supervised learning methods — DINO and Masked Autoencoder — rather than the custom architectures favored by other groups.
“We asked a simple question: if we paired the most straightforward architecture and pretraining objective with scale alone, can we unlock strong performance in 3D imaging? The idea follows a broader lesson from the history of deep learning: sufficiently scaled, simple systems can outperform more carefully engineered approaches that were previously favored by the field,” Huang said.
The architectural choice and scaled-up pretraining paid off. With the foundation model’s learned representations, the team could fine-tune on hundreds of samples and achieve results that would have been impossible training from scratch. They also found that the transformer design was more generalizable, and easier to adopt by the broader research community, which they expected to facilitate further development. The vanilla architecture also made the model easier to extend with newer techniques.
Razavian emphasized that the work demonstrates the continued relevance of academic medical centers in an era dominated by industry labs. The scaling analysis showed that performance had not plateaued at 361,663 scans — the entirety of CT scans available at a single large clinical institution.
“Multi-institute collaboration has so much more potential to bring this even further,” Razavian said. “There’s so much more potential for national initiatives to really think about the medical side of the infrastructure, the data, the training.”
The clinical applications are already moving forward. The team is working with Frontera on deploying the model to predict functional outcomes after stroke. For dementia detection in particular, elderly patients who fall and end up in emergency rooms typically receive head CTs rather than MRIs, regardless of whether providers suspect cognitive impairment — making CT-based detection a practical path to catching cases that would otherwise go undiagnosed.
By Stephen Thomas
