AI Diagnostic Tools Improve When They Consider Patient History, Not Just the Latest Scan
Medical diagnosis often hinges on the subtle evolution of symptoms over time, yet most artificial intelligence tools used in healthcare ignore historical patient data. A recent study led by CDS PhD student Haoxu Huang addressed this oversight, demonstrating that diagnostic models significantly improved when they integrated a patient’s previous medical reports alongside current imaging scans in an end-to-end multi-modal framework.
The paper, titled “HIST-AID: Leveraging Historical Patient Reports for Enhanced Multi-Modal Automatic Diagnosis,” was presented at the Machine Learning for Health (ML4H) conference in December 2024, an event held alongside the annual NeurIPS gathering. Courant PhD Divyam Madaan was the senior author, and CDS Professor Kyunghyun Cho also contributed, as did NYU Grossman School of Medicine Associate Professor Cem M. Deniz and CDS-affiliated Associate Professor Sumit Chopra.
Typically, AI models for medical diagnosis focus solely on current chest X-rays, missing a wealth of context that historical data provides. Huang and his colleagues curated a dataset called Temporal MIMIC by combining radiographic images and clinical reports spanning five years from over 12,000 patients. With an end-to-end specifically designed architecture built upon a transformer to extract data features across time, the researchers captured the changing conditions of each patient across time, effectively mimicking the detailed process a radiologist would use.
“Radiologists don’t make judgments based only on what they see today; they always look back at earlier scans to track how a condition has changed,” Huang said. “It would be interesting to see how much an AI model can benefit from it by doing the same.”
Indeed, the researchers observed notable improvements when historical patient data was included. The accuracy of diagnoses, measured by standard benchmarks such as AUROC (Area Under the Receiver Operating Characteristic Curve), rose by approximately 6.6 percent. Precision scores, another critical measure, improved by around 9.5 percent compared to baseline models relying solely on current scans.
These benefits weren’t restricted to overall accuracy. Importantly, the model’s improved performance held consistently across different demographic groups — including varying gender, age, and race — highlighting its potential for fairer and more equitable healthcare outcomes.
Not every piece of past data contributed equally, however. Reports dating back more than a month tended to reduce diagnostic accuracy, likely reflecting significant changes in patient conditions over longer periods. Additionally, historical scans show marginal attribution to model performance when combined with reports. This highlights the importance of carefully disentangling information of different data sources in the future research rather than indiscriminately combining everything together.
“This finding was crucial,” said Madaan. “It showed us that while historical information is important, there’s a limit. Very old data could sometimes confuse the model rather than help.”
The study focused specifically on chest X-ray abnormalities because of dataset availability, but the authors emphasized broader applicability. Madaan noted that this approach could extend naturally into other healthcare domains — like predicting neurological conditions such as Alzheimer’s, which similarly rely on tracking progressive changes.
For Huang, the research represented one of his first significant explorations into what sets healthcare data apart from other types of machine learning applications. “Healthcare data is special because it traces a patient’s journey over time,” he said. “Recognizing and using that fact makes a real difference.”
The full dataset and code for HIST-AID are publicly available on GitHub, making it accessible for further research and clinical use.
By Stephen Thomas
