Pancreatic Cancer Is Often Diagnosed Late. The Clues to Catch It Earlier May Already Exist
When detected early, pancreatic cancer has a five-year survival rate of nearly 44%. When detected after it has spread, that rate falls to about 3%. CDS PhD student Ebrahim Rasromani is tackling the challenge of early detection by using large language models to extract important information from radiology reports, where these details are often buried in unstructured free text.
The work focuses on pancreatic cystic lesions, fluid-filled lesions in the pancreas that are often found incidentally on abdominal scans. These cysts are common, and most never cause harm. But a small subset can progress to high-grade dysplasia or pancreatic cancer, creating a difficult clinical question: should a patient be monitored with repeat imaging, or should the cyst be treated more aggressively? Answering that question requires large datasets that track how cysts evolve over time. Much of that information already exists in hospitals’ radiology archives, but it is written as prose rather than stored as structured data, making it difficult to study at scale.
In the study, “Leveraging Fine-Tuned Large Language Models for Interpretable Pancreatic Cystic Lesion Feature Extraction and Risk Categorization,” accepted in the American Journal of Roentgenology, Rasromani worked with a multidisciplinary team that included his two advisors: CDS Associate Professor of Mathematics and Data Science Carlos Fernandez-Granda, and CDS affiliated faculty and NYU Grossman Assistant Professor of Radiology Yiqiu Shen, a co-senior author, and a CDS PhD alum. Rasromani and his coauthors tested whether large language models could read radiology reports and convert key clinical details into structured data. They built a dataset of 6,469 abdominal MRI and CT reports from 5,615 patients with pancreatic cystic lesions, sampled from more than 55,000 eligible reports spanning 2005 to 2024. The models were trained to extract features used in pancreatic cyst risk assessment, including cyst presence, size, growth, location, main pancreatic duct findings, wall thickening, mural nodules, and were evaluated on a held-out set of human-annotated reports.
The study compared GPT-4o, a closed-source model by OpenAI, with two open-source models, Llama and DeepSeek. GPT-4o performed strongly with standard prompting, achieving 95% accuracy in extracting pancreatic cyst features from held-out radiology reports. The open-source models performed less accurately out of the box, with accuracies of 80% for Llama and 79% for DeepSeek. After fine-tuning with chain-of-thought supervision, however, the performance gap closed: Llama reached 97% accuracy and DeepSeek reached 98%, comparable to GPT-4o with chain-of-thought reasoning at 97%.
Chain-of-thought reasoning asks a model to show intermediate steps it uses before producing an answer. In this study, that made the models both more accurate and easier to audit. For example, performance improved most on features that required temporal reasoning, such as calculating the number of months between two scans. For Llama, accuracy on that task rose from 42% before fine-tuning to 91% after fine-tuning with chain-of-thought supervision.
“With chain-of-thought reasoning, you’re not just extracting a feature from the report,” Rasromani said. “You also get the model’s explanation for why it landed on that value, which someone can review later for auditing purposes. And we found it improves accuracy at the same time.”
The extracted features were then mapped to institutional risk categories based on guideline-derived criteria. To assess how those model-assigned categories compared with human experts, three radiologists independently assigned risk categories to 100 reports. Agreement amongst the radiologists was high, and adding a fine-tuned chain-of-thought model as a fourth reader did not reduce that agreement. By contrast, adding an open-source model that had not been fine-tuned lowered the agreement substantially. The researchers also reviewed the report text cited by the chain-of-thought models as evidence for their outputs and found no evidence of hallucination.
The practical appeal of open-source models is that they offer more control. Close-source models such as GPT-4o can perform well and benefit from continual optimization, but they also come with higher costs and limited transparency, reproducibility, and control over deployment. Open-source models can be more flexible and less expensive to run at scale, though they require greater technical effort to adapt and maintain. In this study, fine-tuning allowed open-source models to match GPT-4o’s performance on pancreatic cyst feature extraction while preserving those practical advantages.
For Rasromani, the work is also personal. His mother died of peritoneal cancer, a rare disease, and he chose cancer research as the focus of his PhD. This study provides a foundation for that broader goal. By turning years of radiology reports into structured data, the system can help researchers build large-scale registries of pancreatic cancer lesions and study how they evolve over time. Rasromani and his collaborators have since applied this approach to NYU Langone’s full archive of abdominal radiology and pathology reports, creating data that can support future models for predicting which patients are at higher risk.
The ultimate goal, Rasromani said, is to develop systems that can detect pancreatic cancer earlier by recognizing subtle imaging signs before a tumor is clearly visible. “If you can catch it early, that’s really the key,” he said. “Sometimes the tumor itself may not be visible yet, but there may be secondary signs in the imaging record, such as changes in the pancreatic duct and surrounding tissue, that can help flag patients earlier and improve their chances.”
By Stephen Thomas
