CDS Members Co-Author Paper on AI That Can Reduce False-Positive Findings in Breast Ultrasound Exams
CDS PhD student Artie (Yiqiu) Shen, alongside Assistant Professor and Emerging Scholar of Computer Engineering at NYU Abu Dhabi Farah Shamont and NYU School of Medicine student Jamie Oliver, has co-authored the “Artificial Intelligence System Reduces False-Positive Findings in the Interpretation of Breast Ultrasound Exams”. Additional co-authors include CDS PhD student Nan Wu and CDS affiliated professor Krzysztof J. Geras (who supervised the project), as well as several other NYU members.
In their work, the team presents an AI system that accomplishes radiologist-level accuracy in identifying breast cancer in ultrasound images. To develop this system the team organized the NYU Breast Ultrasound Dataset v1.0, which consists of 288,767 breast ultrasound exams ( 5,442,907 images) from 143,203 patients that were examined between 2012 and 2019 in the NYU Langone Health system. In the dataset report, the team summarizes the statistics of the dataset, image collection process, and image preprocessing procedures. Their intention is that the information captured in their report be used for the development of deep neural networks for breast cancer detection.
Data Preparation
To allow for the evaluation of and development of deep neural networks, the team split the dataset into three groups: training, validation, and test sets. First, the ultrasound exams were grouped according to their patient identifier. Subsequently, the team divided the patients into disjoint training (60% of the patients), validation (10% of the patients) and test (30% of the patients) sets at random. This was done to ensure that in the event that a patient had multiple ultrasound exams over the study period, the exams would all be included in the same dataset (i.e., training, validation, or test set).
Following data organization, the team utilized a thorough process for image collection and preprocessing, which consisted of six phases: image collection and extraction, image cropping, breast laterality extraction using optical character recognition (OCR), filtering of the overall dataset based on the inclusion and exclusion criteria, refinement of the test set, and removal of burnt-in annotations.
AI System
The AI system presented by the team was trained to perform classification and localization in a “weakly supervised manner” meaning the AI is able to explain its predictions by indicating locations of malignant lesions even though it is trained with only binary labels indicating whether cancer is present in the breast. The explainability of their system allows clinicians to develop trust and better understand its strengths and limitations. Their system provides several advances relative to previous related research. One is that the dataset used is (as far as we know) larger than any prior dataset used for this application. Secondly, the team conducted a reader study to compare its diagnostic accuracy with ten board-certified radiologists. In turn, their system achieved a higher area under the receiver operating characteristic curve (AUROC) and area under the precision-recall curve (AUPRC) than the ten radiologists on average.
Ultimately, what the team found is that with the help of their AI system, radiologists decreased their false positive rates by 37.4% and also reduced the number of requested biopsies by 27.8%, while maintaining the same level of sensitivity. To confirm its generalizability, the team evaluated their system on an external test dataset collected at an Egyptian hospital, where it achieved an AUROC of 0.911. This emphasizes the potential of AI to improve the accuracy, consistency, and efficiency of breast ultrasound diagnosis on a global scale.
Participating CDS Members
Artie (Yiqiu) Shen is a CDS PhD student. His research focuses on deep learning models for medical image analysis.
Nan Wu is a CDS PhD student whose research is centered on both application and theoretical learning problems related to breast cancer screening.
Krzysztof J. Geras is a CDS affiliated professor and assistant professor at the NYU School of Medicine. Krzysztof’s research interests surround unsupervised learning with neural networks, model compression, transfer learning and evaluation of machine learning models, and application of these techniques to medical image analysis.
To read “Artificial Intelligence System Reduces False-Positive Findings in the Interpretation of Breast Ultrasound Exams” in its entirety, please visit its Medrxiv webpage.
By Ashley C. McDonald
