Artificial intelligence system for identification of false-negative interpretations in chest radiographs

Abstract

Objectives To investigate the efficacy of an artificial intelligence (AI) system for the identification of false negatives in chest radiographs that were interpreted as normal by radiologists. Methods We consecutively collected chest radiographs that were read as normal during 1 month (March 2020) in a single institution. A commercialized AI system was retrospectively applied to these radiographs. Radiographs with abnormal AI results were then re-interpreted by the radiologist who initially read the radiograph ("AI as the advisor" scenario). The reference standards for the true presence of relevant abnormalities in radiographs were defined by majority voting of three thoracic radiologists. The efficacy of the AI system was evaluated by detection yield (proportion of true-positive identification among the entire examination) and false-referral rate (FRR, proportion of false-positive identification among all examinations). Decision curve analyses were performed to evaluate the net benefits of applying the AI system. Results A total of 4208 radiographs from 3778 patients (M:F = 1542:2236; median age, 56 years) were included. The AI system identified initially overlooked relevant abnormalities with a detection yield and an FRR of 2.4% and 14.0%, respectively. In the "AI as the advisor" scenario, radiologists detected initially overlooked relevant abnormalities with a detection yield and FRR of 1.2% and 0.97%, respectively. In a decision curve analysis, AI as an advisor scenario exhibited a positive net benefit when the cost-to-benefit ratio was below 1:0.8. Conclusion An AI system could identify relevant abnormalities overlooked by radiologists and could enable radiologists to correct their false-negative interpretations by providing feedback to radiologists.