John T. Murchison, Gillian Ritchie, David Senyszak, Jeroen H. Nijwening, Gerben van Veenendaal, Joris Wakkie, Edwin J. R. van Beek
Abstract
In this study, we evaluated a commercially available computer assisted diagnosis system (CAD). The deep learning algorithm of the CAD was trained with a lung cancer screening cohort and developed for detection, classification, quantification, and growth of actionable pulmonary nodules on chest CT scans. Here, we evaluated the CAD in a retrospective cohort of a routine clinical population.
Introduction
Lung nodule detection and management is one of the most frequent challenges in chest computed tomography (CT), not just in the context of lung cancer screening, but also in the staging of other malignancies in routine clinical practice. Lung cancer remains the third most prevalent cancer worldwide, is both rising in incidence [1], and maintains high mortality rates with around 1.8 million global deaths annually. Several recent studies demonstrated the benefits of lung cancer screening on early detection and improved outcomes [2–4]. The advent of lung cancer screening results in the need to detect smaller nodules, and therefore, the importance of fast and accurate detection is even more pronounced [5].
Lung cancer is ideally diagnosed by histopathological confirmation. However, the diagnostic process usually begins with chest CT where pulmonary nodules are identified incidentally.
Materials & Methods
Subject selection
CT studies from a routine clinical population, in a single academic hospital, between January 2008 and December 2009 (9 years before start of this retrospective study), were searched for the following inclusion criteria: age 50–74 years, current smokers, a smoking history and/or radiological evidence of pulmonary emphysema. CT studies excluded from the analysis had slice thickness >3mm, or the presence of diffuse pulmonary disease in the radiology report, and/or the CT images, with widespread abnormalities such as interstitial lung disease.
Results
Groups 1 and 2 consisted of 273 CT scans with 269 actionable nodules see Table 1. Remarkably, nodules were identified in group 1, highlighting the importance of concurrent reading. The radiologists with CAD readings showed a sensitivity of 93.5% and average FP rate of 3.0. The sensitivity for detecting actionable nodules of radiologists without CAD on scans from groups 1 and 2 was: 71.9% (95% CI: 66.0%, 77.0%) and 80.3% (95% CI: 75.2%, 85.0%) (p < 0.01), respectively. The average FP rate of radiologists alone and radiologists with CAD readers was: 0.11 and 0.16, respectively.
Discussion
The study described here shows improved sensitivity of experienced thoracic radiologists using aided detection from 71.9% to 90.3% with a minor increase in FP rate. The maximum stand-alone CAD sensitivity was 95.9% at an average FP rate of 10.9, which would be unworkable in clinical practice. A more acceptable average FP rate would be between 1 and 2 with corresponding sensitivity range (82.3% - 89.0%), outperforming thoracic radiologists with and without using CAD. The standalone performance of the CAD, when set to the threshold of 0.1 applied in this study, correlates to an average sensitivity of 95% and an average number of 7 false positives per study based on this dataset.
Citation: Murchison JT, Ritchie G, Senyszak D, Nijwening JH, van Veenendaal G, Wakkie J, et al. (2022) Validation of a deep learning computer aided system for CT based lung nodule detection, classification, and growth rate estimation in a routine clinical population. PLoS ONE 17(5): e0266799. https://doi.org/10.1371/journal.pone.0266799
Editor: Chang Min Park, Seoul National University Hospital, Seoul National University College of Medicine, REPUBLIC OF KOREA
Received: March 8, 2021; Accepted: March 28, 2022; Published: May 5, 2022.
Copyright: © 2022 Murchison et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data cannot be shared publicly because of confidential patient information. Anonymous data is stored on a stand-alone server at the Edinburgh Imaging facility QMRI, University of Edinburgh, Edinburgh, UK. http://www.ed.ac.uk/edinburgh-imaging To access the data, please contact the Caldicott Guardian's Office: Caldicott Office NHS Lothian Waverley Gate 2-4 Waterloo Place Edinburgh EH1 3EG Phone +44-131-4655452 [emailprotected].
Funding: This study was funded by NHS England via the SBRI Phase 1 grant for “Early Detection and Diagnosis of Cancer” which was granted to Aidence (Amsterdam, the Netherlands). Aidence provided support with this grant in the form of salaries for authors [JTM, GR, EJRVB], but did not have any additional role in the study design, data collection and most of the analysis, decision to publish, or preparation of the manuscript. The specific roles of these authors are articulated in the ‘author contributions’ section. EJRvB is a member of the Medical Advisory Board of Aidence and received support in the form of salary for the work performed in this study. JTM has no affiliation with Aidence and received support in the form of salary for the work performed in this study. GR has no affiliation with Aidence and received support in the form of salary for the work performed in this study. DS has no affiliation with Aidence and received support in the form of salary for the work performed in this study. JHN is a full time, paid employee of Aidence at the time of submission of this manuscript. GvV is a full time, paid employee of Aidence at the time of submission of this manuscript.
Competing interests: JTM, GR, and DS declare no competing interests. EJRvB declares ownership of QCTIS Ltd and serves on the medical advisory boards of Aidence BV and Imbio LLC. EJRvB received a restricted research grant from Siemens Healthineers and speaker fees from AstraZeneca and Roche Diagnostics. EJRvB’s non-financial competing interest is serving as an expert witness on medicolegal advice on imaging based cases. JHN and GvV are full time, paid employees of Aidence and have aided in part of the raw data analysis (GvV) or re-submitting this manuscript to PLOS ONE (JHN). These interests do not alter our adherence to PLOS ONE policies on sharing data and materials.
Abbreviations: 3D, 3 dimensional; CAD, Computer Assisted Detection; CADe, Computer Assisted Detection Device; CADx, Computer Assisted Diagnostic Device; CE, Conformité Européenne; CI, Confidence Interval; CT, Computed Tomography; CTDlvol, Volume CT Dose Index; DICOM, Digital Imaging and Communications in Medicine; FP, False Positive; FN, False negative; FROC, Free Response Receiver Operating Characteristic; GE, General Electric; MIP, Maximum Intensity Projection; MPR, Multiplanar reconstruction; kVp, Kilovoltage peak; mAs, Milliamp seconds; mGy, Milligray; NLST, National Lung Screening Trial; TP, True positive; VDT, Volume Doubling Time.
