Abstract
Background: Artificial Intelligence (AI)-based automatic lung nodule detection system improves the detection rate of nodules. It is important to evaluate the clinical value of the AI system by comparing AI-assisted nodule detection with actual radiology reports.
Objective: To compare the detection rate of lung nodules between the actual radiology reports and AI-assisted reading in lung cancer CT screening.
Methods: Participants in chest CT screening from November to December 2019 were retrospectively included. In the real-world radiologist observation, 14 residents and 15 radiologists participated in finalizing radiology reports. In AI-assisted reading, one resident and one radiologist reevaluated all subjects with the assistance of an AI system to locate and measure the detected lung nodules. A reading panel determined the type and number of detected lung nodules between these two methods.
Results: In 860 participants (57±7 years), the reading panel confirmed 250 patients with >1 solid nodule, while radiologists observed 131, lower than 247 by AI-assisted reading (p<0.001). The panel confirmed 111 patients with >1 non-solid nodule, whereas radiologist observation identified 28, lower than 110 by AI-assisted reading (p<0.001). The accuracy and sensitivity of radiologist observation for solid nodules were 86.2% and 52.4%, lower than 99.1% and 98.8% by AI-assisted reading, respectively. These metrics were 90.4% and 25.2% for non-solid nodules, lower than 98.8% and 99.1% by AI-assisted reading, respectively.
Conclusion: Comparing with the actual radiology reports, AI-assisted reading greatly improves the accuracy and sensitivity of nodule detection in chest CT, which benefits lung nodule detection, especially for non-solid nodules.
Keywords: Artificial intelligence, lung nodule, detectability, real-world study, radiologist observation, computed tomography.
Graphical Abstract
[1]
Torre LA, Siegel RL, Jemal A. Lung cancer statistics. Adv Exp Med Biol 2016; 893: 1-19.
[http://dx.doi.org/10.1007/978-3-319-24223-1_1] [PMID: 26667336]
[http://dx.doi.org/10.1007/978-3-319-24223-1_1] [PMID: 26667336]
[2]
van Klaveren RJ, Oudkerk M, Prokop M, et al. Management of lung nodules detected by volume CT scanning. N Engl J Med 2009; 361(23): 2221-9.
[http://dx.doi.org/10.1056/NEJMoa0906085] [PMID: 19955524]
[http://dx.doi.org/10.1056/NEJMoa0906085] [PMID: 19955524]
[3]
Church TR, Black WC, Aberle DR, et al. Results of initial low- dose computed tomographic screening for lung cancer. N Engl J Med 2013; 368(21): 1980-91.
[http://dx.doi.org/10.1056/NEJMoa1209120] [PMID: 23697514]
[http://dx.doi.org/10.1056/NEJMoa1209120] [PMID: 23697514]
[4]
Aberle DR, Adams AM, Berg CD, et al. Reduced lung-cancer mortality with low-dose computed tomographic screening. N Engl J Med 2011; 365(5): 395-409.
[http://dx.doi.org/10.1056/NEJMoa1102873] [PMID: 21714641]
[http://dx.doi.org/10.1056/NEJMoa1102873] [PMID: 21714641]
[5]
de Koning HJ, van der Aalst CM, de Jong PA, et al. Reduced lung-cancer mortality with volume CT screening in a randomized trial. N Engl J Med 2020; 382(6): 503-13.
[http://dx.doi.org/10.1056/NEJMoa1911793] [PMID: 31995683]
[http://dx.doi.org/10.1056/NEJMoa1911793] [PMID: 31995683]
[6]
Manning DJ, Ethell SC, Donovan T. Detection or decision errors? Missed lung cancer from the posteroanterior chest radiograph. Br J Radiol 2004; 77(915): 231-5.
[http://dx.doi.org/10.1259/bjr/28883951] [PMID: 15020365]
[http://dx.doi.org/10.1259/bjr/28883951] [PMID: 15020365]
[7]
Hossain R, Wu CC, de Groot PM, Carter BW, Gilman MD, Abbott GF. Missed lung cancer. Radiol Clin North Am 2018; 56(3): 365-75.
[http://dx.doi.org/10.1016/j.rcl.2018.01.004] [PMID: 29622072]
[http://dx.doi.org/10.1016/j.rcl.2018.01.004] [PMID: 29622072]
[8]
Nasrullah N, Sang J, Alam MS, Mateen M, Cai B, Hu H. Automated lung nodule detection and classification using deep learning combined with multiple strategies. Sensors (Basel) 2019; 19(17): 3722.
[http://dx.doi.org/10.3390/s19173722] [PMID: 31466261]
[http://dx.doi.org/10.3390/s19173722] [PMID: 31466261]
[9]
Rebouças Filho PP, Cortez PC, da Silva Barros AC, Albuquerque VH, Tavares JM. Novel and powerful 3D adaptive crisp active contour method applied in the segmentation of CT lung images. Med Image Anal 2017; 35: 503-16.
[http://dx.doi.org/10.1016/j.media.2016.09.002] [PMID: 27614793]
[http://dx.doi.org/10.1016/j.media.2016.09.002] [PMID: 27614793]
[10]
Filho P, Barros A, Ramalho G, et al. Automated recognition of lung diseases in CT images based on the optimum-path forest classifier. Neural Comput Appl 2019; 31(Suppl. 2): S901-14.
[http://dx.doi.org/10.1007/s00521-017-3048-y]
[http://dx.doi.org/10.1007/s00521-017-3048-y]
[11]
Bhandary A, Prabhu GA, Rajinikanth V, et al. Deep-learning framework to detect lung abnormality - a study with chest X-Ray and lung CT scan images. Pattern Recognit Lett 2020; 129: 271-8.
[http://dx.doi.org/10.1016/j.patrec.2019.11.013]
[http://dx.doi.org/10.1016/j.patrec.2019.11.013]
[12]
Pehrson LM, Nielsen MB, Ammitzbøl Lauridsen C. Automatic pulmonary nodule detection applying deep learning or machine learning algorithms to the LIDC-IDRI database: a systematic review. Diagnostics (Basel) 2019; 9(1): 29.
[http://dx.doi.org/10.3390/diagnostics9010029] [PMID: 30866425]
[http://dx.doi.org/10.3390/diagnostics9010029] [PMID: 30866425]
[13]
Li D, Mikela Vilmun B, Frederik Carlsen J, et al. The Performance of deep learning algorithms on automatic pulmonary nodule detection and classification tested on different datasets that are not derived from LIDC-IDRI: A systematic review. Diagnostics (Basel) 2019; 9(4): 207.
[http://dx.doi.org/10.3390/diagnostics9040207] [PMID: 31795409]
[http://dx.doi.org/10.3390/diagnostics9040207] [PMID: 31795409]
[14]
Halder A, Dey D, Sadhu AK. Lung nodule detection from feature engineering to deep learning in thoracic CT images: A comprehensive review. J Digit Imaging 2020; 33(3): 655-77.
[http://dx.doi.org/10.1007/s10278-020-00320-6] [PMID: 31997045]
[http://dx.doi.org/10.1007/s10278-020-00320-6] [PMID: 31997045]
[15]
Shen W, Zhou M, Yang F, Yang C, Tian J. Multi-scale convolutional neural networks for lung nodule classification. Inf Process Med Imaging 2015; 24: 588-99.
[http://dx.doi.org/10.1007/978-3-319-19992-4_46] [PMID: 26221705]
[http://dx.doi.org/10.1007/978-3-319-19992-4_46] [PMID: 26221705]
[16]
Ciompi F, Chung K, van Riel SJ, et al. Towards automatic pulmonary nodule management in lung cancer screening with deep learning. Sci Rep 2017; 7: 46479.
[http://dx.doi.org/10.1038/srep46479] [PMID: 28422152]
[http://dx.doi.org/10.1038/srep46479] [PMID: 28422152]
[17]
Causey JL, Zhang J, Ma S, et al. Highly accurate model for prediction of lung nodule malignancy with CT scans. Sci Rep 2018; 8(1): 9286.
[http://dx.doi.org/10.1038/s41598-018-27569-w] [PMID: 29915334]
[http://dx.doi.org/10.1038/s41598-018-27569-w] [PMID: 29915334]
[18]
Hua KL, Hsu CH, Hidayati SC, Cheng WH, Chen YJ. Computer-aided classification of lung nodules on computed tomography images via deep learning technique. Onco Targets Ther 2015; 8: 2015-22.
[PMID: 26346558]
[PMID: 26346558]
[19]
van de Sande D, van Genderen ME, Huiskens J, Gommers D, van Bommel J. Moving from bytes to bedside: A systematic review on the use of artificial intelligence in the intensive care unit. Intensive Care Med 2021; 47(7): 750-60.
[http://dx.doi.org/10.1007/s00134-021-06446-7] [PMID: 34089064]
[http://dx.doi.org/10.1007/s00134-021-06446-7] [PMID: 34089064]
[20]
Du Y, Li Q, Sidorenkov G, et al. Computed tomography screening for early lung cancer, COPD and cardiovascular disease in Shanghai: Rationale and design of a population-based comparative study. Acad Radiol 2021; 28(1): 36-45.
[http://dx.doi.org/10.1016/j.acra.2020.01.020] [PMID: 32151538]
[http://dx.doi.org/10.1016/j.acra.2020.01.020] [PMID: 32151538]
[21]
National Comprehensive Cancer Network. Lung cancer screening version 1. 2020. Available from: https://www.nccn.org/patients [Accessed on Feb 6, 2021]
[22]
Xia C, Rook M, Pelgrim GJ, et al. Early imaging biomarkers of lung cancer, COPD and coronary artery disease in the general population: Rationale and design of the ImaLife (Imaging in Lifelines) Study. Eur J Epidemiol 2020; 35(1): 75-86.
[http://dx.doi.org/10.1007/s10654-019-00519-0] [PMID: 31016436]
[http://dx.doi.org/10.1007/s10654-019-00519-0] [PMID: 31016436]
[23]
Liu K, Li Q, Ma J, et al. Evaluating a fully automated pulmonary nodule detection approach and Its impact on radiologist performance. Radiol Artif Intell 2019; 1(3): e180084.
[http://dx.doi.org/10.1148/ryai.2019180084] [PMID: 33937792]
[http://dx.doi.org/10.1148/ryai.2019180084] [PMID: 33937792]
[24]
Kozuka T, Matsukubo Y, Kadoba T, et al. Efficiency of a computer-aided diagnosis (CAD) system with deep learning in detection of pulmonary nodules on 1-mm-thick images of computed tomography. Jpn J Radiol 2020; 38(11): 1052-61.
[http://dx.doi.org/10.1007/s11604-020-01009-0] [PMID: 32592003]
[http://dx.doi.org/10.1007/s11604-020-01009-0] [PMID: 32592003]
[25]
Girshick R. Fast R-CNN.IEEE International Conference on Computer Vision; 2015 Dec 15-17;. Santiago, Chile: IEEE Xplor 2015; pp. 1440-8.
[26]
Ren S, He K, Girshick R, Sun J. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 2017; 39(6): 1137-49.
[http://dx.doi.org/10.1109/TPAMI.2016.2577031] [PMID: 27295650]
[http://dx.doi.org/10.1109/TPAMI.2016.2577031] [PMID: 27295650]
[27]
Huang G, Liu Z, Pleiss G, Van Der Maaten L, Weinberger K. Convolutional networks with dense connectivity. IEEE Trans Pattern Anal Mach Intell 2019. [Online ahead of print.].
[http://dx.doi.org/10.1109/TPAMI.2019.2918284] [PMID: 31135351]
[http://dx.doi.org/10.1109/TPAMI.2019.2918284] [PMID: 31135351]
[28]
Mercaldo ND, Lau KF, Zhou XH. Confidence intervals for predictive values with an emphasis to case-control studies. Stat Med 2007; 26(10): 2170-83.
[http://dx.doi.org/10.1002/sim.2677] [PMID: 16927452]
[http://dx.doi.org/10.1002/sim.2677] [PMID: 16927452]
[29]
Croswell JM, Baker SG, Marcus PM, et al. Cumulative incidence of false-positive test results in lung cancer screening: A randomized trial. Ann Intern Med 2010; 152(8): 505-12.
[http://dx.doi.org/10.7326/0003-4819-152-8-201004200-00007]
[http://dx.doi.org/10.7326/0003-4819-152-8-201004200-00007]
[30]
Bach PB, Mirkin JN, Oliver TK, et al. Benefits and harms of CT screening for lung cancer: A systematic review. JAMA 2012; 307(22): 2418-29.
[http://dx.doi.org/10.1001/jama.2012.5521] [PMID: 22610500]
[http://dx.doi.org/10.1001/jama.2012.5521] [PMID: 22610500]
[31]
Wood DE, Eapen GA, Ettinger DS, et al. Lung cancer screening. J Natl Compr Canc Netw 2012; 10(2): 240-65.
[http://dx.doi.org/10.6004/jnccn.2012.0022] [PMID: 22308518]
[http://dx.doi.org/10.6004/jnccn.2012.0022] [PMID: 22308518]
[32]
White CS, Salis AI, Meyer CA. Missed lung cancer on chest radiography and computed tomography: Imaging and medicolegal issues. J Thorac Imaging 1999; 14(1): 63-8.
[http://dx.doi.org/10.1097/00005382-199901000-00006] [PMID: 9894954]
[http://dx.doi.org/10.1097/00005382-199901000-00006] [PMID: 9894954]
[33]
Brogdon BG, Kelsey CA, Moseley RD Jr. Factors affecting perception of pulmonary lesions. Radiol Clin North Am 1983; 21(4): 633-54.
[PMID: 6657962]
[PMID: 6657962]
[34]
Zhang YP, Heuvelmans MA, Zhang H, et al. Changes in quantitative CT image features of ground-glass nodules in differentiating invasive pulmonary adenocarcinoma from benign and in situ lesions: histopathological comparisons. Clin Radiol 2018; 73(5): 504e9-504e16.
[http://dx.doi.org/10.1016/j.crad.2017.12.011]
[http://dx.doi.org/10.1016/j.crad.2017.12.011]
[35]
Del Ciello A, Franchi P, Contegiacomo A, Cicchetti G, Bonomo L, Larici AR. Missed lung cancer: When, where, and why? Diagn Interv Radiol 2017; 23(2): 118-26.
[http://dx.doi.org/10.5152/dir.2016.16187] [PMID: 28206951]
[http://dx.doi.org/10.5152/dir.2016.16187] [PMID: 28206951]
[36]
Doi K. Computer-aided diagnosis in medical imaging: Historical review, current status and future potential. Comput Med Imaging Graph 2007; 31(4-5): 198-211.
[http://dx.doi.org/10.1016/j.compmedimag.2007.02.002] [PMID: 17349778]
[http://dx.doi.org/10.1016/j.compmedimag.2007.02.002] [PMID: 17349778]
[37]
Zhang G, Jiang S, Yang Z, et al. Automatic nodule detection for lung cancer in CT images: A review. Comput Biol Med 2018; 103: 287-300.
[http://dx.doi.org/10.1016/j.compbiomed.2018.10.033] [PMID: 30415174]
[http://dx.doi.org/10.1016/j.compbiomed.2018.10.033] [PMID: 30415174]
[38]
Firmino M, Morais AH, Mendoça RM, Dantas MR, Hekis HR, Valentim R. Computer-aided detection system for lung cancer in computed tomography scans: Review and future prospects. Biomed Eng Online 2014; 13: 41.
[http://dx.doi.org/10.1186/1475-925X-13-41] [PMID: 24713067]
[http://dx.doi.org/10.1186/1475-925X-13-41] [PMID: 24713067]
[39]
Valente IR, Cortez PC, Neto EC, Soares JM, de Albuquerque VH, Tavares JM. Automatic 3D pulmonary nodule detection in CT images: A survey. Comput Methods Programs Biomed 2016; 124: 91-107.
[http://dx.doi.org/10.1016/j.cmpb.2015.10.006] [PMID: 26652979]
[http://dx.doi.org/10.1016/j.cmpb.2015.10.006] [PMID: 26652979]
[40]
Mehta K, Jain A, Mangalagiri J, Menon S, Nguyen P, Chapman DR. Lung nodule classification using biomarkers, volumetric radiomics, and 3D CNNs. J Digit Imaging 2021; 34(3): 647-6. [Online ahead of print.].
[http://dx.doi.org/10.1007/s10278-020-00417-y] [PMID: 33532893]
[http://dx.doi.org/10.1007/s10278-020-00417-y] [PMID: 33532893]
[41]
Gu X, Xie W, Fang Q, Zhao J, Li Q. The effect of pulmonary vessel suppression on computerized detection of nodules in chest CT scans. Med Phys 2020; 47(10): 4917-27.
[http://dx.doi.org/10.1002/mp.14401] [PMID: 32681587]
[http://dx.doi.org/10.1002/mp.14401] [PMID: 32681587]
[42]
Cho J, Kim J, Lee KJ, et al. Incidence lung cancer after a negative CT screening in the national lung screening trial: Deep learning-based detection of missed lung cancers. J Clin Med 2020; 9(12): 3908.
[http://dx.doi.org/10.3390/jcm9123908] [PMID: 33276433]
[http://dx.doi.org/10.3390/jcm9123908] [PMID: 33276433]
[43]
Hsu HH, Ko KH, Chou YC, et al. Performance and reading time of lung nodule identification on multidetector CT with or without an artificial intelligence-powered computer-aided detection system. Clin Radiol 2021; 76(8): 626.e23-32.
[http://dx.doi.org/10.1016/j.crad.2021.04.006] [PMID: 34023068]
[http://dx.doi.org/10.1016/j.crad.2021.04.006] [PMID: 34023068]
[44]
Murphy A, Skalski M, Gaillard F. The utilisation of convolutional neural networks in detecting pulmonary nodules: a review. Br J Radiol 2018; 91(1090): 20180028.
[http://dx.doi.org/10.1259/bjr.20180028] [PMID: 29869919]
[http://dx.doi.org/10.1259/bjr.20180028] [PMID: 29869919]
[45]
Ziyad SR, Radha V, Vayyapuri T. Overview of computer aided detection and computer aided diagnosis systems for lung nodule detection in computed tomography. Curr Med Imaging Rev 2020; 16(1): 16-26.
[http://dx.doi.org/10.2174/1573405615666190206153321] [PMID: 31989890]
[http://dx.doi.org/10.2174/1573405615666190206153321] [PMID: 31989890]
[46]
Nishio M, Nagashima C. Computer-aided diagnosis for lung cancer: Usefulness of nodule heterogeneity. Acad Radiol 2017; 24(3): 328-36.
[http://dx.doi.org/10.1016/j.acra.2016.11.007] [PMID: 28110797]
[http://dx.doi.org/10.1016/j.acra.2016.11.007] [PMID: 28110797]
[47]
Weikert T, Akinci D’Antonoli T, Bremerich J, Stieltjes B, Sommer G, Sauter AW. Evaluation of an AI-powered lung nodule algorithm for detection and 3D segmentation of primary lung tumors. Contrast Media Mol Imaging 2019; 2019: 1545747.
[http://dx.doi.org/10.1155/2019/1545747] [PMID: 31354393]
[http://dx.doi.org/10.1155/2019/1545747] [PMID: 31354393]
Article Metrics
42
1