Abstract
Background: Due to the significant variances in their shape and size, it is a challenging task to automatically segment gliomas. To improve the performance of glioma segmentation tasks, this paper proposed a multilevel attention pyramid scene parsing network (MLAPSPNet) that aggregates the multiscale context and multilevel features.
Methods: First, T1 pre-contrast, T2-weighted fluid-attenuated inversion recovery (FLAIR) and T1 post-contrast sequences of each slice are combined to form the input. Afterwards, image normalization and augmentation techniques are applied to accelerate the training process and avoid overfitting, respectively. Furthermore, the proposed MLAPSPNet that introduces multilevel pyramid pooling modules (PPMs) and attention gates is constructed. Eventually, the proposed network is compared with some existing networks.
Results: The dice similarity coefficient (DSC), sensitivity and Jaccard score of the proposed system can reach 0.885, 0.933 and 0.8, respectively. The introduction of multilevel pyramid pooling modules and attention gates can improve the DSC by 0.029 and 0.022, respectively. Moreover, compared with Res-UNet, Dense-UNet, residual channel attention UNet (RCA-UNet), DeepLab V3+ and UNet++, the DSC is improved by 0.032, 0.026, 0.014, 0.041 and 0.011, respectively.
Conclusion: The proposed multilevel attention pyramid scene parsing network can achieve stateof- the-art performance, and the introduction of multilevel pyramid pooling modules and attention gates can improve the performance of glioma segmentation tasks.
Keywords: Gliomas, segmentation, magnetic resonance imaging, MLAPSPNet, attention gates, feature fusion, context.
Graphical Abstract
[1]
Patel AP, Fisher JL, Nichols E, et al. GBD 2016 Brain and Other CNS Cancer Collaborators. Global, regional, and national burden of brain and other CNS cancer, 1990-2016: a systematic analysis for the Global Burden of Disease Study 2016. Lancet Neurol 2019; 18(4): 376-93.
[http://dx.doi.org/10.1016/S1474-4422(18)30468-X] [PMID: 30797715]
[http://dx.doi.org/10.1016/S1474-4422(18)30468-X] [PMID: 30797715]
[2]
Gupta N, Bhatele P, Khanna P. Identification of Gliomas from brain MRI through adaptive segmentation and run length of centralized patterns. J Comput Sci 2017; 25: 213-20.
[http://dx.doi.org/10.1016/j.jocs.2017.02.009]
[http://dx.doi.org/10.1016/j.jocs.2017.02.009]
[3]
Chen SC, Ding CX, Liu MF. Dual-force convolutional neural networks for accurate brain tumor segmentation. Pattern Recognit 2019; 88: 90-100.
[http://dx.doi.org/10.1016/j.patcog.2018.11.009]
[http://dx.doi.org/10.1016/j.patcog.2018.11.009]
[4]
Zhao X, Wu Y, Song G, Li Z, Zhang Y, Fan Y. A deep learning model integrating FCNNs and CRFs for brain tumor segmentation. Med Image Anal 2018; 43: 98-111.
[http://dx.doi.org/10.1016/j.media.2017.10.002] [PMID: 29040911]
[http://dx.doi.org/10.1016/j.media.2017.10.002] [PMID: 29040911]
[5]
Yang TJ, Song JK, Li L. A deep learning model integrating SK-TPCNN and random forests for brain tumor segmentation in MRI. Biocybern Biomed Eng 2019; 39(3): 613-23.
[http://dx.doi.org/10.1016/j.bbe.2019.06.003]
[http://dx.doi.org/10.1016/j.bbe.2019.06.003]
[6]
Soltaninejad M, Yang G, Lambrou T, et al. Automated brain tumour detection and segmentation using superpixel-based extremely randomized trees in FLAIR MRI. Int J CARS 2017; 12(2): 183-203.
[http://dx.doi.org/10.1007/s11548-016-1483-3] [PMID: 27651330]
[http://dx.doi.org/10.1007/s11548-016-1483-3] [PMID: 27651330]
[7]
Tong JJ, Zhao YL, Zhang P, Chen LY, Jiang LR. MRI brain tumor segmentation based on texture features and kernel sparse coding. Biomed Signal Proces 2019; 47: 387-92.
[http://dx.doi.org/10.1016/j.bspc.2018.06.001]
[http://dx.doi.org/10.1016/j.bspc.2018.06.001]
[8]
Shelhamer E, Long J, Darrell T. Fully Convolutional Networks for Semantic Segmentation. IEEE Trans Pattern Anal Mach Intell 2017; 39(4): 640-51.
[http://dx.doi.org/10.1109/TPAMI.2016.2572683] [PMID: 27244717]
[http://dx.doi.org/10.1109/TPAMI.2016.2572683] [PMID: 27244717]
[9]
Wang Y, Li C, Zhu T, Zhang J. Multimodal brain tumor image segmentation using WRN-PPNet. Comput Med Imaging Graph 2019; 75: 56-65.
[http://dx.doi.org/10.1016/j.compmedimag.2019.04.001] [PMID: 31154088]
[http://dx.doi.org/10.1016/j.compmedimag.2019.04.001] [PMID: 31154088]
[10]
Hao SJ, Zhou Y, Guo YR. A Brief Survey on Semantic Segmentation with Deep Learning. Neurocomputing 2020; 406: 302-21.
[http://dx.doi.org/10.1016/j.neucom.2019.11.118]
[http://dx.doi.org/10.1016/j.neucom.2019.11.118]
[11]
Yu F, Koltun V. Multi-Scale Context Aggregation by Dilated Convolutions. Proceedings of the 4th International Conference on Learning Representations. 2016 May 2-4; San Juan, Puerto Rico. 2016.
[12]
Zhao H, Shi J, Qi X, Wang X, Jia J. Pyramid scene parsing network. Proceedings of the 30-th IEEE Conference on Computer Vision and Pattern Recognition. 2017 July 21-26; Honolulu, HI, United states. New York: IEEE 2017.
[13]
Chen LC, Papandreou G, Kokkinos I, Murphy K, Yuille AL. DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs. IEEE Trans Pattern Anal Mach Intell 2018; 40(4): 834-48.
[http://dx.doi.org/10.1109/TPAMI.2017.2699184] [PMID: 28463186]
[http://dx.doi.org/10.1109/TPAMI.2017.2699184] [PMID: 28463186]
[14]
Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation. Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention. 2015 Oct 5-9; Munich, Germany. Switzerland: Springer 2015.
[http://dx.doi.org/10.1007/978-3-319-24574-4_28]
[http://dx.doi.org/10.1007/978-3-319-24574-4_28]
[15]
Ribalta Lorenzo P, Nalepa J, Bobek-Billewicz B, et al. Segmenting brain tumors from FLAIR MRI using fully convolutional neural networks. Comput Methods Programs Biomed 2019; 176: 135-48.
[http://dx.doi.org/10.1016/j.cmpb.2019.05.006] [PMID: 31200901]
[http://dx.doi.org/10.1016/j.cmpb.2019.05.006] [PMID: 31200901]
[16]
Li H, Li A, Wang M. A novel end-to-end brain tumor segmentation method using improved fully convolutional networks. Comput Biol Med 2019; 108: 150-60.
[http://dx.doi.org/10.1016/j.compbiomed.2019.03.014] [PMID: 31005007]
[http://dx.doi.org/10.1016/j.compbiomed.2019.03.014] [PMID: 31005007]
[17]
Naser MA, Deen MJ. Brain tumor segmentation and grading of lower-grade glioma using deep learning in MRI images. Comput Biol Med 2020; 121
[http://dx.doi.org/10.1016/j.compbiomed.2020.103758] [PMID: 32568668]
[http://dx.doi.org/10.1016/j.compbiomed.2020.103758] [PMID: 32568668]
[18]
Clark K, Vendt B, Smith K, et al. The Cancer Imaging Archive (TCIA): maintaining and operating a public information repository. J Digit Imaging 2013; 26(6): 1045-57.
[http://dx.doi.org/10.1007/s10278-013-9622-7] [PMID: 23884657]
[http://dx.doi.org/10.1007/s10278-013-9622-7] [PMID: 23884657]
[19]
Buda M, Saha A, Mazurowski MA. Association of genomic subtypes of lower-grade gliomas with shape features automatically extracted by a deep learning algorithm. Comput Biol Med 2019; 109: 218-25.
[http://dx.doi.org/10.1016/j.compbiomed.2019.05.002] [PMID: 31078126]
[http://dx.doi.org/10.1016/j.compbiomed.2019.05.002] [PMID: 31078126]
[20]
Scheufele K, Mang A, Gholami A, Davatzikos C, Biros G, Mehl M. Coupling brain-tumor biophysical models and diffeomorphic image registration. Comput Methods Appl Mech Eng 2019; 347: 533-67.
[http://dx.doi.org/10.1016/j.cma.2018.12.008] [PMID: 31857736]
[http://dx.doi.org/10.1016/j.cma.2018.12.008] [PMID: 31857736]
[21]
Liu SP, Tian GH, Xu Y. A novel scene classification model combining ResNet based transfer learning and data augmentation with a filter. Neurocomputing 2019; 338: 191-206.
[http://dx.doi.org/10.1016/j.neucom.2019.01.090]
[http://dx.doi.org/10.1016/j.neucom.2019.01.090]
[22]
He K, Zhang X, Ren S, Sun J. Deep Residual Learning for Image Recognition. Proceedings of the 29th IEEE Conference on Computer Vision and Pattern Recognition. 2016 June 26 - July 1; Las Vegas, NV, United states. New York: IEEE 2017.
[23]
Kawaguchi K, Bengio Y. Depth with nonlinearity creates no bad local minima in ResNets. Neural Netw 2019; 118: 167-74.
[http://dx.doi.org/10.1016/j.neunet.2019.06.009] [PMID: 31295691]
[http://dx.doi.org/10.1016/j.neunet.2019.06.009] [PMID: 31295691]
[24]
Liu B, Liu Q, Zhu ZY, Zhang TP, Yang Y. MSST-ResNet: Deep multi-scale spatiotemporal features for robust visual object tracking. Knowl Base Syst 2019; 164: 235-52.
[http://dx.doi.org/10.1016/j.knosys.2018.10.044]
[http://dx.doi.org/10.1016/j.knosys.2018.10.044]
[25]
Tang P, Liang Q, Yan X, et al. Efficient skin lesion segmentation using separable-Unet with stochastic weight averaging. Comput Methods Programs Biomed 2019; 178: 289-301.
[http://dx.doi.org/10.1016/j.cmpb.2019.07.005] [PMID: 31416556]
[http://dx.doi.org/10.1016/j.cmpb.2019.07.005] [PMID: 31416556]
[26]
Yang J, Faraji M, Basu A. Robust segmentation of arterial walls in intravascular ultrasound images using Dual Path U-Net. Ultrasonics 2019; 96: 24-33.
[http://dx.doi.org/10.1016/j.ultras.2019.03.014] [PMID: 30947071]
[http://dx.doi.org/10.1016/j.ultras.2019.03.014] [PMID: 30947071]
[27]
Matuszewski DJ, Sintorn IM. Reducing the U-Net size for practical scenarios: Virus recognition in electron microscopy images. Comput Methods Programs Biomed 2019; 178: 31-9.
[http://dx.doi.org/10.1016/j.cmpb.2019.05.026] [PMID: 31416558]
[http://dx.doi.org/10.1016/j.cmpb.2019.05.026] [PMID: 31416558]
[28]
Rundo L, Han C, Nagano Y, et al. USE-Net: Incorporating Squeeze-and-Excitation blocks into U-Net for prostate zonal segmentation of multi-institutional MRI datasets. Neurocomputing 2019; 365: 31-43.
[http://dx.doi.org/10.1016/j.neucom.2019.07.006]
[http://dx.doi.org/10.1016/j.neucom.2019.07.006]
[29]
Liu Z, Song YQ, Sheng VS, et al. Liver CT sequence segmentation based with improved U-Net and graph cut. Expert Syst Appl 2019; 126: 54-63.
[http://dx.doi.org/10.1016/j.eswa.2019.01.055]
[http://dx.doi.org/10.1016/j.eswa.2019.01.055]
[30]
Dash M, Londhe ND, Ghosh S, Semwal A, Sonawane RS. PsLSNet: Automated psoriasis skin lesion segmentation using modified U-Net-based fully convolutional network. Biomed Signal Proces 2019; 52: 226-37.
[http://dx.doi.org/10.1016/j.bspc.2019.04.002]
[http://dx.doi.org/10.1016/j.bspc.2019.04.002]
[31]
Zhang Y, Chen JH, Chang KT, et al. Automatic Breast and Fibroglandular Tissue Segmentation in Breast MRI Using Deep Learning by a Fully-Convolutional Residual Neural Network U-Net. Acad Radiol 2019; 26(11): 1526-35.
[http://dx.doi.org/10.1016/j.acra.2019.01.012] [PMID: 30713130]
[http://dx.doi.org/10.1016/j.acra.2019.01.012] [PMID: 30713130]
[32]
Oh SL, Ng EYK, Tan RS, Acharya UR. Automated beat-wise arrhythmia diagnosis using modified U-net on extended electrocardiographic recordings with heterogeneous arrhythmia types. Comput Biol Med 2019; 105: 92-101.
[http://dx.doi.org/10.1016/j.compbiomed.2018.12.012] [PMID: 30599317]
[http://dx.doi.org/10.1016/j.compbiomed.2018.12.012] [PMID: 30599317]
[33]
Lucchi A, Li Y, Boix X, Smith K, Fua P. Are spatial and global constraints really necessary for segmentation? Proceedings of the IEEE International Conference on Computer Vision. 2011 November 6-13; Barcelona, Spain. New York: IEEE 2011.
[http://dx.doi.org/10.1109/ICCV.2011.6126219]
[http://dx.doi.org/10.1109/ICCV.2011.6126219]
[34]
Tong Q, Li C, Si W, et al. RIANet: Recurrent interleaved attention network for cardiac MRI segmentation. Comput Biol Med 2019; 109: 290-302.
[http://dx.doi.org/10.1016/j.compbiomed.2019.04.042] [PMID: 31100582]
[http://dx.doi.org/10.1016/j.compbiomed.2019.04.042] [PMID: 31100582]
[35]
Schlemper J, Oktay O, Schaap M, et al. Attention gated networks: Learning to leverage salient regions in medical images. Med Image Anal 2019; 53: 197-207.
[http://dx.doi.org/10.1016/j.media.2019.01.012] [PMID: 30802813]
[http://dx.doi.org/10.1016/j.media.2019.01.012] [PMID: 30802813]
[36]
Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep convolutional neural networks. Proceedings of the 26th Annual Conference on Neural Information Processing Systems. 2012 December 3-6; Lake Tahoe, NV, United states. 2012.
[37]
Zhao H, Shi J, Qi X, Wang X, Jia J. Pyramid Scene Parsing Network. Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition. 2017 July 21-26; Honolulu, HI, United states. New York: IEEE 2017.
[38]
Gragera A, Suppakitpaisarn V. Relaxed triangle inequality ratio of the Sørensen–Dice and Tversky indexes. Theor Comput Sci 2018; 718: 37-45.
[http://dx.doi.org/10.1016/j.tcs.2017.01.004]
[http://dx.doi.org/10.1016/j.tcs.2017.01.004]
[39]
An FP, Liu ZW. Medical image segmentation algorithm based on feedback mechanism convolutional neural network. Biomed Signal Proces 2019; 47
[40]
Shen T, Gou C, Wang FY, He Z, Chen W. Learning from adversarial medical images for X-ray breast mass segmentation. Comput Methods Programs Biomed 2019; 180
[http://dx.doi.org/10.1016/j.cmpb.2019.105012] [PMID: 31421601]
[http://dx.doi.org/10.1016/j.cmpb.2019.105012] [PMID: 31421601]
[41]
Lin TY, Goyal P, Girshick R, He K, Dollar P. Focal Loss for Dense Object Detection. IEEE Trans Pattern Anal Mach Intell 2020; 42(2): 318-27.
[http://dx.doi.org/10.1109/TPAMI.2018.2858826] [PMID: 30040631]
[http://dx.doi.org/10.1109/TPAMI.2018.2858826] [PMID: 30040631]
[42]
Hashemi SR, Mohseni SS, Erdogmus D, Prabhu SP, Warfield SK, Gholipour A. Asymmetric Loss Functions and Deep Densely-Connected Networks for Highly-Imbalanced Medical Image Segmentation: Application to Multiple Sclerosis Lesion Detection. 2019; 7: 1721-35.
[43]
Selvaraju R, Cogswell M, Das A, Vedantam R, Parikh D, Batra D. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. Int J Comput Vis 2020; 128(2): 336-59.
[http://dx.doi.org/10.1007/s11263-019-01228-7]
[http://dx.doi.org/10.1007/s11263-019-01228-7]
[44]
Zhang ZX, Liu QJ, Wang YH. Road Extraction by Deep Residual U-Net. IEEE Geosci Remote S 2018; 15(5): 749-53.
[http://dx.doi.org/10.1109/LGRS.2018.2802944]
[http://dx.doi.org/10.1109/LGRS.2018.2802944]
[45]
Jegou S, Drozdzal M, Vazquez D, Romero A, Bengio Y. The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation. Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2017 july 21-26; Honolulu, HI, United states. New York: IEEE 2017.
[http://dx.doi.org/10.1109/CVPRW.2017.156]
[http://dx.doi.org/10.1109/CVPRW.2017.156]
[46]
F. ZH. Proceedings of the 22th International Conference on Medical Image Computing and Computer-Assisted Intervention. 2019 Oct 13-17; Shenzhen, China. Switzerland: Springer 2019.
[47]
Chen LC, Zhu YK, Papandreou G, Schroff F, Adam H. Encoder-decoder with atrous separable convolution for semantic image segmentation. Proceedings of the 15th European Conference on Computer Vision. 2018 September 8-14; Munich, Germany. Switzerland: Springer 2018.
[http://dx.doi.org/10.1007/978-3-030-01234-2_49]
[http://dx.doi.org/10.1007/978-3-030-01234-2_49]
[48]
Zhou Z, Rahman S, Md M, Tajbakhsh N, Liang JM. Unet++: A nested u-net architecture for medical image segmentation. Proceedings of the 21st International Conference on Medical Image Computing and Computer-Assisted Intervention. 2018 September 16-20; Granada, Spain. Switzerland: Springer 2019.
[49]
Azad R, Asadi-Aghbolaghi M, Fathy M. Bi-directional ConvLSTM U-net with densley connected convolutions. Proceedings of the 17th IEEE/CVF International Conference on Computer Vision Workshop. 2019 October 27-28; Seoul, Korea.
[http://dx.doi.org/10.1109/ICCVW.2019.00052]
[http://dx.doi.org/10.1109/ICCVW.2019.00052]
Article Metrics
10
10