Abstract
Background: Using a classification network to generate class activation mapping (CAM) is a mainstream method for weakly supervised semantic segmentation. However, for brain tumor images, CAM cannot fit the boundary of the tumor well.
Objective: To improve the performance of brain tumor CAM, we propose a weakly supervised learning strategy based on a multi-level sub-category and membership matrix.
Methods: Firstly, a multi-level sub-category strategy is used to intensively classify the data set. It allows the convolutional network to learn the in-depth characteristics of the input for enhancing CAM. Secondly, the idea of fuzzy clustering is introduced into model learning. The membership matrix is combined with CAM to construct the loss function.
Results: Exhaustive experiments on the brain tumor dataset BraTS2019 demonstrate that the proposed method can effectively improve the performance of CAM. Compared with the baseline method, our approach significantly improved by 17.1% using the common dice similarity coefficient evaluation approach, and compared with the recent study, our score also improved by almost 9%.
Conclusion: The proposed methods train the network under image-level labels and help the convolutional network mine the target boundary information. They can help CAM fit the target border more accurately.
Keywords: Weakly supervised segmentation, multi-level sub-category, membership matrix supervision, boundary exploration, tumor, medical imaging.
Graphical Abstract
[1]
Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems2012. Available from: https://proceedings.neurips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html
[2]
Deng J, Dong W, Socher R, Li L-J, Li K, Fei-Fei L. ImageNet: A large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition. 248-55.
[http://dx.doi.org/10.1109/CVPR.2009.5206848]
[http://dx.doi.org/10.1109/CVPR.2009.5206848]
[3]
Long J, Shelhamer E, Darrell T. Fully Convolutional Networks for Semantic Segmentation 2015. Available from: https://openaccess. thecvf.com/content_cvpr_2015/html/Long_Fully_Convolutional_Networks_2015_CVPR_paper.html
[http://dx.doi.org/10.1109/CVPR.2015.7298965]
[http://dx.doi.org/10.1109/CVPR.2015.7298965]
[4]
Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition 2015. Available from: http://arxiv.org/abs/1409.1556
[5]
He K, Zhang X, Ren S, Sun J. Deep Residual Learning for Image Recognition 2016. Available from: https://openaccess.thecvf.com/content_cvpr_2016/html/He_Deep_Residual_Learning_CVPR_2016_paper.html
[http://dx.doi.org/10.1109/CVPR.2016.90]
[http://dx.doi.org/10.1109/CVPR.2016.90]
[6]
Ronneberger O, Fischer P, Brox T. U-Net: Convolutional Networks
for Biomedical Image Segmentation.Medical Image Computing and
Computer-Assisted Intervention – MICCAI 2015. Lecture Notes in
Computer ScienceCham: Springer International Publishing 2015; 234-41.
[http://dx.doi.org/10.1007/978-3-319-24574-4_28]
[http://dx.doi.org/10.1007/978-3-319-24574-4_28]
[7]
Zhou Z-H. A brief introduction to weakly supervised learning. Natl Sci Rev 2018; 5(1): 44-53.
[http://dx.doi.org/10.1093/nsr/nwx106]
[http://dx.doi.org/10.1093/nsr/nwx106]
[8]
Menze BH, Jakab A, Bauer S, et al. The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Trans Med Imaging 2015; 34(10): 1993-2024.
[http://dx.doi.org/10.1109/TMI.2014.2377694] [PMID: 25494501]
[http://dx.doi.org/10.1109/TMI.2014.2377694] [PMID: 25494501]
[9]
Bakas S, Akbari H, Sotiras A, et al. Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Sci Data 2017; 4: 170117.
[http://dx.doi.org/10.1038/sdata.2017.117] [PMID: 28872634]
[http://dx.doi.org/10.1038/sdata.2017.117] [PMID: 28872634]
[10]
Bakas S, Reyes M, Jakab A, Bauer S, Rempfler M, Crimi A, et al. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BRATS challenge 2019. Available from: http://arxiv.org/abs/1811.02629
[11]
Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A. Learning Deep Features for Discriminative Localization. 2016. Available from: https://openaccess.thecvf.com/content_cvpr_2016/html/Zhou_Learning_Deep_Features_CVPR_2016_paper.html
[http://dx.doi.org/10.1109/CVPR.2016.319]
[http://dx.doi.org/10.1109/CVPR.2016.319]
[12]
Patel G, Dolz J. Weakly supervised segmentation with cross-modality equivariant constraints 2022. Available from: http://arxiv.org/abs/2104.02488
[14]
Wei Y, Feng J, Liang X, Cheng M-M, Zhao Y, Yan S. Object Region Mining With Adversarial Erasing: A Simple Classification to Semantic Segmentation Approach. 2017. Available from: https://openaccess. thecvf.com/content_cvpr_2017/html/Wei_Object_Region_Mining_CVPR_2017_paper.html
[http://dx.doi.org/10.1109/CVPR.2017.687]
[http://dx.doi.org/10.1109/CVPR.2017.687]
[15]
Zhang X, Wei Y, Feng J, Yang Y, Huang TS. Adversarial Complementary Learning for Weakly Supervised Object Localization 2018. Available from: https://openaccess.thecvf.com/content_cvpr_2018/html/Zhang_Adversarial_Complementary_Learning_CVPR_2018_paper.html
[http://dx.doi.org/10.1109/CVPR.2018.00144]
[http://dx.doi.org/10.1109/CVPR.2018.00144]
[16]
Chang Y-T, Wang Q, Hung W-C, Piramuthu R, Tsai Y-H, Yang M-H. Weakly-Supervised Semantic Segmentation via Sub-Category Exploration 2020. Available from: https://openaccess.thecvf.com/content_CVPR_2020/html/Chang_WeaklySupervised_Semantic_Segmentation_via_SubCategory_Exploration_CVPR_2020_paper.html
[http://dx.doi.org/10.1109/CVPR42600.2020.00901]
[http://dx.doi.org/10.1109/CVPR42600.2020.00901]
[17]
Kolesnikov A, Lampert CH. Seed, Expand and Constrain: Three Principles for Weakly-Supervised Image Segmentation.Computer Vision – ECCV 2016. Lecture Notes in Computer. Springer International Publishing: ScienceCham 2016; pp. 695-711.
[http://dx.doi.org/10.1007/978-3-319-46493-0_42]
[http://dx.doi.org/10.1007/978-3-319-46493-0_42]
[18]
Ahn J, Kwak S. Learning Pixel-Level Semantic Affinity With Image-Level Supervision for Weakly Supervised Semantic Segmentation 2018. Available from: https://openaccess.thecvf.com/content_cvpr_2018/html/Ahn_Learning_PixelLevel_Semantic_CVPR_2018_paper.html
[http://dx.doi.org/10.1109/CVPR.2018.00523]
[http://dx.doi.org/10.1109/CVPR.2018.00523]
[19]
Wang X, You S, Li X, Ma H. Weakly-Supervised Semantic Segmentation by Iteratively Mining Common Object Features 2018. Available from: https://openaccess.thecvf.com/content_cvpr_2018/html/Wang_Weakly-Supervised_Semantic_Segmentation_CVPR_2018_paper.html
[http://dx.doi.org/10.1109/CVPR.2018.00147]
[http://dx.doi.org/10.1109/CVPR.2018.00147]
[20]
Lee S, Lee M, Lee J, Shim H. Railroad is not a train: Saliency as pseudo-pixel supervision for weakly supervised semantic segmentation 2021. Available from: https://openaccess.thecvf.com/content/CVPR2021/html/Lee_Railroad_Is_Not_a_Train_Saliency_As_Pseudo-Pixel_Supervision_for_CVPR_2021_paper.html
[21]
Wang Y, Zhang J, Kan M, Shan S, Chen X. Self-Supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation 2020. Available from: https://openaccess.thecvf.com/content_CVPR_2020/html/Wang_Self-Supervised_Equivariant_Attention_Mechanism_for_Weakly_Supervised_Semantic_Segmentation_CVPR_2020_paper.html
[http://dx.doi.org/10.1109/CVPR42600.2020.01229]
[http://dx.doi.org/10.1109/CVPR42600.2020.01229]
[22]
Lee J, Yi J, Shin C, Yoon S. BBAM: Bounding box attribution map for weakly supervised semantic and instance segmentation 2021. Available from: https://openaccess.thecvf.com/content/CVPR2021/html/Lee_BBAM_Bounding_Box_Attribution_Map_for_Weakly_Supervised_Semantic_and_CVPR_2021_paper.html
[23]
Kulharia V, Chandra S, Agrawal A, Torr P, Tyagi A. Box2Seg: Attention Weighted Loss and Discriminative Feature Learning for Weakly Supervised Segmentation. Computer Vision – ECCV 2020. Lecture Notes in Computer. ScienceCham: Springer International Publishing 2020; pp. 290-308.
[http://dx.doi.org/10.1007/978-3-030-58583-9_18]
[http://dx.doi.org/10.1007/978-3-030-58583-9_18]
[24]
Wei Y, Ji S. Scribble-based weakly supervised deep learning for road surface extraction from remote sensing images. IEEE Trans Geosci Remote Sens 2022; 60: 1-12.
[http://dx.doi.org/10.1109/TGRS.2021.3061213]
[http://dx.doi.org/10.1109/TGRS.2021.3061213]
[25]
Liu X, Yuan Q, Gao Y, et al. Weakly supervised segmentation of COVID19 infection with scribble annotation on CT images. Pattern Recognit 2022; 122: 108341.
[http://dx.doi.org/10.1016/j.patcog.2021.108341] [PMID: 34565913]
[http://dx.doi.org/10.1016/j.patcog.2021.108341] [PMID: 34565913]
[26]
Li X, Zhou T, Li J, Zhou Y, Zhang Z. Group-wise semantic mining for weakly supervised semantic segmentation. 2020. Available from: http://arxiv.org/abs/2012.05007
[27]
Zhang D, Zhang H, Tang J, Hua X, Sun Q. Causal intervention for weakly-supervised semantic segmentation. 2020. Available from: http://arxiv.org/abs/2009.12547
[28]
Fan J, Zhang Z, Song C, Tan T. Learning Integral Objects With Intra-Class Discriminator for Weakly-Supervised Semantic Segmentation 2020. Available from: https://openaccess.thecvf.com/content_CVPR_2020/html/Fan_Learning_Integral_Objects_With_Intra-Class_Discriminator_for_Weakly-Supervised_Semantic_Segmentation_CVPR_2020_paper.html
[http://dx.doi.org/10.1109/CVPR42600.2020.00434]
[http://dx.doi.org/10.1109/CVPR42600.2020.00434]
[29]
Ji Z, Shen Y, Ma C, Gao M. Scribble-Based Hierarchical Weakly Supervised Learning for Brain Tumor Segmentation. Medical Image Computing and Computer Assisted Intervention – MICCAI 2019 Lecture Notes in Computer. ScienceCham: Springer International Publishing 2019; pp. 175-83.
[http://dx.doi.org/10.1007/978-3-030-32248-9_20]
[http://dx.doi.org/10.1007/978-3-030-32248-9_20]
[30]
Valvano G, Leo A, Tsaftaris SA. Learning to segment from scribbles using multi-scale adversarial attention gates. IEEE Trans Med Imaging 2021; 40(8): 1990-2001.
[http://dx.doi.org/10.1109/TMI.2021.3069634] [PMID: 33784616]
[http://dx.doi.org/10.1109/TMI.2021.3069634] [PMID: 33784616]
[31]
Kervadec H, Dolz J, Wang S, Granger E, Ayed IB. Bounding boxes for weakly supervised segmentation: Global constraints get close to full supervision. 2020. Available from: http://arxiv.org/abs/2004.06816
[32]
Wang J, Xia B. Bounding box tightness prior for weakly supervised image segmentation 2021.Available from: .
[http://dx.doi.org/10.1007/978-3-030-87196-3_49]
[http://dx.doi.org/10.1007/978-3-030-87196-3_49]
[33]
Peng J, Kervadec H, Dolz J, Ayed IB, Pedersoli M, Desrosiers C. Discretely-constrained deep network for weakly supervised segmentation. 2019. Available from: http://arxiv.org/abs/1908.05770
[34]
Belharbi S, Rony J, Dolz J, Ayed IB, Mccaffrey L, Granger E. Deep interpretable classification and weakly-supervised segmentation of histology images via max-min uncertainty. IEEE Trans Med Imaging 2022; 41(3): 702-14.
[http://dx.doi.org/10.1109/TMI.2021.3123461] [PMID: 34705638]
[http://dx.doi.org/10.1109/TMI.2021.3123461] [PMID: 34705638]
[35]
Feng X, Yang J, Laine AF, Angelini ED. Discriminative localization in CNNs for weakly-supervised segmentation of pulmonary nodules. Med Image Comput Comput Assist Interv 2017; 10435: 568-76.
[http://dx.doi.org/10.1007/978-3-319-66179-7_65]
[http://dx.doi.org/10.1007/978-3-319-66179-7_65]
[36]
Izadyyazdanabadi M, Belykh E, Cavallo C, Zhao X, Gandhi S, Moreira LB, et al. Weakly-Supervised Learning-Based Feature Localization for Confocal Laser Endomicroscopy Glioma Images. Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 Lecture Notes in Computer ScienceCham. Springer International Publishing 2018; pp. 300-8.
[http://dx.doi.org/10.1007/978-3-030-00934-2_34]
[http://dx.doi.org/10.1007/978-3-030-00934-2_34]
[37]
Ahn J, Cho S, Kwak S. Weakly Supervised Learning of Instance Segmentation With Inter-Pixel Relations. 2019. Available from: https://openaccess.thecvf.com/content_CVPR_2019/html/Ahn_Weakly_Supervised_Learning_of_Instance_Segmentation_With_Inter-Pixel_Relations_CVPR_2019_paper.html
[http://dx.doi.org/10.1109/CVPR.2019.00231]
[http://dx.doi.org/10.1109/CVPR.2019.00231]
[38]
Viniavskyi O, Dobko M, Dobosevych O. Weakly-Supervised Segmentation for Disease Localization in Chest X-Ray ImagesArtificial Intelligence in Medicine Lecture Notes in Computer ScienceCham. Springer International Publishing 2020; pp. 249-59.
[http://dx.doi.org/10.1007/978-3-030-59137-3_23]
[http://dx.doi.org/10.1007/978-3-030-59137-3_23]
[39]
Agarwal V, Tang Y, Xiao J, Summers RM. Weakly supervised lesion co-segmentation on CT scans 2020. Available from: http://arxiv.org/abs/2001.09174
[http://dx.doi.org/10.1109/ISBI45749.2020.9098657]
[http://dx.doi.org/10.1109/ISBI45749.2020.9098657]
[40]
Kingma DP, Ba J. Adam: A method for stochastic optimization Available from: http://arxiv.org/abs/1412.6980
[41]
Pal NR, Bezdek JC. On cluster validity for the fuzzy c-means model. IEEE Trans Fuzzy Syst 1995; 3(3): 370-9.
[http://dx.doi.org/10.1109/91.413225]
[http://dx.doi.org/10.1109/91.413225]
Article Metrics
5