Abstract
Background: Liver and tumor segmentation from CT images is a complex and crucial step in achieving full-course adaptive radiotherapy and also plays an essential role in computer-aided clinical diagnosis systems. Deep learning-based methods play an important role in achieving automatic segmentation.
Objective: This research aims to improve liver tumor detection performance by proposing a dual path feature extracting strategy and employing Swin-Transformer.
Methods: The hierarchical Swin-Transformer is embedded into the encoder and decoder and combined with CNN to form a dual coding path structure incorporating long-range dependencies and multi-scale contextual connections to capture coarse-tuned features at different semantic scales. The features of the two encoding paths and the upsampling path are fused, tested and validated with LITS and in-house datasets.
Results: The proposed method has a DG of 97.95% and a DC of 96.2% for liver segmentation; a DG of 80.6% and a DC of 68.1% for tumor segmentation; and a classification study of the tumor dataset shows a DG of 91.1% and a DC of 87.2% for large and continuous tumors and a DG of 71.7% and a DC of 66.4% for small and scattered tumors.
Conclusion: Swin-Transformer can be used as a robust encoder for medical image segmentation networks and, combined with CNN networks, can better recover local spatial information and enhance feature representation. Accurate localization before segmentation can achieve better results for small and scattered tumors.
[1]
Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. arXivorg 2014; 2014: 1411.4038.
[2]
He X, Zhou Y, Zhao J, Zhang D, Yao R, Xue Y. Swin transformer embedding UNet for remote sensing image semantic segmentation. IEEE Trans Geosci Remote Sens 2022; 60: 3144165.
[http://dx.doi.org/10.1109/TGRS.2022.3144165]
[http://dx.doi.org/10.1109/TGRS.2022.3144165]
[3]
Han X. MR-based synthetic CT generation using a deep convolutional neural network method. Med Phys 2017; 44(4): 1408-19.
[http://dx.doi.org/10.1002/mp.12155] [PMID: 28192624]
[http://dx.doi.org/10.1002/mp.12155] [PMID: 28192624]
[4]
Christ PF, Ettlinger F, Ezzeldin M, et al. Automatic liver and tumor segmentation of CT and MRI volumes using cascaded fully convolutional neural networks. arXiv 2017; 2017: 1702.05970.
[5]
Ronneberger O, Fischer P, Brox T. U-Net: Convolutional networks for biomedical image segmentation. Lect Notes Comput Sci 2015; 9351: 234-41.
[http://dx.doi.org/10.1007/978-3-319-24574-4_28]
[http://dx.doi.org/10.1007/978-3-319-24574-4_28]
[6]
Zhou Z, Rahman Siddiquee MM, Tajbakhsh N, Liang J. UNet++: A nested U-net architecture for medical image segmentation. In: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support. Springer 2018; pp. 3-11.
[7]
Xiao X, Lian S, Luo Z, Li S. Weighted res-UNet for high-quality retina vessel segmentation. Int Conf Inform Technol Med Educat (ITME) 2018; 2018: 327-31.
[8]
Schlemper J, Oktay O, Schaap M, et al. Attention gated networks: Learning to leverage salient regions in medical images. Med Image Anal 2019; 53: 197-207.
[http://dx.doi.org/10.1016/j.media.2019.01.012] [PMID: 30802813]
[http://dx.doi.org/10.1016/j.media.2019.01.012] [PMID: 30802813]
[9]
Petit O, Thome N, Rambour C, Themyr L, Collins T, Soler L. U-net transformer: Self and cross attention for medical image segmentation. Machine Learn Med Imag 2021; 2021: 267-76.
[10]
Huang H, Lin L, Tong R, et al. UNet 3+: A full-scale connected UNet for medical image segmentation. IEEE Xplore 2020; 2020: 1055-9.
[11]
Chen J, Lu Y, Yu Q, et al. TransUNet: Transformers make strong encoders for medical image segmentation. arXiv 2021; 2021: 210204306.
[12]
Pan X, Ge C, Lu R, et al. On the integration of self-attention and convolution. arXiv 2022; 2022: 111.14556.
[13]
Zhang H, Wu C, Zhang Z, et al. ResNeSt: Split-attention networks. 2022. Available from: openaccess.thecvf.com
[14]
Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need. Adv Neural Inf Process Syst 2017; 30: 5998-6008.
[15]
Zheng S, Lu J, Zhao H, et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 20-25 June 2021; Nashville, TN, USA.
[16]
Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv 2020; 2020: 201011929.
[17]
Zhang Y, Liu H, Hu Q. Transfuse: Fusing transformers and cnns for medical image segmentation. arXiv 2021; 2021: 210208005.
[http://dx.doi.org/10.1007/978-3-030-87193-2_2]
[http://dx.doi.org/10.1007/978-3-030-87193-2_2]
[18]
Codalab. The leaderboard of the LiTS.. Available from: https://competitions.codalab.org/competitions/17094#results
[19]
Araújo JDL, da Cruz LB, Diniz JOB, et al. Liver segmentation from computed tomography images using cascade deep learning. Comput Biol Med 2022; 140: 105095.
[http://dx.doi.org/10.1016/j.compbiomed.2021.105095] [PMID: 34902610]
[http://dx.doi.org/10.1016/j.compbiomed.2021.105095] [PMID: 34902610]
[20]
Gao Q, Almekkawy M. ASU-Net++: A nested U-Net with adaptive feature extractions for liver tumor segmentation. Comput Biol Med 2021; 136: 104688.
[http://dx.doi.org/10.1016/j.compbiomed.2021.104688] [PMID: 34523421]
[http://dx.doi.org/10.1016/j.compbiomed.2021.104688] [PMID: 34523421]
[21]
Li S, Tso GKF, He K. Bottleneck feature supervised U-Net for pixel-wise liver and tumor segmentation. Expert Syst Appl 2020; 145: 113131.
[http://dx.doi.org/10.1016/j.eswa.2019.113131]
[http://dx.doi.org/10.1016/j.eswa.2019.113131]
[22]
Liu T, Liu J, Ma Y, et al. Spatial feature fusion convolutional network for liver and liver tumor segmentation from CT images. Med Phys 2021; 48(1): 264-72.
[http://dx.doi.org/10.1002/mp.14585] [PMID: 33159809]
[http://dx.doi.org/10.1002/mp.14585] [PMID: 33159809]
[23]
Khan RA, Luo Y, Wu FX. RMS-UNet: Residual multi-scale UNet for liver and lesion segmentation. Artif Intell Med 2022; 124: 102231.
[http://dx.doi.org/10.1016/j.artmed.2021.102231] [PMID: 35115126]
[http://dx.doi.org/10.1016/j.artmed.2021.102231] [PMID: 35115126]
[24]
Meng L, Tian Y, Bu S. Liver tumor segmentation based on 3D convolutional neural network with dual scale. J Appl Clin Med Phys 2020; 21(1): 144-57.
[http://dx.doi.org/10.1002/acm2.12784] [PMID: 31793212]
[http://dx.doi.org/10.1002/acm2.12784] [PMID: 31793212]
[25]
Kaur A, Chauhan APS, Aggarwal AK. An automated slice sorting technique for multi-slice computed tomography liver cancer images using convolutional network. Expert Syst Appl 2021; 186: 115686.
[http://dx.doi.org/10.1016/j.eswa.2021.115686]
[http://dx.doi.org/10.1016/j.eswa.2021.115686]
[26]
Alirr OI. Deep learning and level set approach for liver and tumor segmentation from CT scans. J Appl Clin Med Phys 2020; 21(10): 200-9.
[http://dx.doi.org/10.1002/acm2.13003] [PMID: 33113290]
[http://dx.doi.org/10.1002/acm2.13003] [PMID: 33113290]
