Abstract
Background: Collaborative Representation (CR) has been widely used in Single Image Super Resolution (SISR) with the assumption that Low-resolution (LR) and high-resolution (HR) features can be linearly represented by neighborhoods and share consistent CR coefficients. Numerous patents and journal papers have been published. However, this CR consistency does not hold in the reconstruction phase, which leads to degraded performance.
Methods: To fulfill this gap, we propose a novel bi-directional projection model (BDPM) to establish a bi-directional mapping between LR and HR features without any consistency constraint. The multiple projection matrices are offline computed to reduce reconstruction time greatly. We further develop several strategies to extract features and group neighborhoods such that local structures can be preserved better.
Results: Compared to the learning-based methods, BDPM is about 2 to 10 times faster and compared to the reconstruction-based methods, it is about 500 to 2,000 times faster.
Conclusion: The empirical studies verify the effectiveness of BDPM and extensive experimental results demonstrate that BDPM achieves better SISR performance than many state-of-the-arts.
Graphical Abstract
[1]
P.V. Arun, K.M. Buddhiraju, A. Porwal, and J. Chanussot, "CNN based spectral super-resolution of remote sensing images", Signal Processing, vol. 169, p. 107394, 2020.
[http://dx.doi.org/10.1016/j.sigpro.2019.107394]
[http://dx.doi.org/10.1016/j.sigpro.2019.107394]
[2]
M. Zhang, and Q. Ling, "Supervised pixel-wise GAN for face super-resolution", IEEE Trans. Multimed., vol. 23, pp. 1938-1950, 2021.
[http://dx.doi.org/10.1109/TMM.2020.3006414]
[http://dx.doi.org/10.1109/TMM.2020.3006414]
[3]
Y. Liu, S. Zhang, J. Xu, J. Yang, and Y.W. Tai, "An accurate and lightweight method for human body image super-resolution", IEEE Trans. Image Process., vol. 30, pp. 2888-2897, 2021.
[http://dx.doi.org/10.1109/TIP.2021.3055737] [PMID: 33539298]
[http://dx.doi.org/10.1109/TIP.2021.3055737] [PMID: 33539298]
[4]
P. Yi, Z. Wang, K. Jiang, J. Jiang, T. Lu, and J. Ma, "A progressive fusion generative adversarial network for realistic and consistent video super-resolution", IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 5, pp. 2264-2280, 2022.
[PMID: 33270559]
[PMID: 33270559]
[5]
C. Zhao, B.E. Dewey, D.L. Pham, P.A. Calabresi, D.S. Reich, and J.L. Prince, "SMORE: A self-supervised anti-aliasing and super-resolution algorithm for MRI using deep learning", IEEE Trans. Med. Imaging, vol. 40, no. 3, pp. 805-817, 2021.
[http://dx.doi.org/10.1109/TMI.2020.3037187] [PMID: 33170776]
[http://dx.doi.org/10.1109/TMI.2020.3037187] [PMID: 33170776]
[7]
University of Electronic Science and Technology of China, "Image super-resolution reconstruction method based on high-low frequency information decomposition", C.N. 110310227B, 2017.
[8]
South China University of Technology SCUT, "Single image super-resolution reconstruction method based on deep convolutional neural network.", C.N. 106910161B, 2020.
[9]
Xidian University, "Image super-resolution reconstruction method based on sparse representation and deep learning", CN109741256B, 2015.
[10]
H. Hou, and H. Andrews, "Cubic splines for image interpolation and digital filtering", IEEE Trans. Acoust. Speech Signal Process., vol. 26, no. 6, pp. 508-517, 1978.
[http://dx.doi.org/10.1109/TASSP.1978.1163154]
[http://dx.doi.org/10.1109/TASSP.1978.1163154]
[11]
D. Cheng, and K.I. Kou, "FFT multichannel interpolation and application to image super-resolution", Signal Processing, vol. 162, pp. 21-34, 2019.
[http://dx.doi.org/10.1016/j.sigpro.2019.03.025]
[http://dx.doi.org/10.1016/j.sigpro.2019.03.025]
[12]
L. Wang, S. Xiang, G. Meng, H. Wu, and C. Pan, "Edge-directed single-image super-resolution via adaptive gradient magnitude self-interpolation", IEEE Trans. Circ. Syst. Video Tech., vol. 23, no. 8, pp. 1289-1299, 2013.
[http://dx.doi.org/10.1109/TCSVT.2013.2240915]
[http://dx.doi.org/10.1109/TCSVT.2013.2240915]
[13]
G. El, "Mireille, J. F. Aujol, Y. Berthoumieu, and C. A. Deledalle, “Texture reconstruction guided by a high-resolution patch”", IEEE Trans. Image Process., vol. 26, pp. 549-560, 2016.
[14]
J. Yang, J. Wright, T. Huang, and Y. Ma, "Image super-resolution as sparse representation of raw image patches", In 2008 IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, AK, USA, 2008, pp. 1-8
[15]
L. Zhang, M. Yang, and X. Feng, "Sparse representation or collaborative representation: Which helps face recognition?", In 2011 International Conference on Computer Vision, Barcelona, Spain, 2011, pp. 471-478
[http://dx.doi.org/10.1109/ICCV.2011.6126277]
[http://dx.doi.org/10.1109/ICCV.2011.6126277]
[16]
C. Dong, C.C. Loy, K. He, and X. Tang, "Image super-resolution using deep convolutional networks", IEEE Trans. Pattern Anal. Mach. Intell., vol. 38, no. 2, pp. 295-307, 2016.
[http://dx.doi.org/10.1109/TPAMI.2015.2439281] [PMID: 26761735]
[http://dx.doi.org/10.1109/TPAMI.2015.2439281] [PMID: 26761735]
[17]
L. Wang, T.K. Kim, and K.J. Yoon, "Eventsr: From asynchronous events to image reconstruction, restoration, and super-resolution via end-to-end adversarial learning", In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 2020, pp. 8315-8325
[http://dx.doi.org/10.1109/CVPR42600.2020.00834]
[http://dx.doi.org/10.1109/CVPR42600.2020.00834]
[18]
J. Yang, Z. Wang, Z. Lin, S. Cohen, and T. Huang, "Coupled dictionary training for image super-resolution", IEEE Trans. Image Process., vol. 21, no. 8, pp. 3467-3478, 2012.
[http://dx.doi.org/10.1109/TIP.2012.2192127] [PMID: 22481818]
[http://dx.doi.org/10.1109/TIP.2012.2192127] [PMID: 22481818]
[19]
Y. Xu, Z. Wu, J. Chanussot, and Z. Wei, "Nonlocal patch tensor sparse representation for hyperspectral image super-resolution", IEEE Trans. Image Process., vol. 28, no. 6, pp. 3034-3047, 2019.
[http://dx.doi.org/10.1109/TIP.2019.2893530] [PMID: 30668472]
[http://dx.doi.org/10.1109/TIP.2019.2893530] [PMID: 30668472]
[20]
S. Mandal, A. Bhavsar, and A.K. Sao, "Noise adaptive super-resolution from single image via non-local mean and sparse representation", Signal Processing, vol. 132, pp. 134-149, 2017.
[http://dx.doi.org/10.1016/j.sigpro.2016.09.017]
[http://dx.doi.org/10.1016/j.sigpro.2016.09.017]
[21]
S.T. Roweis, and L.K. Saul, "Nonlinear dimensionality reduction by locally linear embedding", Science, vol. 290, no. 5500, pp. 2323-2326, 2000.
[http://dx.doi.org/10.1126/science.290.5500.2323] [PMID: 11125150]
[http://dx.doi.org/10.1126/science.290.5500.2323] [PMID: 11125150]
[22]
H. Chang, D.Y. Yeung, and Y. Xiong, "Super-resolution through neighbor embedding", In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Washington, DC, USA, 2004, pp. I-I
[23]
M. Bevilacqua, A. Roumy, C. Guillemot, and M.L.A. Morel, "Low-complexity single-image super-resolution based on nonnegative neighbor embedding", In Proceedings of the 23rd British Machine Vision Conference, Surrey, UK, 2012.
[http://dx.doi.org/10.5244/C.26.135]
[http://dx.doi.org/10.5244/C.26.135]
[24]
R. Timofte, V. De Smet, and L. Van Gool, "Anchored neighborhood regression for fast example-based super-resolution", In Proceedings of the IEEE International Conference on Computer Vision, Sydney, NSW, Australia, 2013, pp. 1920-1927
[http://dx.doi.org/10.1109/ICCV.2013.241]
[http://dx.doi.org/10.1109/ICCV.2013.241]
[25]
R. Timofte, V. De Smet, and L. Van Gool, "A+: Adjusted anchored neighborhood regression for fast super-resolution", In Proceedings of the 12th Asian Conference on Computer Vision, Singapore, 2014, pp. 111-126
[26]
Y. Zhang, Y. Zhang, J. Zhang, and Q. Dai, "CCR: Clustering and collaborative representation for fast single image super-resolution", IEEE Trans. Multimed., vol. 18, no. 3, pp. 405-417, 2016.
[http://dx.doi.org/10.1109/TMM.2015.2512046]
[http://dx.doi.org/10.1109/TMM.2015.2512046]
[27]
A. Fujimoto, T. Ogawa, K. Yamamoto, Y. Matsui, T. Yamasaki, and K. Aizawa, "Manga109 dataset and creation of metadata", In Proceedings of the 1st International Workshop on Comics Analysis, Processing and Understanding, Cancun, Mexico, 2016, pp. 1-5
[28]
S. Izadpanahi, and H. Demirel, "Motion based video super resolution using edge directed interpolation and complex wavelet transform", Signal Processing, vol. 93, no. 7, pp. 2076-2086, 2013.
[http://dx.doi.org/10.1016/j.sigpro.2013.01.006]
[http://dx.doi.org/10.1016/j.sigpro.2013.01.006]
[29]
S. Mallat, and G. Yu, "Super-resolution with sparse mixing estimators", IEEE Trans. Image Process., vol. 19, no. 11, pp. 2889-2900, 2010.
[http://dx.doi.org/10.1109/TIP.2010.2049927] [PMID: 20457549]
[http://dx.doi.org/10.1109/TIP.2010.2049927] [PMID: 20457549]
[30]
F. Zhou, W. Yang, and Q. Liao, "Interpolation-based image super-resolution using multisurface fitting", IEEE Trans. Image Process., vol. 21, no. 7, pp. 3312-3318, 2012.
[http://dx.doi.org/10.1109/TIP.2012.2189576] [PMID: 22389144]
[http://dx.doi.org/10.1109/TIP.2012.2189576] [PMID: 22389144]
[31]
Y. Zhang, Q. Fan, F. Bao, Y. Liu, and C. Zhang, "Single-image super-resolution based on rational fractal interpolation", IEEE Trans. Image Process., vol. 27, no. 8, pp. 3782-3797, 2018.
[http://dx.doi.org/10.1109/TIP.2018.2826139] [PMID: 29698209]
[http://dx.doi.org/10.1109/TIP.2018.2826139] [PMID: 29698209]
[32]
Z. Ren, C. He, and Q. Zhang, "Fractional order total variation regularization for image super-resolution", Signal Processing, vol. 93, no. 9, pp. 2408-2421, 2013.
[http://dx.doi.org/10.1016/j.sigpro.2013.02.015]
[http://dx.doi.org/10.1016/j.sigpro.2013.02.015]
[33]
W. Dong, L. Zhang, G. Shi, and X. Wu, "Image deblurring and super-resolution by adaptive sparse domain selection and adaptive regularization", IEEE Trans. Image Process., vol. 20, no. 7, pp. 1838-1857, 2011.
[http://dx.doi.org/10.1109/TIP.2011.2108306] [PMID: 21278019]
[http://dx.doi.org/10.1109/TIP.2011.2108306] [PMID: 21278019]
[34]
J. Yang, J. Wright, T.S. Huang, and Y. Ma, "Image super-resolution via sparse representation", IEEE Trans. Image Process., vol. 19, no. 11, pp. 2861-2873, 2010.
[http://dx.doi.org/10.1109/TIP.2010.2050625] [PMID: 20483687]
[http://dx.doi.org/10.1109/TIP.2010.2050625] [PMID: 20483687]
[35]
J. Liu, W. Yang, X. Zhang, and Z. Guo, "Retrieval compensated group structured sparsity for image super-resolution", IEEE Trans. Multimed., vol. 19, no. 2, pp. 302-316, 2017.
[http://dx.doi.org/10.1109/TMM.2016.2614427]
[http://dx.doi.org/10.1109/TMM.2016.2614427]
[36]
M. Ebrahimi, and S. Bohun, "Single image super-resolution via non-local normalized graph Laplacian regularization: A self-similarity tribute", Commun. Nonlinear Sci. Numer. Simul., vol. 93, p. 105508, 2021.
[http://dx.doi.org/10.1016/j.cnsns.2020.105508]
[http://dx.doi.org/10.1016/j.cnsns.2020.105508]
[37]
A. Rahimi, P. Moallem, K. Shahtalebi, and M. Momeni, "Using Kalman filter in the frequency domain for multi-frame scalable super resolution", Signal Processing, vol. 155, pp. 108-129, 2019.
[http://dx.doi.org/10.1016/j.sigpro.2018.09.012]
[http://dx.doi.org/10.1016/j.sigpro.2018.09.012]
[38]
F. Hao, T. Zhang, L. Zhao, and Y. Tang, "Efficient residual attention network for single image super-resolution", Appl. Intell., vol. 52, no. 1, pp. 652-661, 2022.
[http://dx.doi.org/10.1007/s10489-021-02489-x]
[http://dx.doi.org/10.1007/s10489-021-02489-x]
[39]
J. Liu, W. Zhang, Y. Tang, J. Tang, and G. Wu, "Residual feature aggregation network for image super-resolution", In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 2020, pp. 2359-2368
[http://dx.doi.org/10.1109/CVPR42600.2020.00243]
[http://dx.doi.org/10.1109/CVPR42600.2020.00243]
[40]
S. Zhou, J. Zhang, W. Zuo, and C.C. Loy, "Cross-scale internal graph neural network for image super-resolution", In: roceedings of Advances in Neural Information Processing Systems, 2020, pp. 3499-3509.
[41]
W. Han, S. Chang, D. Liu, M. Yu, M. Witbrock, and T.S. Huang, "Image super-resolution via dual-state recurrent networks", In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018, pp. 1654-1663
[42]
M. Haris, G. Shakhnarovich, and N. Ukita, "Recurrent back-projection network for video super-resolution", In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, California, USA, 2019, pp. 3897-3906
[43]
J. Wan, H. Yin, A.X. Chong, and Z.H. Liu, "Progressive residual networks for image super-resolution", Appl. Intell., vol. 50, no. 5, pp. 1620-1632, 2020.
[http://dx.doi.org/10.1007/s10489-019-01548-8]
[http://dx.doi.org/10.1007/s10489-019-01548-8]
[44]
Z. Zhang, Z. Wang, Z. Lin, and H. Qi, "Image super-resolution by neural texture transfer", In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, California, USA, 2019, pp. 7982-7991
[45]
Z. Liu, J. Huang, C. Zhu, X. Peng, and X. Du, "Residual attention network using multi-channel dense connections for image super-resolution", Appl. Intell., vol. 51, no. 1, pp. 85-99, 2021.
[http://dx.doi.org/10.1007/s10489-020-01723-2]
[http://dx.doi.org/10.1007/s10489-020-01723-2]
[46]
M. Haris, G. Shakhnarovich, and N. Ukita, "Deep back-projectinetworks for single image super-resolution", IEEE Trans. Pattern Anal. Mach. Intell., vol. 43, no. 12, pp. 4323-4337, 2021.
[http://dx.doi.org/10.1109/TPAMI.2020.3002836] [PMID: 32750788]
[http://dx.doi.org/10.1109/TPAMI.2020.3002836] [PMID: 32750788]
[47]
Y. Chen, L. Liu, V. Phonevilay, K. Gu, R. Xia, J. Xie, Q. Zhang, and K. Yang, "Image super-resolution reconstruction based on feature map attention mechanism", Appl. Intell., vol. 51, no. 7, pp. 4367-4380, 2021.
[http://dx.doi.org/10.1007/s10489-020-02116-1]
[http://dx.doi.org/10.1007/s10489-020-02116-1]
[48]
X. Song, Y. Dai, D. Zhou, L. Liu, W. Li, H. Li, and R. Yang, "Channel attention based iterative residual learning for depth map super-resolution", In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 2020, pp. 5631-5640
[http://dx.doi.org/10.1109/CVPR42600.2020.00567]
[http://dx.doi.org/10.1109/CVPR42600.2020.00567]
[49]
Q.M. Liu, R.S. Jia, Y.B. Liu, H.B. Sun, J.Z. Yu, and H.M. Sun, "Infrared image super-resolution reconstruction by using generative adversarial network with an attention mechanism", Appl. Intell., vol. 51, no. 4, pp. 2018-2030, 2021.
[http://dx.doi.org/10.1007/s10489-020-01987-8]
[http://dx.doi.org/10.1007/s10489-020-01987-8]
[50]
M. Ranzato, Y. Boureau, and Y. Cun, "Sparse feature learning for deep belief networks", In NIPS’07: Proceedings of the 20th International Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada, 2007, pp. 1185-1192
[51]
R.H. Bartels, and G.W. Stewart, "Algorithm 432 [C2]: Solution of the matrix equation AX + XB = C [F4]", Commun. ACM, vol. 15, no. 9, pp. 820-826, 1972.
[http://dx.doi.org/10.1145/361573.361582]
[http://dx.doi.org/10.1145/361573.361582]
[53]
Z. Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli, "Image quality assessment: From error visibility to structural similarity", IEEE Trans. Image Process., vol. 13, no. 4, pp. 600-612, 2004.
[http://dx.doi.org/10.1109/TIP.2003.819861] [PMID: 15376593]
[http://dx.doi.org/10.1109/TIP.2003.819861] [PMID: 15376593]
[54]
R. Zeyde, M. Elad, and M. Protter, "On single image scale-up using sparse-representations", In Proceedings of the 7th International Conference on Curves and Surfaces, Avignon, France, 2012, pp. 711-730
[55]
J.B. Huang, A. Singh, and N. Ahuja, "Single image super-resolution from transformed self-exemplars", In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 2015, pp. 5197-5206
[56]
P. Arbeláez, M. Maire, C. Fowlkes, and J. Malik, "Contour detection and hierarchical image segmentation", IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 5, pp. 898-916, 2011.
[http://dx.doi.org/10.1109/TPAMI.2010.161] [PMID: 20733228]
[http://dx.doi.org/10.1109/TPAMI.2010.161] [PMID: 20733228]
Related Books
Advanced Characterization Technologies for Secondary Batteries
Design and Construction of Buildings and Foundations with Illustrative Examples
Robotics and Automation in Industry 4.0
Biomedical Applications of Perovskites: The Era of Bio-Piezoelectric Systems
Smart Kinematics For Modern Engineering Students
Computational Modeling and Simulation in Biomedical Research
Introduction to Advanced Soft Robotics
Intelligent Technologies for Research and Engineering
The Production of Biodiesel and Related Fuel Additives
Intelligent Technologies for Research and Engineering
