Abstract
Background: In VTBDs (Vision Technology-Based Devices), vision technology is utilized to acquire abundant information about the external environment and process such information to achieve certain functions. They are used in various fields to solve practical problems. Various patents have been discussed in this article, hoping to provide ideas for solving practical problems in the future.
Objective: The study aimed to provide an overview of the existing VTBDs and introduce their classifications, characteristics, as well as the stage and trend of development.
Methods: This paper reviews various patents, especially Chinese patents related to VTBDs. The structural characteristics, differentiation, and engineering applications of VTBDs are also introduced.
Results: The existing VTBDs are analyzed and compared, and their typical characteristics are summarized. The main applications, as well as the pros and cons, in the current development stage, are summarized and analyzed, as well. In addition, the development trend of VTBDsrelated patents is also discussed.
Conclusion: VTBDs can be categorized into DsBMV (Devices Based on Monocular Visual), DsBBV (Devices Based on Binocular Visual), and DsBMCV (Devices Based on Multi-Camera Visual). All of these categories exhibit their own relative advantages and disadvantages. Therefore, it is of much importance to analyze the specific problems, followed by selecting appropriate machine vision technologies and reasonable mechanical structures to design VTBDs accordingly.
Keywords: China Patent, computer vision, machine vision, robot vision, three-dimensional reconstruction, vision technology.
Graphical Abstract
[1]
S. Janusz, and H. Pawel, “Workspace supervising system for material handling devices with machine vision assistance”, J. Konbin., vol. 11-12, no. 1, pp. 7-16, 2009.
[2]
Y. Mezouar, and F. Chaumette, “Path planning for robot imaged-based control”, IEEE Trans. Robot. Autom., vol. 18, pp. 534-549, 2002.
[http://dx.doi.org/10.1109/TRA.2002.802218]
[http://dx.doi.org/10.1109/TRA.2002.802218]
[3]
H. Golnabi, and A. Asadpour, “Design and application of industrial machine vision systems”, Robot. Comp. Integr., vol. 23, pp. 630-637, 2007.
[http://dx.doi.org/10.1016/j.rcim.2007.02.005]
[http://dx.doi.org/10.1016/j.rcim.2007.02.005]
[4]
J.H. Gu, J.D. Zhu, J.H. Yin, J. Feng, C. Luo, and W. Han, “Roller pin surface defect detecting device based on machine vision”, China Patent 202,903,690,
[5]
F. Chen, X.Q. Wu, Y.B. Ying, Y.T. Liao, and Y.X. Fan, “Machine vision-based real-time detection and grading method and grading device for pork appearance quality”, China Patent 101,561,402,
[7]
M. Baigvand, A. Banakar, S. Minaei, J. Khodaei, and N. Behroozi-Khazaei, “Machine vision system for grading of dried figs”, Comput. Electron. Agric., vol. 119, pp. 158-165, 2015.
[http://dx.doi.org/10.1016/j.compag.2015.10.019]
[http://dx.doi.org/10.1016/j.compag.2015.10.019]
[8]
Y.J. Qiao, S.Z. Tan, and Y. Cao, “A multi-degree-of-freedom binocular stereo visual mechanism”, "China Patent 206,164,745", , 2017.
[9]
H.I. Celik, L.C. Duelger, and M. Topalbekiroglu, “Development of a machine vision system: Real time fabric defect detection and classification with neural networks”, J. Textil. Inst., vol. 105, pp. 575-585, 2014.
[http://dx.doi.org/10.1080/00405000.2013.827393]
[http://dx.doi.org/10.1080/00405000.2013.827393]
[10]
D. Berco, “An optoneuronic device with realistic retinal expressions for bioinspired machine vision”, Adv. Intell. Syst, vol. 2, 2020.
[11]
A.H. Li, and Y. Zhang, “Variable visual axis monocular stereo vision measurement method”, China Patent 108,253,939,
[12]
W.F. Xie, Z. Li, X.W. Tu, and C. Perron, “Switching control of image-based visual servoing with laser pointer in robotic manufacturing systems”, IEEE Trans. Ind. Electron., vol. 6, pp. 520-529, 2009.
[13]
Y.J. Xiong, Z.C. Liu, and Y.S. Zhao, “Method, device and equipment for depth estimation based on monocular camera”, China Patent 109,816,709,
[14]
C.C. Cheah, C. Liu, and J.J.E. Slotine, “Adaptive Jacobian vision based control for robots with uncertain depth information”, Automatica, vol. 46, pp. 1228-1233, 2010.
[http://dx.doi.org/10.1016/j.automatica.2010.04.009]
[http://dx.doi.org/10.1016/j.automatica.2010.04.009]
[15]
K. Qi, “Method and device for gesture recognition based on monocular visual”, China Patent 107,688,391,
[16]
Y.P. Tang, K. Liu, S.J. Xia, J.K. Zhou, L.L. Lin, H.T. Xu, H.C. Yan, L.L. Huang, B.Q. Ma, L. Yu, Y.J. Feng, X.F. Chen, and Q.L. Zhu, “Pearl online detecting and grading device based on monocular multi-view machine vision”, China Patent 102,928,431,
[17]
X.S. Zhang, S.L. Zhang, Y.P. Cai, X.Y. Wang, Y.L. Chen, Z.G. Huang, Y.Z. Yang, and F. Zhao, “A movable binocular camera mounting bracket”, China Patent 209,402,656,
[19]
J. Stavnitzky, and D. Capson, “Multiple camera model-based 3D visual servo”, IEEE Trans. Robot. Autom., vol. 16, pp. 732-739, 2001.
[http://dx.doi.org/10.1109/70.897784]
[http://dx.doi.org/10.1109/70.897784]
[20]
M. Cao, Y.M. Gao, P.J. Deng, C.Z. Wang, R.P. Hu, J.J. Liu, and L.B. Liao, “Multi-camera visual inspection structure for underwater fuel assembly”, "China Patent 209,342,058", , 2019.
[21]
X.F. Liu, and Y. Zhang, “Multi-camera stereo visual human face 3D reconstruction device and method”, China Patent 109,461,206,
[22]
R. Xiong, B. Fu, and Y. Yang, “Method and device for robot multi-camera visual inertial real-time positioning”, China Patent 109,506,642,
[23]
Z.R. Wang, H.S. Liu, T. Qin, and Z.J. Wu, “A device for on-site visual quick calibration”, China Patent 205,651,374,
[24]
J.G. Li, and Z.J. Wu, “A monocular camera calibration device used in industrial field”, China Patent 209,026,418,
[25]
T.L. Zhang, D.S. Zhong, and J.X. Gong, “Monocular visual measuring device”, China Patent 208,419,890,
[26]
E.Q. Wu, S.Q. Tian, M. Jiang, B. Zhao, Q. Gao, M.H. Li, S.P. Xu, J.B. Xu, Y.F. Shi, and Z.M. Gu, “Monocular visual measuring device and its three-dimensional modeling system”, China Patent 106,895,795,
[30]
Y.J. Zhu, J. Luo, H. Liu, S.X. Song, G.X. Qin, and Y.B. Qin, “Metal workpiece defect detection device based on monocular stereo vision”, China Patent 211,426,296,
[31]
W. Li, T. Yuan, Q.C. Feng, C.L. Zhang, and J.X. Zhang, “Binocular vision information acquisition device of cucumber picking robot”, China Patent 101,828,469,
[32]
X.D. Ye, J.H. Zhao, M.Z. Luo, T. Mei, M. Bao, and X.B. Song, “Service robot wide-angle binocular visual recognition and positioning device”, China Patent 103,048,995,
[33]
H.X. Zhang, H.R. Qiu, and X. Zhang, “Support structure of binocular visual camera and method for adjusting its pose”, China Patent 108,953,905,
[34]
H.X. Zhang, Y. Cheng, and P. Hu, “Automatic adjustment and support device for binocular visual camera”, China Patent 109,882,689,
[35]
H.X. Zhang, P. Hu Ping, and Y. Cheng, “Support and position adjustment device for stereo vision shooting”, China Patent 209,196,438,
[36]
H.X. Zhang, Y. Cheng, P. Hu, and H.R. Qiu, “Camera supporting structure based on binocular stereo visual technology”, China Patent 110,566,772,
[37]
Z. Zhang, W.A. Zhu, B.F. Lang, W.L. Ouyang, S.H. Zhang, and X.X. Huang, “A simple binocular vision teaching platform”, China Patent 109,559,627,
[38]
J.J. Zhang, Z.L. Fang, Z.Y. Zhao, P.C. Niu, and R.Z. Gao, “A binocular visual precision adjustment bracket”, China Patent 109,668,021,
[39]
J.L. Shi, J.H. Qin, and Y.L. Shen, “Multi-camera device for three-dimensional scanning of human body”, China Patent 107,438,147,
[40]
R. Shu, H.Y. Bu, G. Gu, Q. Xu, C. Yuan, W. Wang, G.J. Dong, Y. Guo, and H.B. Xu, “An instant three-dimensional camera system based on camera array”, China Patent 203,708,345,
[41]
J.L. Chen, Y.B. Chen, and C. Tang, “Multi-camera full-face image acquisition device for human face skin state evaluation”, China Patent 209,462,473,
[42]
H. Yang, F. Liang, and B.J. Luo, “A wearable multi-camera visual acquisition image device”, China Patent 209,692,893,
[43]
M. Li, X.T. Li, S.C. Li, Z.P. Liu, and M.M.Q. You, “A simple multi-camera visiual calibration device”, China Patent 109,064,518,
[44]
D.J. Liu, J. Cao, and Z.W. Mo, “Multi-camera measuring device and multi-camera measuring method”, China Patent 110,108,235,
[46]
D.L. Wang, Y.Q. Dong, Y. Pan, and R.J. Ma, “Machine vision-based monitoring methodology for the fatigue cracks in U-rib-to-deck weld seams”, IEEE Access, vol. 99, pp. 1-1, 2020.
[http://dx.doi.org/10.1109/ACCESS.2020.2995276]
[http://dx.doi.org/10.1109/ACCESS.2020.2995276]
[47]
A. Kumar, and A. Ghosh, “Infrared imaging based machine vision system to determine transient shape of isotherms in submerged arc welding”, Infrared Phys. Technol., vol. 109, p. 103410, 2020.
[48]
S.S. Mehta, C. Ton, Z. Kan, and J.W. Curtis, “Vision-based navigation and guidance of a sensorless missile”, J. Franklin Inst., vol. 352, pp. 5569-5598, 2015.
[http://dx.doi.org/10.1016/j.jfranklin.2015.09.010]
[http://dx.doi.org/10.1016/j.jfranklin.2015.09.010]
[49]
B. Zhang, M.M. Liu, Y.Z. Tian, G. Wu, X.H. Yang, S.Y. Shi, and J.N. Li, “Defect inspection system of nuclear fuel pellet end faces based on machine vision”, J. Nucl. Sci. Technol., vol. 57, pp. 1-7, 2020.
[http://dx.doi.org/10.1080/00223131.2019.1708827]
[http://dx.doi.org/10.1080/00223131.2019.1708827]
[50]
S.S. Mehta, and T.F. Burks, “Vision-based control of robotic manipulator for citrus harvesting”, Comput. Electron. Agric., vol. 102, pp. 146-158, 2014.
[http://dx.doi.org/10.1016/j.compag.2014.01.003]
[http://dx.doi.org/10.1016/j.compag.2014.01.003]
[51]
L. Zhang, J.W. Chen, and K. Wang, “A visual servo laser positioning system and method for picking robot”, China Patent 110,065,074,
[52]
H.Y. Yu, K.B. Li, S.J. Zhou, J.J. Ye, and Y.H. Jiang, “Machine vision detection device and method for surface defects of automobile engine turbine shell”, China Patent 107,741,427,
[53]
M.M. Dong, Z.G. Lv, and Y. Zhang, “Research on measurement device of vehicle profile size based on machine vision”, Zidonghua Yu Yibiao, vol. 34, pp. 71-74, 2019.
[54]
A. Kumari, R.K. Behera, K.S. Sahoo, A. Nayyar, and S.P. Sahoo, “Supervised link prediction using structured-based feature extraction in social network”, Concurr. Comput., 2020.
[http://dx.doi.org/10.1002/cpe.5839]
[http://dx.doi.org/10.1002/cpe.5839]
[55]
R.K. Behera, D. Naik, S.K. Rath, and R. Dharavath, “Genetic algorithm-based community detection in large-scale social networks”, Neural Comput. Appl., vol. 32, pp. 9649-9665, 2020.
[http://dx.doi.org/10.1007/s00521-019-04487-0]
[http://dx.doi.org/10.1007/s00521-019-04487-0]
[56]
R.K. Behera, S. Das, S.K. Rath, S. Misra, and R. Dharavath, “Comparative study of real time machine learning models for stock prediction through streaming data”, J. Univers. Comput. Sci., vol. 26, pp. 1128-1147, 2020.
[59]
K. Joshi, and B. Patil, “Prediction of surface roughness by machine vision using principal components based regression analysis”, Procedia Comput. Sci., vol. 167, pp. 382-391, 2020.
[http://dx.doi.org/10.1016/j.procs.2020.03.242]
[http://dx.doi.org/10.1016/j.procs.2020.03.242]
[60]
V.H. Nguyen, V.H. Pham, X. Cui, M. Ma, and H. Kim, “Design and evaluation of features and classifiers for OLED panel defect recognition in machine vision”, J. Inf. Telecommun., vol. 1, pp. 334-350, 2017.
[http://dx.doi.org/10.1080/24751839.2017.1355717]
[http://dx.doi.org/10.1080/24751839.2017.1355717]
[61]
F.J. Pontes, J.R. Ferreira, M.B. Silva, A.P. Paiva, and P.P. Balestrassi, “Artificial neural networks for machining processes surface roughness modeling”, Int. J. Adv. Manuf. Technol., vol. 49, pp. 879-902, 2010.
[http://dx.doi.org/10.1007/s00170-009-2456-2]
[http://dx.doi.org/10.1007/s00170-009-2456-2]
[62]
R. Wang, A. Wu, X. Chen, and J. Wang, "A point and distance constraint based 6R robot calibration method through machine vision", Robot. Comput. Integr. Manuf., vol. 65, 2020.
[63]
D.P. Penumuru, S. Muthuswamy, and P. Karumbu, “Identification and classification of materials using machine vision and machine learning in the context of industry 4.0”, J. Intell. Manuf., vol. 31, pp. 1229-1241, 2020.
[http://dx.doi.org/10.1007/s10845-019-01508-6]
[http://dx.doi.org/10.1007/s10845-019-01508-6]
[64]
N. Herakovic, “Robot vision in industrial assembly and quality control processes”, In Robot Vision., InTech, 2010.
[http://dx.doi.org/10.5772/9285]
[http://dx.doi.org/10.5772/9285]
[65]
N.V. Ngo, Q.C. Hsu, W.L. Hsiao, and C.J. Yang, “Development of a simple three-dimensional machine-vision measurement system for in-process mechanical parts”, Adv. Mech. Eng., vol. 9, pp. 1-11, 2017.
[http://dx.doi.org/10.1177/1687814017717183]
[http://dx.doi.org/10.1177/1687814017717183]
[66]
C. Wang, N. Wang, M. Ho, X. Chen, and G. Song, “Design of a new vision-based method for the bolts looseness detection in flange connections”, IEEE Trans. Ind. Electron., vol. 99, pp. 1-1, 2019.
[67]
S.P. Liao, Y. Zhang, S.B. Zhang, J. Cui, L. He, Y. Yu, and A.G. Jiang, “Machine vision detection device and method based on conveying chain”, China Patent 102,680,471,
[68]
S.B. Zhang, S.P. Liao, B. Chen, Y. Zhang, and M. Cheng, “Machine vision detection device and method based on line scan camera”, China Patent 102,636,436,
[70]
R. Guo, Z.W. Su, B.Y. Deng, X.Y. He, and X. Nong, “Machine vision device for detecting surface defects of cylindrical workpieces”, China Patent 206,208,780,
[71]
Z. Wang, L. Yang, L. Gao, X.P. Bai, F. Xiong, C.Y. Yin, Z.X. Liu, and J.B. Liu, “Method and device for online detection of corn kernel crushing rate based on machine vision”, China Patent 108,120,715,
[72]
J. Blasco, N. Aleixos, J.M. Roger, G. Rabatel, and E. Molto, “Robotic weed control using machine vision”, Biosyst. Eng., vol. 83, pp. 149-157, 2002.
[http://dx.doi.org/10.1006/bioe.2002.0109]
[http://dx.doi.org/10.1006/bioe.2002.0109]
[73]
N. Li, X.L. Ping, D.H. Wang, and P. Jiang, “A monocular camera robot”, China Patent 109,571,437,
[74]
H.X. Dong, “An intelligent robot based on visual servo control system”, China Patent 109,648,553,
[75]
J. Gao, A.A. Proctor, Y. Shi, and C. Bradley, “Hierarchical model predictive image-based visual servoing of underwater vehicles with adaptive neural network dynamic control”, IEEE Trans. Cybern., vol. 46, no. 10, pp. 2323-2334, 2016.
[http://dx.doi.org/10.1109/TCYB.2015.2475376] [PMID: 26441459]
[http://dx.doi.org/10.1109/TCYB.2015.2475376] [PMID: 26441459]
[76]
J.J. Wang, B. Zhang, C. Martinez, and C.W. Morato, “Three-dimensional visual servoing for robot positioning”, U.S. Patent 20,170,140,539,
[77]
A. Djajadi, F. Laoda, R. Rusyadi, T. Prajogo, and M. Sinaga, “A model vision of sorting system application using robotic manipulator”, Telkomnika, vol. 8, pp. 137-148, 2010.
[http://dx.doi.org/10.12928/telkomnika.v8i2.615]
[http://dx.doi.org/10.12928/telkomnika.v8i2.615]
[78]
Z.H. Yan, C.L. Liu, Z.Y. Lu, and L.L. Bai, “Machine vision device for welding zone with self-adaptive liquid crystal filter screen”, China Patent 103,934,600,
[79]
P. Wan, H.B. Pan, C.J. Long, and H. Chen, “Design of the on-line identification device of freshwater fish species based on machine vision technology”, Food Machin., vol. 28, pp. 164-167, 2012.
[80]
P. Xu, C.H. Liu, B. Xu, and Y.J. Chang, “A double-sided imaging machine vision device”, China Patent 103,616,043,
[81]
Z. Dai, H.S. Song, H.X. Liang, F.F. Wu, X. Wang, J.M. Jia, and Y. Fang, “Traffic parameter estimation and control system based on machine vision”, J. Ambient Intell. Humaniz. Comput., 2020.
[http://dx.doi.org/10.1007/s12652-020-02052-5]
[http://dx.doi.org/10.1007/s12652-020-02052-5]
Related Journals
International Journal of Sensors, Wireless Communications and Control
Related Books
Challenges and Opportunities for Deep Learning Applications in Industry 4.0
Recent Advances in IoT and Blockchain Technology
Augmented Intelligence: Deep Learning, Machine Learning, Cognitive Computing, Educational Data Mining
Human-Computer Interaction and Beyond: Advances Towards Smart and Interconnected Environments Part II
Machine Learning and Its Application: A Quick Guide for Beginners
Computational Intelligence For Data Analysis
Applied Machine Learning and Multi-Criteria Decision-Making in Healthcare
Advanced Computing Techniques: Implementation, Informatics and Emerging Technologies
How to Design Optimization Algorithms by Applying Natural Behavioral Patterns
Human-Computer Interaction and Beyond: Advances Towards Smart and Interconnected Environments (Part I)
Article Metrics
6