This paper reviews robot visual servoing technology, introducing the concept, development history, and classification of robot visual servoing systems, with a focus on position-based and image-based visual servoing systems. It summarizes the cutting-edge issues in robot vision and points out current research challenges and future directions.
Currently, industrial robots are playing an increasingly important role in manufacturing worldwide. To enable robots to perform more complex tasks, they not only need better control systems but also greater sensitivity to environmental changes. Among these, robot vision, with its large information capacity and completeness, has become the most crucial sensory function for robots.
Robot visual servoing systems are an organic combination of machine vision and robot control. They are complex, nonlinear, and strongly coupled systems, encompassing research areas such as image processing, robot kinematics and dynamics, and control theory. With the improvement in the performance-price ratio of camera equipment and the speed of computer information processing, as well as the increasing sophistication of related theories, visual servoing has reached a point where it is ready for practical application, and related technical issues have become a current research hotspot.
Robot Vision Servo System
Definition of visual servoing:
Humans acquire most of their external information through their eyes. For millennia, humanity has dreamed of creating intelligent machines that possess the functions of human eyes, enabling them to perceive and understand the external world. Many components of the human brain are involved in processing visual information, allowing for the easy handling of numerous visual problems. However, our understanding of visual cognition as a process remains limited, hindering the realization of the dream of intelligent machines. With the development of camera technology and the emergence of computer technology, intelligent machines with visual capabilities have begun to be created, gradually forming the discipline and industry of machine vision. The definition of machine vision, given by the Machine Vision Division of the Society of Manufacturing Engineers (SME) and the Automation Vision Division of the RIA Robotics Industries Association (RIA Robotics Industries Association), is as follows:
"Machine vision is a device that automatically receives and processes images of a real object through optical devices and non-contact sensors to obtain the required information or to control the movement of a robot."
Machine vision, as a biomimetic system similar to the human eye, broadly refers to anything that acquires information about real objects through optical devices and processes and executes such information. This includes both visible and non-visible vision, and even the acquisition and processing of information about the interior of objects that cannot be directly observed by human vision.
Development History of Robot Vision
In the 1960s , with the development of robotics and computer technology, researchers began to study robots with vision capabilities. However, in these studies, the robot's vision and its movements were strictly speaking open-loop. The robot's vision system obtained the target pose through image processing, and then calculated the robot's motion pose based on the target pose. In this process, the vision system "provided" information only once and then did not participate in the process. In 1973 , some people applied the vision system to robot control systems, and this process was called visual feedback at that time . It wasn't until 1979 that Hill and Park proposed the concept of " visual servo ." Clearly, visual feedback only means extracting feedback signals from visual information, while visual servoing includes the entire process from visual signal processing to robot control. Therefore, visual servoing reflects the relevant research content of robot vision and control more comprehensively than visual feedback.
Since the 1980s , with the development of computer technology and camera equipment, the technical problems of robot visual servoing systems have attracted the attention of many researchers. In the past few years, robot visual servoing has made great progress both theoretically and in application. Visual servoing technology is frequently listed as a topic at many academic conferences. Visual servoing has gradually developed into an independent technology spanning robotics, automatic control, and image processing.
Robot vision servo system classification:
Currently, robot vision servo control systems can be classified in the following ways:
●Based on the number of cameras, visual servo systems can be divided into monocular visual servo systems, binocular visual servo systems, and multi-view visual servo systems.
Monocular vision systems can only obtain two-dimensional planar images and cannot directly obtain the depth information of the target ; multi-view vision servoing systems can acquire images of the target from multiple directions, obtaining rich information, but the amount of image information processing is large, and the more cameras there are, the more difficult it is to ensure the stability of the system. Current vision servoing systems mainly use binocular vision.
●Based on the placement of the camera, systems can be divided into eye -in- hand systems and fixed camera systems (eye to hand or stand alone).
In theory, hand-eye systems can achieve precise control, but they are sensitive to system calibration errors and robot motion errors ; fixed camera systems are not sensitive to robot kinematic errors, but under the same conditions, the accuracy of the target pose information obtained is not as good as that of hand-eye systems, so the control accuracy is relatively low.
●Based on the robot's spatial position or image features, visual servoing systems are divided into position-based visual servoing systems and image-based visual servoing systems.
In a position-based visual servoing system ( as shown in Figure 1 ) , the target's pose relative to the camera and robot is calculated after image processing. This requires calibration of the models of the camera, target, and robot. The calibration accuracy affects the control accuracy, which is a challenge of this method. During control, the pose that needs to change is converted into the angle of robot joint rotation, and the joint controller controls the robot joint rotation.
Image-based direct visual servo system
In image-based visual servoing systems ( as shown in the figure ) , control error information arises from the difference between the features of the target image and the features of the desired image. A key challenge for this control method is establishing an image Jacobian matrix that reflects the relationship between changes in image differences and changes in the robot's pose and velocity . Another challenge is that since the image is two-dimensional, calculating the image Jacobian matrix requires estimating the target depth ( three-dimensional information ) , and depth estimation has always been a difficult area in computer vision.
There are several methods for calculating the Jacobian matrix, including formula derivation, calibration, estimation, and learning. The former can be derived or calibrated from a model, while the latter can be estimated online. The learning method mainly utilizes neural networks.
●Based on the robot employing a closed-loop joint controller, visual servoing systems are divided into dynamic observation - movement systems and direct visual servoing systems.
The former uses a robot joint feedback inner loop to stabilize the robotic arm, and the image processing module calculates the speed or position increment that the camera should have and feeds it back to the robot joint controller ; the latter directly calculates the control quantities of the robot arm's joint movements by the image processing module.
The main problems faced by visual servoing
The study of visual servoing has a history of nearly 20 years. However, due to the large number of disciplines involved, its development depends on the development of these disciplines. Currently, there are still many problems in the research of visual servoing that have not been well solved.
●Image processing methods represent the greatest challenge for image servoing, both in terms of theoretical aspects and practical computational speed .
●After image processing, establishing a model between image features and robot joint motion is another challenge in image servoing ;
● Many current control methods cannot guarantee that the system is stable over a wide range of conditions during operation, so research on relevant control methods is also necessary.
The Development Prospect of Visual Servo
The main research directions for future visual servoing are as follows:
● The key issue for visual servoing systems is to acquire image features quickly and robustly in real-world environments.
Due to the large amount of information in image processing and the development of programmable device technology, recent approaches to hardware-based general-purpose algorithms to accelerate information processing may make progress in this area of research.
●Establish relevant theories and software suitable for robot vision systems
Many current image processing methods for robot vision servoing systems are not designed for robot vision systems. If there were such a dedicated software platform, the workload could be reduced when performing vision servoing tasks, and the performance of the vision servoing system could even be improved by hardware-based visual information processing.
● Applying various artificial intelligence methods to robot vision servo systems
Although neural networks have been applied in robot visual servoing, many intelligent methods have not yet been fully utilized in robot visual servoing systems. Moreover, current research tends to rely too heavily on mathematical modeling and computation, which makes the computational load of robot visual servoing systems too large during operation. The current processing speed of computers is difficult to meet the requirements of system speed. However, humans do not accomplish related functions through a large amount of computation. This inspires us to explore whether artificial intelligence methods can be used to reduce the amount of mathematical computation in order to meet the requirements of system speed.
● Applying active vision technology to robot vision servo systems
Active vision is a hot topic in the field of computer vision and machine vision research. In this field, the vision system can actively perceive the environment and actively extract the required image features according to certain rules, which makes it possible to solve problems that are difficult to solve under normal circumstances.
● Integrate vision sensors with other external sensors
To enable robots to perceive their environment more comprehensively, especially to supplement the information of the robot's vision system, multiple sensors can be added to the robot's vision system. This can overcome some of the difficulties of the robot's vision system. However, the introduction of multiple sensors requires solving the problems of information fusion and information redundancy in the robot's vision system.
Conclusion
In recent years, robot vision servo technology has made great strides, and the practical applications of robot vision systems are increasing both domestically and internationally. Many technical challenges are expected to be addressed in future research. In the near future, robot vision servo systems will occupy a prominent position in robotics technology, and their application in industrial production will become increasingly widespread.
