A robot's sensing system includes visual, auditory, tactile, olfactory, and gustatory systems. These sensing systems consist of exchangers, or sensors, that are sensitive to images, light, sound, pressure, odors, and tastes.
Humans obtain 80% of the information they receive from the outside world through vision, and machines are similar. Machine vision is essentially about implanting "eyes" into machines, using the reflection of light from the environment and objects to acquire and perceive information.
A visual sensor is a device that uses photoelectric sensors to acquire images of objects. It converts these images into digital signals and processes and analyzes them. The working process of a visual sensor includes four parts: detection, analysis, depiction, and recognition. Visual detection mainly uses image signal input devices to convert visual information into electrical signals. Visual image analysis removes noise and worthless pixels from all acquired signals and rearranges valuable pixels into pixel sets according to line segments or regions. Visual depiction and recognition extract features from the object image and assign them labels.
Based on the different dimensions of image information acquisition and the types of data processed, machine vision can be divided into 2D vision and 3D vision. 2D vision uses industrial cameras to acquire planar images, primarily focusing on the planar features of an object for subsequent analysis, and cannot acquire the object's spatial coordinate information. 3D vision, on the other hand, can acquire the three-dimensional coordinate information of every point in the field of view, obtain three-dimensional stereoscopic images through algorithms, and analyze this data to derive information about the target object's position, shape, volume, flatness, etc., in space, achieving functions such as detection, guidance, measurement, and positioning.
As intelligent manufacturing continues to advance, 2D vision faces limitations in accuracy and distance measurement for complex object recognition, dimensional measurement, and the intricate interactions required for human-machine interaction. Consequently, the market demand for 3D vision is growing rapidly. In the field of humanoid robot applications, 3D vision sensors can help robots efficiently perform functions such as face recognition, distance perception, obstacle avoidance, and navigation, making them more intelligent.
Robot perception systems transform various internal state information and environmental information of robots from signals into data and information that can be understood and applied by the robot itself or between robots. In addition to sensing mechanical quantities related to its own working state, such as displacement, velocity, acceleration, force and torque, visual perception technology is an important aspect of industrial robot perception.
Visual servoing systems use visual information as feedback signals to control and adjust the robot's position and posture. Their applications are primarily seen in the semiconductor and electronics industries. Machine vision systems are also widely used in quality inspection, workpiece identification, food sorting, and packaging.
Typically, robot vision servo control is based on position-based vision servoing or image-based vision servoing, also known as 3D vision servoing and 2D vision servoing, respectively. Each of these methods has its own advantages and applicability, but also some drawbacks.
Position-based visual servoing systems utilize camera parameters to establish a mapping relationship between image information and the position/attitude information of the robot's end effector, thereby achieving closed-loop control of the robot's end effector position.
The end effector position and attitude errors are estimated by extracting the end effector position information from real-time captured images and using the geometric model of the target. Then, based on these position and attitude errors, new pose parameters for each joint are obtained. Position-based visual servoing requires that the end effector be always observable in the visual scene, and its 3D position and attitude information calculated. Eliminating interference and noise in the images is crucial to ensuring accurate position and attitude error calculations.
Two-dimensional visual servoing compares the features of images captured by a camera with a given image (not 3D geometric information) to derive an error signal. This signal is then corrected by the joint controller, vision controller, and the robot's current operating state, enabling the robot to perform servo control. Compared to 3D visual servoing, 2D visual servoing is more robust to camera and robot calibration errors. However, in the design of the visual servo controller, issues such as the singularity of the image's Jacobian matrix and local minima are inevitably encountered.
To address the limitations of 3D and 2D visual servoing methods, a 2.5D visual servoing method has been proposed. It decouples the closed-loop control of camera translational displacement from rotational displacement, and reconstructs the object's orientation and imaging depth ratio in 3D space based on image feature points. The translational component is represented by the coordinates of feature points on the image plane.
This method successfully combines image signals and image-derived pose signals, and uses the resulting error signals as feedback, largely solving problems related to robustness, singularity, and local minima. However, some issues still need to be addressed, such as ensuring the reference object remains within the camera's field of view during servoing, and the non-uniqueness of solutions when decomposing the homography matrix.
When building a vision controller model, it is necessary to find a suitable model to describe the mapping relationship between the robot's end effector and the camera. The image Jacobian matrix method is a widely used approach in the field of robot visual servoing research. Since the image Jacobian matrix is time-varying, it needs to be calculated or estimated online.
