Creating robots with human-like intelligence that can replace human labor has always been a human dream, as humans acquire over 80% of their information through vision. Therefore, research on vision-enabled robots has become paramount in intelligent robot research. Research on robot visual servoing systems is a crucial aspect of robotics, with results applicable to problems such as automatic obstacle avoidance, trajectory tracking, and moving target tracking. Classified by feedback information type, robot visual servoing systems can be divided into position-based and image - based systems . Position-based systems first estimate the target object's position relative to the camera in Cartesian coordinate space. The visual servoing error is defined in three-dimensional Cartesian space. Visual or feature information is used to estimate the relative pose of the robot's end effector to the target. This method requires precise calibration of the vision system and the robot, and the computational load is significant due to the need for image interpretation. The servo error of an image-based visual servoing system is directly defined in the image feature space, meaning that the feature information observed by the camera is directly used for feedback without the need to estimate the 3D pose. However, the system needs to calculate the image Jacobian matrix (the relationship matrix between the changes in image feature parameters and the changes in pose in the task space) and its inverse matrix online. The image Jacobian matrix is related to many real-time changing parameters, which is a complex nonlinear process that is difficult to analyze theoretically and places high demands on the robot's control design [ 2 ]. The robot visual servoing system is a very complex system that integrates the content of many disciplines, and the development of each discipline is extremely uneven, affecting its further development. Based on this, the research on robot visual servoing is currently stagnant. Most domestic research in this area has only conducted simulation experiments and has not been implemented in actual robot systems. This paper uses a Panasonic AC servo system, a PMAC motion control card, a DSP image processing system, an industrial computer, and a robot to form a servo system. Experimental research is conducted on it in a specific robot system to explore the implementation problem of image-based robot visual servoing and the components of the robot visual servoing system.
Robotic Systems and Research Content
The first joint of the two-DOF planar robot is vertically fixed to the work platform. Both joints are driven by Panasonic AC servo motors and can only rotate in the horizontal plane. A monocular CCD camera is fixed to the end of the second link. A DSP image processing system is used to acquire and process images, extracting target feature values as visual information feedback. This paper selects the coordinates of the gray-scale centroid of a circular target object in the image plane as image features. This paper studies the servo implementation problem of an image-based robot visual servo system. By controlling the movement of the robot's two joints according to the changes in the target object's image feature values, the end of the second link can track the stationary or moving object on the work platform, thus achieving visual servo control of the system.
Mathematical model of robot vision system
This paper aims to conduct experimental research on the system. The coordinate transformation relationships between robot coordinates, camera coordinates, image coordinates, and target point coordinates are established. Figure 2 shows the coordinate transformation relationship established when a camera is installed at the end of the robot's second link, forming an eye-in-hand configuration. Figure 3 is a schematic diagram illustrating the relationship between the camera system coordinates and image coordinates, as well as the imaging principle.
Let the rotation angles of the first and second joints of the robot be θ1 and θ2, respectively , the length of the first link be l1 , and the length of the second link be l2 . Select point o in Figure 2 as the base coordinate reference point, denoted by [ x0 y0 z0 ] t , the camera coordinate system by [ xc yc zc ] t , and the image coordinate system by [ uw ] t ; the first joint of the robot is denoted by [ x1 y1 z1 ] t , and the second joint by [ x2 y2 z2 ] t . Let the homogeneous coordinates of the fixed target point pc in the camera coordinate system ( c ) be: pc = [ xc yc zc1 ] t , and the homogeneous coordinates in the base coordinate system ( o ) be: p0 = [ x0 y0 z0 1 ] t . The transformation relationship between the two coordinate systems is: p0 = 0tcpc .
Based on robot kinematics, the transformation matrix between the camera coordinate system and the base coordinate system is as follows:
Panasonic AC servo system
In this experiment, both joints of the robot are driven by Panasonic AC servo motors. The motors and their drive controllers together constitute the Panasonic AC servo system. The Panasonic Minasa series AC servo system consists of an A- series driver and a matching AC servo motor. It is based on the three-phase AC servo system principle [ 5 ], and uses a photoelectric encoder as the motor speed and rotor magnetic pole position detection unit. The system integrates relevant functional modules inside the driver, forming a highly integrated, highly accurate, digital, and intelligent closed-loop servo control system. According to the desired input command value, the driver controls each relevant functional unit according to the pre-set control mode and parameter settings, generates a signal to control the IGBT PWM module, and finally controls the servo motor to operate according to the desired value. The system has a frequency division and multiplication function, and can output feedback pulses to an external controller or receive position control command signals in various forms. The system has multiple control modes, such as speed and torque control modes, with analog signals as input. This paper uses the torque control mode [ 6 ].
PMAC motion control card
The PMAC motion control card is a programmable, high-performance servo motion multi-axis controller manufactured by Delta Tau Data Systems, Inc. in the United States. It uses Motorola 's DSP56001 digital signal processing chip as its CPU .
The PMAC can be considered a complete computer, using a DSP chip as the main processor to handle all calculations for eight axes and simultaneously control the motion of all eight axes. It has independent storage space, input/output interfaces, and other peripheral expansion circuits. It can be combined with various types of host computers, amplifiers, motors, and sensors to complete various types of functions. As long as we make good use of its hardware and software characteristics and set it up according to specific functional requirements, it can work normally and efficiently. The PMAC card provides users with pwin32 application software and pcomm32pro dynamic link library. High-level languages can directly call relevant functions to communicate with the PMAC card through the pcomm32pro dynamic link library, thereby realizing the control of the PMAC card [ 7 ]. This paper uses it in conjunction with the Panasonic AC servo system.
DSP image processing system and image processing
System Introduction
This experiment uses the Seed-VPM642 video processing system from Beijing Hezhongda Company . This system is a PCI card or processing system with a 10/100M Ethernet interface , specifically developed for various video applications . It is based on the TMS320DM642 chip and expands upon it with a 4M × 64- bit synchronous dynamic memory (SDRAM); a 4M × 8 -bit in-circuit electrically erasable / write asynchronous memory (FRAM); two UART serial interfaces configurable to RS232/RS422/RS485 standards ; four PAL /NTSC standard analog video inputs, one PAL /NTSC standard analog video output ; four stereo audio inputs / outputs ; a real-time clock (RTC) and a 512 × 8- bit EEPROM; a 32- bit 33MHz PCI interface supporting master / slave mode or a 10/100MBase-TX standard Ethernet interface ; a standard ATA hard drive interface ; and an ESM hardware encryption module. The system can perform real-time encoding and decoding operations on multiple digital video / audio channels, such as MPEG4 , H.264 , and G.729 ; it can receive four video / audio inputs in real time and output them in real time. It can also achieve high-speed real-time data transmission and processing with other I/O devices, computers, storage devices, and Ethernet networks. The system consists of an experimental enclosure, a CCD camera, a Seed-VPM642 processing board, etc. Two serial interfaces, a hard disk interface, four image input and output interfaces, and an LCD display are already fixed on the enclosure [ 8 ].
Image processing
The specific implementation process of image processing in this experiment is shown in Figure 4.
The color analog video signal captured by the CCD camera and transmitted in a 4:2 : 2 format is acquired by the decoding chip. The Y , CB , and CR signals are sampled at a 2:1 : 1 sampling rate and separated. In the DSP system, the color image is converted into a grayscale image. After threshold binarization, the image becomes a binary image. Then, the noise points are filtered out by median filtering, and the Sobel operator is used for boundary extraction. The centroid image coordinates are obtained by the Hough centroid retrieval algorithm [ 3 ] for the boundary image signal, and the centroid coordinate values are transmitted to the industrial control computer program through serial communication . At the same time, the processed digital video signal is encoded by the encoding chip and sent to the display device for real-time display.
Robot Vision Servo System Workflow
Hardware process
This experimental system consists of an industrial control computer, a PMAC motion control card, a Panasonic AC servo system, a DSP image processing system, and a robot. After understanding and becoming familiar with the principles and functions of the Panasonic AC servo system, the operation of the robot's joint servo motors was debugged, and relevant control parameters were optimized to obtain the best servo effect . The principles and functions of the PMAC motion control card were studied , and in conjunction with the Panasonic AC servo system, motion programs were written. The control parameters of the PMAC card were continuously adjusted and optimized to obtain satisfactory motion control effects . After in-depth study and research on the Seed-VPM642 image processing system, a video image processing program was developed. The hardware flowchart of the system is shown in Figure 5 .
The PMAC control card is installed in the industrial PC and communicates with it via the ISA bus. The industrial PC mainly runs the main control program, the PMAC application program, and the DSP application software CCS . The Seed-VPM642 experimental box is connected to the industrial PC via an RS232 serial data cable for data communication. The PMAC control card is connected to the Panasonic AC servo system via a cable, receiving encoder signals from the Panasonic servo controller and outputting command signals to the Panasonic AC servo system. The CCD camera captures images of the target object and outputs continuous color video images to the DSP image processing system. The Seed-VPM642 image processing system processes the images captured by the camera, calculates the centroid coordinates of the circular object, and transmits them to the industrial PC via serial port.
System Workflow
Given the centroid image coordinates of the desired target object, the main control program calculates the difference between these coordinates and the centroid image coordinates fed back by the DSP image processing system. It then determines whether the system has reached the servo position. If it has, the servo process ends ; otherwise, it solves the inverse of the image Jacobian matrix to obtain the angle values that the robot's two joints should rotate, and outputs these values to the PMAC motion program. The PMAC card calculates and outputs torque commands to the Panasonic servo controller according to the motion program, which directly controls the operation of the robot's joint motors. The encoder detects the joint position and feeds it back to the servo controller and PMAC card (the servo controller processes the encoder signals for its own use and can also output them simultaneously). The CCD camera follows the robot's movement, capturing images of the target object. After processing by the DSP image processing program, the centroid image coordinates of the target object are obtained and transmitted to the main control program via serial communication as visual feedback, forming a visual servo loop.
Experimental results
Based on the above system principles, a main control program was written in the VC++ 6.0 environment. The main control program mainly includes the interface display and human-computer interaction part, the image Jacobian matrix inverse part, the PMAC application part, the serial communication part, and the data storage part. The program running interface is shown in Figure 6 .
The program interface allows users to set the desired pixel value, the initial value of the actual pixel value, the gain matrix, the initial joint angle, and the data acquisition interval. The program can display and store the actual pixel value, joint angle increment, and actual joint position value in real time. Communication with the PMAC card is achieved by calling PMAC dynamic link library functions . The gain matrix, a diagonal matrix, is used to set the amplification ratio of joint angle changes. The program sets two timers: one for acquiring the actual position values of the two joints, and the other for periodically receiving serial port data values, calculating the Jacobian matrix, and communicating with the PMAC card; this constitutes the system's servo cycle.
