Machine vision technology uses machine vision products, i.e., image acquisition devices, to convert the captured target into image signals, which are then transmitted to a dedicated image processing system. Based on pixel distribution and information such as brightness and color, the signals are converted into digital signals. The image system performs various calculations on these signals to extract the target's features, and then controls the on-site equipment actions based on the judgment results.
Machine vision technology, serving as the "eyes of industry," is one of the essential technological means to achieve industrial automation, intelligence, and interconnectivity in the era of intelligent manufacturing and Industry 4.0. In recent years, machine vision technology in the automotive manufacturing industry has developed and iterated rapidly in terms of automatic defect detection, intelligent identification, intelligent measurement, intelligent inspection, and intelligent interconnection. Machine vision technology is an extension of the human eye onto machines, a comprehensive technology that uses machines to replace the human eye for measurement and judgment. It is easy to integrate information and is a fundamental technology for realizing computer-integrated manufacturing.
Machine vision technology improves the reliability of quality inspection, increases production efficiency, flexibility, and automation. In hazardous work environments, it can replace human visual inspection, thus meeting the ergonomic requirements of the manufacturing process. The main functions of machine vision technology are concentrated in the following eight aspects.
a. Automated detection during the production process improves production efficiency;
b. Quality improvement and quality assurance;
c. Improve production processes and enhance ergonomics;
d. Precise measurement of parts;
e. Flexible and integrated production;
f. Production process monitoring;
g. Reduce production costs;
h. Shorten the product time-to-market.
2. Application and Classification of Machine Vision Technology in Intelligent Manufacturing
Machine vision technology is an indispensable part of the application of artificial intelligence in the automotive manufacturing industry. With the rapid development of science and technology, the application of machine vision technology in the automotive manufacturing industry will become increasingly important in the era of intelligent manufacturing and Industry 4.0.
2.1 Main Applications of Machine Vision in Intelligent Manufacturing
Currently, machine vision technology is widely used in electronics and semiconductors, automobile manufacturing, and pharmaceutical manufacturing. Its application in electronics and semiconductors accounts for approximately 47%, automobile manufacturing approximately 16%, pharmaceutical manufacturing approximately 7%, and other industries approximately 30%. With the sweeping trends of vehicle electrification, connectivity, intelligence, and sharing, the application of machine vision in automobile manufacturing is becoming increasingly widespread and in-depth, and its application share is also increasing. Its main applications in intelligent manufacturing within the automobile manufacturing industry are as follows.
a. Guidance and positioning. Generally, 3D vision is used to accurately locate and guide the robotic arm to find the material position, grasp the material, and place it in the designated location for loading and unloading operations;
b. Visual inspection. This step involves replacing human eyes in inspecting for defects in parts, such as scratches and dents during machining, and over-assembly, missing, incorrect, or reversed parts during assembly.
c. High-precision inspection. Measurement is the foundation of industry. For high-precision parts with dimensions of 0.01–0.02 mm or even micrometers, which are invisible to the human eye, machine identification is essential. d. Intelligent recognition. Image processing, analysis, and understanding are used to identify target objects and trace and collect data. Big data is applied for rapid convergence to find key features within massive amounts of information. e. Intelligent interconnection. Primarily applied in autonomous driving technology for automobiles, it interconnects data from operators, process equipment, production materials, and the production environment in intelligent manufacturing scenarios. Through deep learning, intelligent optimization, and intelligent prediction, it demonstrates the power of Industry 4.0.
2.2 Differences and connections between machine vision and human vision
As shown in Table 1, machine vision technology has many advantages and differences compared to the human eye, which is an important reason for its widespread application in the field of manufacturing.
2.3 Classification of Industrial Application Vision Cameras
Machine vision primarily utilizes industrial cameras. Currently, well-known manufacturers of industrial cameras worldwide include Cognex (USA), National Instruments (USA), Banner (USA), Keyence (Japan), Omron (Japan), Panasonic (Japan), Teledyne Dalsa (Canada), and Baumer (Switzerland). The automotive industry currently primarily uses cameras from Cognex (USA) and Keyence (Japan). As shown in Figure 1, industrial cameras are mainly classified as follows.
Figure 1. Main Classifications of Industrial Cameras 3. Machine Vision Systems
Figure 2 Main components of a machine vision system
As shown in Figure 2, the machine vision system mainly consists of an image acquisition unit, an image processing unit, image processing software, and a network communication device. The image acquisition unit is equivalent to a CCD or CMOS camera and an image acquisition card, converting optical images into analog or digital images and outputting them to the image processing unit. The image processing unit is similar to an image acquisition and processing card, storing the image data from the image acquisition unit in real time and performing image processing using the image processing software. The image processing software primarily performs image processing functions with the support of the image processing unit's hardware environment. The network communication device mainly handles the communication of control information and image data.
3.1 Image Acquisition Unit
3.1.1 Lighting source
In machine vision, the main functions of the illumination source are to create a sufficient imaging environment, ensure stable light, highlight the color of the object to be identified, illuminate the target and increase its brightness, form an imaging effect most conducive to image processing, overcome ambient light interference and ensure image stability, and serve as a measurement tool or reference. Illumination sources for machine vision can be categorized into halogen lamps, incandescent lamps, xenon flash lamps, laser lamps, fluorescent lamps, and light-emitting diodes (LEDs). Among these, LEDs are widely used due to their advantages such as small size, low power consumption, long lifespan, fast response, low operating cost, non-toxicity and environmental friendliness, and the ability to be manufactured in various shapes, sizes, and illumination angles.
3.1.2 Industrial Camera Lenses
Industrial camera lenses are tools for acquiring images. Based on different light-sensing principles, they can be divided into Complementary Metal-Oxide-Semiconductor (CMOS) and Charge-Coupled Device (CCD). Semiconductor component factories generally use CCDs to convert optical images into digital signals (making the images easier to process in post-processing) and to ensure stable information acquisition. As shown in Figure 3, lenses can be mainly classified into the following four types according to their field of view, aperture, focal length, and interface.
Figure 3. Classification of Industrial Camera Lenses. The main technical parameters of lenses include focal length (EFL), with shorter focal lengths providing a wide-angle view and longer focal lengths providing telephoto view; field of view (FOV), also called field of view range, is the size of the area actually captured by the camera, mainly related to the sensor size, the working distance between the camera and the sensor, and the lens type; FOV = sensor size / optical magnification; magnification, also known as optical magnification, is the ratio of image height to object height; CCD/FOV, i.e., chip size divided by field of view range; depth of field indicates the range of focus, referring to the range of motion of an object when it is in focus; lens mount is the connection method between the lens and the camera, commonly including C, CS, F, V, T2, Leica, M42x1, M75x0.75, etc., C... The interface is a common type of interface used in industrial lenses; the sensor size is the diagonal dimension of the image sensor; sharpness is the result of a combination of resolution and contrast. Resolution represents the lens's ability to record details of an object; the higher the resolution, the sharper the image; the aperture number (F.No.) indicates the amount of light passing through the lens. A smaller F. number means more light passes through. The F. number is the ratio of focal length to effective aperture, F.No. = EFL/D. The aperture size of the lens determines the brightness of the image. In applications involving shooting fast-moving objects or very short exposure times, a large aperture lens should be selected to improve image brightness.
3.1.3 Lighting Technology
Bright field of view and dark field of view. Bright field of view receives reflected light directly, while dark field of view receives scattered light. Dark field of view allows you to observe the outline of an object and reveal some structural problems.
Exposure. Normal exposure means neither overexposure nor underexposure. Overexposure means the image is too bright, and underexposure means the image is too dark. Polarizing technology, polarizers, and polarized light. Polarizers are used to reduce glare or specular reflection, controlling the amount of specular reflection from glossy elements. Light is redirected by a polarizer, and is used in conjunction with an analyzer in front of the lens. The advantage is that it separates specular and diffuse reflection; the disadvantage is that it requires higher light intensity due to the polarizer.
3.2 Image Processing Unit
Figure 4 shows the image processing flowchart. The image processing process is roughly divided into four steps: First, shooting, pressing the shutter to take a picture; second, transmission, transmitting the image data from the camera to the controller; third, processing, which is divided into preprocessing, which is to process the image data to make its features more obvious, and measurement processing, which calculates damage, size, etc. based on the image data; fourth, outputting the results, outputting the processing results in the form of signals to the connected control device such as PLC.
Taking Cognex cameras as an example, image processing methods include pattern matching (PatMax), histogram extraction, line/segment finding, and blob extraction. The PatMax pattern tool is used to train pattern features, comparing images with a pattern library to verify or identify specific pattern features in the image. This tool is primarily used to ensure the correctness of products running on the production line, or to identify which product is operating correctly and convey this information to other workshop equipment. The camera's acceptance threshold defines the required similarity between the model pattern and the found pattern. PatMax patterns are pixel-independent; their characteristic is that the feature-based representation of the boundaries between different regions in an image is converted faster and more accurately than the pixel-based representation. PatMax is an effective pattern localization search technology capable of handling rotation and angular changes; PatMax can also consider or ignore additional features.
Histograms can be used to determine the presence of features and whether there are any missing parts. The total number of bright and dark pixels in the histogram search box (a pixel is the smallest unit of information in an image) is used, and the score represents the number of white or black dots in the search box. Figure 5 shows the location of the gearbox oil pump seal assembly part within the red circle in the camera histogram binarization. The camera at this station uses histograms to detect whether there are any missing parts in the oil pump seal assembly. To prevent supplementary detection in case of PATMAX failure, a binarization method is used. When there is an oil pump seal part, it is black; when there is no oil pump seal, this area is pure white (the threshold is set to 40; a score less than 40 is considered as no missing oil pump seal part, and the detection is qualified; a score greater than 40 is judged as a missing oil pump seal part. Here, the score represents the number of white dots in the search box). The edge finding function is used to search for edge features within the region; the blob tool is used to find a group of pixels with grayscale values higher (lower) than a specified threshold. This function is used to find bright spots on a dark background.
Figure 5. Camera histogram binary representation of oil pump oil seal characteristics. 3.3 Information communication unit.
Common interface types for industrial cameras include analog interfaces, Cameralink, USB 2.0, 1394a, 1394b, CigE, and Ethernet. Currently, cameras used in on-site production in the automotive manufacturing industry mainly use Ethernet communication.
4 Machine Vision Algorithms
Binarization algorithm. Grayscale is in the RGB (Red, Green, Blue) model. If R=G=B, then color represents a grayscale color. The value of R=G=B is called the grayscale value. Therefore, each pixel of a grayscale image only needs one byte to store the grayscale value (also known as the intensity value or brightness value). The grayscale range is 0~255. The weighted average method is commonly used to obtain the grayscale value of each pixel. Image binarization is to set the grayscale value of the pixels in the image to 0 or 255, which means that the entire image presents a visual effect of only black and white. In digital image processing, binary images occupy a very important position. Image binarization greatly reduces the amount of data in the image, thereby highlighting the outline of the target [7]. In the process of automobile manufacturing, such as the photo-inspection of engine and gearbox glue application and the photo-inspection of stamping numbers, the binarization algorithm is used for processing.
Defect (flaw) detection algorithm. It uses a camera to detect the density of a region by dividing the region into segments and comparing each segment. This identifies areas with large differences (determined by density differentiation). A brightness tool determines the presence of a feature based on the average grayscale value. This is suitable when good components have features that are significantly darker or brighter than defective components. By scanning the product in any direction (X, Y, XY, radius, circumference, etc.), the size, direction, comparison interval, and movement of the segment can be selected according to the object being inspected. Defect levels are represented by color (dark blue → light blue → green → yellow → red), allowing for two-dimensional confirmation of the defect's range and distribution. In automotive manufacturing, applications primarily include determining the presence of machined holes, engine and transmission mounting parts, missing bolts, and machining marks based on brightness.
Deep Learning and Machine Vision. Deep learning is a field of machine learning that trains camera software to learn. Deep learning can be achieved through architectures such as artificial neural networks, mimicking the workings of the human brain by processing data and creating patterns for decision-making. Before the advent of deep learning algorithms, visual algorithms could be broadly divided into five steps: feature perception, image preprocessing, feature extraction, feature selection, inference prediction, and recognition. Cognex cameras utilize a state-of-the-art machine learning algorithm to supplement traditional machine vision. The system is trained using samples to distinguish acceptable variations and defects, specifically designed for factory automation applications. It is now a widely tested and optimized reliable software solution, and deep learning algorithms are also used in autonomous driving technology.
Convolutional Neural Networks (CNNs) and Machine Vision. CNNs are frequently used in machine vision as a relatively accurate simulation of the human brain. In machine vision, convolution can be viewed as an abstraction process, statistically abstracting information from a small region. The operation of taking the inner product of data from different data windows in an image with the convolution kernel (a filtering matrix) is called convolution. This computation process is also called filtering. The essence of convolution is extracting features from different frequency bands of the image. If a convolutional neural network is too shallow, its recognition ability is often inferior to that of a typical shallow model. However, if it is very deep, a large amount of data is needed for training; otherwise, overfitting in machine learning will be inevitable.
5. Application of Machine Vision Technology in the Automobile Manufacturing Industry
5.1 Application of Machine Vision Technology in Automobile Engine Manufacturing Process
In the engine manufacturing process, machine vision can be used in the following scenarios: cylinder block and cylinder head assembly line work, where machine vision guides a robotic arm to pick up the cylinder head onto or remove it from an automated guided vehicle (AGV).
Cylinder block stamping quality inspection. As shown in Figure 6, the main inspection principle is to use OCR character detection to build a character library. During inspection, characters are compared against the library's contents. The resulting character is then compared with the character provided by the PLC. If the stamped number detected by the camera does not match the EUN code in the code block, the camera will alarm, and the engine will be sent to the rework route or rework area. To reduce false alarms during machine vision inspection, the following points should be noted.
a. Using a fixed-focus lens improves photo stability;
b. In the camera program, "segmented reading and separate comparison" improves detection accuracy more than "whole line reading and comparison," so the former is recommended; c. Water stains, oil stains, rust, and impurities on the cylinder stamp surface will affect the photographic detection effect. Keep the stamp surface clean; d. Reflection and unstable position of the stamp number bit will affect the photographic detection effect. The bit needs to be coated with black to prevent reflection and kept in a stable position, or the camera software program should be set to take a delayed photo; e. The stability of the stamp number character quality will affect the photographic detection effect. The camera software needs deep learning to continuously train the template.
Figure 6 Machine vision inspection of engine stamp number
Parts missing or incorrect model detection. The detection process involves checking the presence and model of parts. If a part is missing or the model is incorrect, an alarm is triggered, and the engine, transmission, etc., to be assembled cannot proceed to the next workstation. It is also widely used for detecting missing or incorrectly assembled parts in other automotive components (such as valves, oil seals, flexible discs, cylinder blocks, cylinder heads, camshafts, pistons, etc., and valve, oil seal springs, etc.). Part model identification can be achieved through three methods: code recognition, character reading, and pattern recognition. Code recognition mainly includes barcodes, QR codes, label codes, and DPM codes; character reading includes OCR and OCV methods; and pattern recognition includes color and shape recognition.
Figure 7 Machine vision inspection of the presence and model of valve stems in engine cylinder head.
Figure 7 shows the model number and presence/absence of valve stem inspection on a car engine cylinder head. Initially, color-coded markings were used for visual error prevention, but due to numerous false alarms from camera detection, this was changed to character-based visual error prevention. For automotive parts, it is recommended to use combinations with significant differences between characters and color-coded markings used for machine vision recognition. For character parts, combinations like 1/7, 2/3/5/6, and 8/0 are not recommended; other character combinations are acceptable. For color-coded markings, white/blue, white/red, blue/yellow, and red/yellow combinations are recommended. Other combinations with smaller differences tend to produce more false alarms when using cameras for visual error prevention; therefore, the reference parts in the field should be kept as consistent as possible with the normal parts. When parts have excessive oil that easily reflects light, a high-penetration infrared light source can be used for the camera.
Photographic inspection of adhesive application quality. This includes applications such as adhesive application to engine oil pans, oil seals, covers, and hoods. The visual inspection and error-proofing process involves photographic detection of the adhesive application trajectory and any breaks in the adhesive. The adhesive line trajectory is determined through this inspection, and the entire trajectory is monitored. If any point deviates from the trajectory or breaks in the adhesive, the camera alarms, and the workpiece cannot proceed to the next workstation.
Figure 8 Machine vision inspection of adhesive application trajectory and adhesive breakage on engine hood
Figure 8 shows a camera inspection of adhesive application on the engine hood. When using an industrial camera for adhesive application quality inspection, instability in the adhesive application trajectory and quality affects the camera's detection performance. For example, if the adhesive strip is too short in a small area, starts thick and ends thin, overlaps, trails, or has a poor trajectory, the camera's false alarm rate is high, requiring adjustment of camera parameters. A misaligned or uncleaned application head also increases the false alarm rate, necessitating timely cleaning and inspection of the application head during each shift or long downtime. For visual error-proofing inspection of black-based adhesive on the oil pan, using a polarizing filter can better highlight the adhesive strip outline. Visual inspection is also used for classifying and comparing characters on engine cylinder block bearings.
Detection principle: An OCR character detection function is used to build a character library. During detection, characters are compared against the library's contents to obtain the correct character. Simultaneously, the QR code information is read for cross-verification, ensuring accuracy. In terms of visual inspection technology, attention is mainly paid to the proper setting of exposure parameters and template settings in the camera detection program.
Visual inspection is used to compare the installation orientation of engine pistons and connecting rod bearing caps. The inspection principle involves shape pattern recognition and comparison with a standard template. If a mismatch is found, the camera takes a picture and triggers an alarm, sending the engine to the repair detour or loop. Key aspects of the visual inspection technology include proper setting of exposure parameters and template settings in the camera inspection program.
The engine and transmission workshop also uses 3D cameras for adhesive application inspection, rocker arm installation status inspection, and automatic robot grasping, as shown in Figure 9, where a 3D camera guides the application of adhesive to the engine high-pressure oil pump.
Figure 9. Application of 3D machine vision technology in engine adhesive application guidance and tracking. 5.2 Application of machine vision technology in automotive transmission manufacturing process.
The main applications of machine vision in gearbox manufacturing workshops are defect inspection, error prevention, positioning, measurement, and QR code recognition. Error prevention uses comparative analysis of features to determine the presence, reverse installation, improper installation, or model discrepancies of workpieces. Positioning outputs workpiece coordinates in 2D and 3D to guide machine movement. Measurement obtains various parameters for quality assessment and process improvement by inspecting products. QR code recognition reads product barcodes and QR codes to obtain product models and enable traceability. Figure 10 shows the distribution of over 50 cameras in a gearbox assembly workshop, primarily used for three applications: detecting whether workpieces are properly installed, checking for workpiece presence, and preventing model discrepancies.
Applications of machine vision in positioning include part positioning and material handling, and machine path guidance. For example, when loading and unloading gearbox housing assemblies and torque converter housing assemblies, machine vision guides a robotic arm to pick up the gearbox housing assemblies and torque converter housing assemblies onto or remove them from the AGV.
Figure 10 shows the application of machine vision technology in a gearbox workshop, specifically in the detection of the correctness of parts in KIT trays or meal boxes. The detection checks for the presence or correct placement of parts such as valve cores, springs, and caps of control valve bodies and solenoid control valve bodies in the KIT trays or meal boxes. The photographed results are compared with a standard template; if discrepancies are found, the camera alarms, and the KIT tray or meal box cannot proceed to the next workstation.
Parts inspection and measurement, such as determining whether a retaining ring is installed correctly by measuring the distance between the two holes of the retaining ring.
Check the presence of the sun gear in the transmission, the model of the differential, and whether the retaining rings are missing or improperly installed.
The system checks whether the solenoid valve body wiring harness clips are missing or not properly installed. As shown in Figure 11, a color histogram is used to detect the area within a fixed range. A red value less than a certain threshold indicates the clip is in place, while a red value greater than a certain threshold indicates it is not in place. For example, based on the condition of a defective part, a judgment threshold of 180 can be set. A red score less than 180 indicates the clip is in place, and a red score greater than 180 indicates it is not in place. The judgment threshold needs to be set and verified according to the actual defective part.
Figure 11 illustrates the use of machine vision technology to detect whether gearbox wiring harness clips are properly installed or missing, and to check the levelness of the electronic valve body press-fit. The detection principle involves first creating a pattern model, then locating the pattern, determining the rotation angle, and judging whether it meets the requirements. If it fails, a camera alarm is triggered, and the workpiece will not proceed to the next station. Machine vision is also used for detecting customer-specific characteristics and quality defects at the CARE and final inspection stations. For example, it can be used to prevent errors such as missing machining of machined holes and threaded holes, missing tightening or installation of gearbox seals and pressure test plugs, and misalignment of gearbox wiring harness connector pins.
Figure 12 summarizes the potential factors affecting camera uptime in a workshop's machine vision application process over a year. During the image capture stage, factors such as light source brightness, image positioning, consistency of incoming part surface quality, presence of oil on the part surface, exposure, and focal length all affect image quality, while the camera's field of view affects the imaging range. To address camera false alarms caused by the program, the main approach is to increase the training sample size and optimize parameters based on the sample data. For false alarms caused by differences in incoming materials, the main approach is to optimize camera parameters (such as exposure and judgment thresholds). For false alarms caused by lighting conditions, the main approach is to add a black box to the camera's workstation to ensure lighting stability, or consider color processing of some skylights.
5.3 Application of Machine Vision Technology in Automobile Manufacturing Process
Machine vision technology is also widely used in the automotive manufacturing process, including bodywork, painting, stamping, and final assembly. Its main applications include: detecting quality defects in automotive manufacturing, such as mis-assembly, omissions, and reversed parts; checking for threaded holes, machining marks, cracks, burrs, keyholes, and welding quality; 2D and 3D vision measurement, enabling 3D dimensional inspection of automotive parts, assembly dimensions, panel inspection, and body flatness inspection; vision positioning or guidance systems, such as 2D robot positioning systems (e.g., automatic screw tightening machines, automatic soldering machines, automatic dispensing machines) and 3D robot vision guidance systems (e.g., automatic guided glue application, parts processing, sorting, and assembly); equipment diagnostics, testing, and maintenance; robot control and CNC machining; material handling equipment (e.g., parts sorting, palletizing, and depalletizing); equipment motion control (e.g., autonomous driving and automatic operation in automobiles); and continuous and batch processing.
6. Future Development Trends of Machine Vision in Automobile Manufacturing Applications
The future development trends of machine vision technology in the automotive manufacturing industry are as follows: a. Positioning and grasping of disordered automotive parts; b. Surface defect detection of automotive parts; c. Error-proofing inspection based on deep learning; d. Intelligent measurement of automotive part dimensions; e. Intelligent inspection based on complex logic-based intelligent judgment; f. Intelligent interconnection technology and autonomous driving, etc. 7. Conclusion
This paper analyzes the fundamental theories of machine vision technology and its application in the automotive manufacturing field, summarizes its role and solutions to practical problems in its application, and predicts its future development trends. Machine vision technology can effectively reduce automotive manufacturing costs, improve production efficiency, production flexibility, and the degree of automation, thus laying a solid foundation for the electrification, connectivity, intelligence, and sharing of the automotive manufacturing industry, and promoting higher, faster, better, and more sustainable development in the automotive manufacturing industry.
