Demanding machine vision applications increasingly rely on the design, specification, and integration of more complex components and systems to achieve successful and reliable performance.
The rapid growth of the machine vision industry in recent years is partly due to the continuous development of machine vision technologies and components. The overall success of machine vision applications is undoubtedly closely related to competent system integration – the analysis, specification, design, and implementation of task-specific components are crucial. In this discussion, we will review machine vision system integration and introduce some important technologies in the machine vision industry, such as 3D imaging, color imaging, deep learning, and line scan imagers, along with practical information to help you successfully integrate and implement them in inspection, measurement, and guidance applications.
Components for acquiring and processing simple grayscale images undoubtedly dominate the foundation of machine vision applications in today's market. These devices are certainly reliable, easy to use, and suitable for a wide range of inspection, guidance, and metrology projects. However, other available technologies are better suited for certain applications and may be the only appropriate solution for some. In all cases, basic integration concepts should be followed to ensure the optimal design of the machine vision solution.
With that in mind, let’s start with a brief review of best practices for system integration, and then discuss how to apply and integrate some key machine vision technologies that go beyond basic grayscale cameras.
Review of best practices in machine vision system integration
While these guidelines are essential for machine vision projects, they can be applied to any automation project across various disciplines. Modify the basic concepts to best suit your needs.
Developing and using formal application analytics documentation
Analyze and document the requirements for existing operations, processes, or automation, and collect all details related to the target application, component, and product.
Document Project Specifications
Define the functions and operations the system will perform (inspection, guidance, measurement, etc.) and related performance metrics. Provide a detailed description of the technologies supporting the proposed system, as well as potential system limitations and exceptions.
Focus on the "critical path" component
Machine vision systems are often “critical path” components. If they cannot perform as needed in the system, they may hinder the overall operation of the project. Therefore, it is very important to design the system around the functionality of critical components.
There is an integrated or project plan and an organized list of tasks.
Use project management software tools to keep projects running smoothly, but be sure to define the details of tasks, not just timelines. Develop guidelines for implementing and configuring machine vision, automation, user interfaces, and other components in the system, and test them frequently.
Project Management
No software tool can replace good project management; technical projects truly require management. Effective communication between the project team, sales, management, and clients helps ensure successful project implementation.
Installation – Optimization, not Design
While there are instances where online installation is necessary for imaging, processing, or program development, this should be the exception, not the norm. Effective testing and validation before final installation are crucial for successful project implementation.
Own and use a verification program
Always maintain and use a written validation plan to quantify system performance. This document should outline functional and operational standards, as well as metrics indicating system compliance with specifications.
Application of key machine vision technologies
For machine vision system architects, there are now more viable component options to consider and apply to successful integration across a variety of applications. Not all technologies are new, but improvements in equipment and software help make components more widely applicable and easier to integrate. Here, we describe some general categories of technologies beyond basic grayscale cameras, and some methods for integrating them into successful machine vision solutions.
Line scan imaging
Line scan refers to the fact that the imaging device does not take a snapshot (“area” imaging) at once. The device only captures thin “slices” or single “lines” of image information and processes this linear data (uncommonly) or combines many lines into consecutive lines of a standard image for subsequent processing. In other words, the object or camera must be moved to create the image.
Line scan imaging offers several significant advantages over area imaging when properly integrated. Today's line scan cameras have sensors with widths ranging from approximately 0.5K pixels to over 16K pixels (the number of rows is variable, depending on how many rows are selected for the resulting image). The resulting combined image can produce a higher resolution in terms of pixel count than area cameras, and potentially at a lower cost. Because line scan imaging captures only a single linear "slice" of an object at a time, the technique can produce very consistent images in situations where area cameras cannot. Consider a cylindrical object (e.g., a beverage can or similar container). By imaging a small slice of the cylinder's circumference, a very spatially accurate representation of the cylinder's surface can be reproduced, which is impossible with area images due to the perspective of the cylinder's curvature. In some implementations, line scan imaging can produce faster processing times than area camera imaging. High-speed line scan cameras can capture each row of an image at rates exceeding 120 kHz (frame rate varies depending on the camera and interface type).
In many cases, 3D imaging provides useful data that standard 2D grayscale imaging techniques do not have, including depth and contour information.
Since a single row of pixels must be illuminated, lighting for line scan applications can be simpler than for area cameras. Typically, a linear light source (front or back illumination) is all that's needed to obtain a suitable image using a line scan camera. However, note that using line scan may not always allow for some lighting techniques that are advantageous for area cameras.
Optics (lenses) for line scan applications is often the biggest integration challenge. When the linear pixel width is greater than 2K (depending on the pixel size), standard, familiar C-mount lenses are typically incompatible with the camera, and F-mount or even more specialized lenses must be specified. In some cases, lens assemblies for line scan cameras can be very large and require specialized mounting.
Integrated line scan cameras typically require the use of an encoder to trigger the line at appropriate spatial intervals. The configuration of precise pulse timing relative to movement depends on metrics such as pixel spatial resolution, motion speed, and frame rate. Fortunately, advancements in cameras and camera software have helped to make configuration more user-friendly.
3D imaging
Many applications can benefit from the recognition or representation of objects and features appearing in 3D space. In recent years, the availability and variety of 3D imaging systems for critical machine vision applications such as inspection, defect analysis, metrology, and robot guidance have increased significantly. In fact, 3D technology has successfully enabled the realization of certain applications that could not be reliably addressed using grayscale imaging.
The key value of 3D imaging lies in the fact that all image information represents actual points in space, rather than grayscale or color intensity values as in 2D imaging. These points can be calibrated to some known world coordinate system (e.g., when the information is used for guidance) or simply analyzed relative to each other to extract features, defects, and perform 3D measurements. Note that a 3D system provides the surface “profile” as seen by the camera.
When describing how to select components, it can help in considering 3D technology within the application category.
Measurement and Defect Detection
Many 3D imaging systems used for metrology and defect detection employ scanning and laser triangulation (sometimes referred to as distance or depth sensors) to construct images. These systems can provide higher accuracy than other 3D imaging techniques, although these components have a slightly limited field of view and do require the sensor or part to be in motion to capture the image. The imaging tools and algorithms typically provided are designed to evaluate 3D features and provide measurements including depth and, sometimes, position.
Guidance and object location or identification
Other 3D systems primarily focus on guidance, localization, and/or recognition applications. Some components in this category also utilize 3D laser scanning, while others employ various techniques involving structured illumination to collect 3D points. For the latter, the advantage is that the imager takes a single snapshot without camera or part movement. The images returned by these systems can be depth images like those from a distance sensor, or collections of 3D mapped points known as “point clouds.” The software in these systems localizes guidance and location tasks and extracts geometric features by combining neighboring points in various ways. These features can be simple localization points or even 3D spatial matching of objects defined by CAD data. In implementation, systems can typically locate known and unmixed parts with good reliability in random orientation; however, the identification and localization of randomly mixed parts or objects is an extremely difficult task and remains a research endeavor for many applications.
CAD reconstruction
Some 3D imaging systems are specifically designed to scan objects and capture surface data in order to convert that data into a 3D CAD model for further processing. These are highly specialized systems that perform reconstruction tasks very well, but are not suitable for other tasks.
Deep learning
Perhaps the hottest buzzword in the machine vision industry right now is "deep learning." Machine learning, implemented using software algorithms called "neural networks," has been around for years. Deep learning is the latest extension of this technology, specifically combining network layers called convolutional neural networks, or CNNs. A simple description of this technology is that it learns to classify images by examples. (Think of Google's image search function; you can upload a picture of a cat, and "artificial intelligence" will classify it as a cat, and might even know the breed.)
Most computer programming enthusiasts can understand the most basic neural networks and deep learning. However, machine vision integrators seek to implement the technology using a growing number of products and libraries that offer deep learning capabilities for machine vision, thus requiring no understanding of how deep learning works. The integration of deep learning into machine vision is almost entirely a "demonstration and move-through" process.
Nevertheless, deep learning is not suitable for all applications. Furthermore, the key detail in integration remains the purely imaging task. Deep learning is particularly useful for applications requiring subjective classification, such as detecting defects or non-conforming features—parts that are subjectively unacceptable—and where nominally good parts can be well defined (even if the good parts have some inherent variations). Applications that typically do not readily employ deep learning are those requiring discrete analysis, metrology, or localization. Some applications can be adequately addressed using standard machine vision, 3D imaging, color imaging, or deep learning; in these cases, a trade-off between integration and configuration costs and technology is necessary.
Many misconceptions about deep learning stem from the idea that the technology automatically overcomes integration considerations such as lighting, part display, and optics. In this sense, deep learning imaging is the same as all machine vision applications: objects, defects, or other features of interest must be distinguished from other parts in terms of contrast and sufficient pixel resolution, and the presentation must be controlled to ensure correct imaging.
Color, multispectral and hyperspectral imaging
Color imaging integration in machine vision is extremely useful for applications that need to identify, differentiate, or verify the colors of objects or features. Having color image data can enhance feature contrast and isolate features in situations where grayscale images cannot. Standard color cameras and tools are widely used in basic machine vision systems and are an imaging option that can be easily considered and integrated when appropriate. The most common color cameras capture images using three broadband color planes; red, green, and blue, and combine these color planes to create a full-color image.
A more powerful color imaging technique, which has become more widespread in recent years but is not new, is hyperspectral imaging. This type of camera acquires images through many (sometimes 100 or more) very narrow color bandwidths, typically ranging from ultraviolet to near-infrared or short-wave infrared. This enhanced capability allows integrators to define complex color “profiles” for objects or features with a precision unavailable in basic RGB color imaging. Even features that may be indistinguishable or invisible to the human eye, defects and colors can sometimes be easily detected with hyperspectral imaging. Closely related to hyperspectral imaging is multispectral imaging. Multispectral cameras provide fewer color planes in an image (typically less than 10) and often select color bandwidths and tune them for a specific application. The specification and integration of hyperspectral and multispectral cameras can be complex. Most hyperspectral systems require an object. Selecting bandwidths and tuning color profiles can require some technical expertise.
In all color applications, a key integration challenge is illumination. Without a light source that illuminates all the colors of interest, it's impossible to image color content. The first instinct is to use a "white" light, but how do we define "white"? Unfortunately, popular machine vision light sources like LEDs may not be "white" because they don't provide uniform illumination without loss of color or strong peaks at certain wavelengths. For a given application, the available "white" may be sufficient, as long as applications requiring closer color differentiation generally need a more uniform white light source. The usual solution is to use incandescent light sources like halogen or xenon, but recently, new LED technologies called "multispectral illumination" have emerged that combine multiple LED colors and balanced outputs to achieve near-full-bandwidth illumination.
In summary, standard machine vision components and systems remain a good integration option, but having a wider set of tools can enhance the range of machine vision applications that can be successfully and reliably integrated.
