The environmental perception sensors involved in autonomous driving technology mainly include visual cameras (including monocular, binocular stereo, panoramic, and infrared cameras) and radar ranging sensors (LiDAR, millimeter-wave radar, ultrasonic radar, etc.), as shown in Figure 1. Open-source SDKs are available for rapid development of these sensors. Based on my development experience over the past few years, I have compiled a list of APIs for commonly used sensors to help beginners save development time.
Of course, for wealthy companies that can customize sensors according to their needs, this article is redundant and they can skip it. This article is mainly for junior developers of autonomous driving systems, primarily researchers in universities and research institutions, focusing on performance and giving less consideration to cost. The author mainly develops on the Linux system, and the APIs introduced are based on Ubuntu 14.04 and above (Ubuntu 18.04 will be released in April, a stable version every two years, which is worth looking forward to).
Figure 1 Mainstream Sensors for Autonomous Driving
Monocular camera
AVT industrial cameras are highly recommended, with numerous domestic distributors, such as Daheng. They offer a comprehensive range of cameras with interfaces including 1394 FireWire and Ethernet, with prices ranging from several thousand to tens of thousands of yuan. The SDK can be found on their official website ( https://www.alliedvision.com/cn.html ) . The latest Vimba_v2.0 development kit is recommended, as installation is relatively simple, and the SDK includes multiple development examples, including QT and OpenCV, making it easy to get started.
Applications: The development of monocular camera applications mainly includes the detection and recognition of feature-based symbols, such as lane line detection, traffic sign recognition, traffic light recognition, pedestrian and vehicle detection, etc. Machine learning-based visual computing will definitely be an indispensable part when autonomous driving becomes widespread. Although the reliability of visual detection is not very high at present, its application in autonomous vehicles that use LiDAR as the main perception method has not met expectations.
Figure 2 Road surface and vehicle recognition
Binocular camera
Binocular vision applications in outdoor scenarios are indeed rare. The Bumblebee binocular system I used before was primarily for indoor use, and the ZED camera also performed better indoors (after all, the baseline length is fixed, it's only so short!). For outdoor applications like visual odometry or recognition algorithms, I recommend building your own binocular sensor and determining the baseline length based on specific needs while ensuring synchronized triggering. Binocular vision inevitably involves disparity maps and binocular calibration. Currently, a common approach is to use Zhang Zhengyou's method, calibrated using CameraCalibrationToolbox (website: http://www.vision.caltech.edu/bouguetj/calib_doc/ ). Of course, OpenCV also integrates this method; those accustomed to C programming can refer to pages 464-492 of the classic OpenCV textbook "Learning OpenCV" (Chinese version ) , which provides detailed content and code. I won't elaborate further.
Figure 3. Binocular stereo vision calibration
Applications: Obstacle detection is possible, but limited to indoor use, structured light, and short-range applications; visual odometry is indeed a promising application area, used for intersection localization, etc. It is also used in various open-source SLAM technologies, such as ORBSLAM2 ( https://github.com/raulmur/ORB_SLAM2 ). "Visual SLAM in Fourteen Lectures" provides a detailed introduction and application of visual SLAM, which is a good book and I recommend it.
panoramic camera
It is divided into single-lens panoramic cameras and multi-lens stitched panoramic cameras. I haven't developed many applications based on panoramic vision (when used with multi-lens cameras, their panoramic advantages are not fully utilized). I seem to recall seeing one installed in a car when Tencent was developing street view maps in 2015.
Applications: Panoramic cameras are highly recommended for visual odometry due to their wide field of view and high correlation of feature points. I personally believe that using panoramic vision in conjunction with integrated navigation for high-precision map reconstruction is a good choice to achieve lane-level positioning for autonomous vehicles. Other applications have not been studied in depth.
Infrared camera
Infrared cameras belong to another category of vision, offering better night vision than daytime vision, and can be used for pedestrian and vehicle detection. Previously, it seemed that infrared cameras were in an awkward position in autonomous driving applications after the advent of LiDAR, being expensive and not as direct as LiDAR results. However, after use, it was found that in obstacle recognition (such as people), the discrete point cloud of LiDAR still cannot compare to the spatially continuous image. Furthermore, surprisingly, tests revealed that infrared cameras can actually capture human shadows on glass! This is quite alarming! Infrared cameras can distinguish heat-generating objects to some extent, such as roads and pedestrians, but post-processing is still required, unlike LiDAR's direct obstacle detection using absolute height or gradients. In nighttime conditions, they can replace color cameras for forward-looking obstacle detection and monitoring.
Figure 4 Infrared visual imaging
millimeter-wave radar
Suitable for high-speed environments and platooning, it seems unnecessary to install in other scenarios. Individual purchases are expensive, wholesale purchases are cheaper. I've used a Delphi-based system, which is probably widely used in China. It has a maximum range of 150 meters and two working modes: single-point tracking and multi-target detection. The former is suitable for detecting the vehicle in front during platooning, while the latter is suitable for long-range target detection in high-speed environments. Because millimeter-wave radar's imaging principle is cone imaging (equivalent to radiating outwards from a point into a cone), it infers obstacles based on the area of the obstacle within the cone. Therefore, in complex scenarios, obstacles can appear quite cluttered. Highway scenarios are more ideal, with good road conditions and relatively few vehicles, making it suitable for early warning of vehicles ahead. (Tesla believes that vision + millimeter-wave radar is sufficient to support autonomous driving systems, which I largely agree with, but it seems the algorithm still needs further development.)
Development: Data is sent and received via CAN. The data format is relatively simple and is generally defined in the protocol file. Don't panic if you have no background in CAN port development; it's quite simple. Purchase a USB-CAN converter . The website ( http://www.pudn.com/Download/item/id/3353963.html ) has dependency libraries for receiving CAN data on Ubuntu systems. After that , just follow the protocol to parse the data.
LiDAR
2018 was undoubtedly a year of explosive growth for LiDAR. Industry giant Velodyne launched new series such as the 128-line and VLP-32C, while existing models saw price reductions (consult domestic distributors for specific prices). The products shown in Figure 1 are all from Velodyne. A usable data interface is available in ROS ( https://github.com/ros-drivers/velodyne ). I feel that the Sick single-line LiDAR has fulfilled its historical mission, as its price advantage has disappeared with the price reduction of the VLP-16, and its supply chain advantage is unlikely to last long (currently, there is a supply shortage for all Velodyne models). Meanwhile, domestic companies like RoboSense, Beike Tianhui, and Hesai Technology also launched their own products, with significant price and supply advantages. As for the actual testing results, seeing is believing; I recommend trying them out in practice. I won't elaborate further.
Applications: Obstacle detection is mainstream, used by Waymo, Baidu, major traditional car manufacturers, startups, universities, research institutes, etc. Almost all autonomous vehicles are equipped with LiDAR (except Tesla); high-precision map reconstruction and environmental modeling; SLAM, etc.
Multi-radar fusion
