Perception is the basic ability to realize self driving. There are several ways to achieve this.
- With expensive sensor combination kits, such as single or multiple Lidar, increasing the perception redundancy of the vehicle.
- Use and optimize the advanced fusion perception algorithms and deep learning neural networks, such as BEV+Transformer.
- With the help of “perception sensors”, such as “HD maps”, obtain global over-the-horizon perception information of the environment.
However, in the process of intelligent driving development, the stacking of multi-sensing sensors improves the parameters and also increases the cost. Meanwhile, the “heavy perception, light map” route, also lets the HD map into an awkward situation.
When competitors of intelligent driving enter complex urban scenes with “no map” solutions, the improvement of perception ability has to be made. It is necessary to find a perfect balance in the of cost, reliability and efficiency.
High Definition Map
HD Maps in self driving
In smart driving, three different types of maps are commonly used: standard navigation maps (SDmaps), high definition maps (HD maps), and ADAS maps.
Standard navigation map contains the global road network, road topology semantic information, and road background, etc., which is currently mainly used for vehicle navigation.
ADAS map, on the basis of the standard map, the relative position information such as lane slope, curvature, heading and simple lane are added, but there is no absolute position information. It is currently used for some L2 level intelligent driving.
HD map, compared with standard navigation map and ADAS map, is a HD electronic map. It covers almost all the information on the road, can provide over-the-horizon environment awareness ability, and provide lane level optimal path planning.
Because of the rich road information, HD maps can achieve more reliable fusion high-precision positioning in intelligent driving, thus becoming the “external perception sensor” of vehicles in different driving scenes. In short, HD maps provide two essential perceptual capabilities for intelligent driving. They are rich global road information and high-precision localization of the vehicle.
How to make High Definition Map (HD map)?
High-definition maps usually includes acquisition and making. In the acquisition, a professional data acquisition vehicle equipped with high-precision integrated inertial navigation, camera and LIDAR collects road data. The accuracy of the sensors on the vehicle, the time synchronization between different sensors and the space calibration accuracy, all affect the accuracy of the collected data. The higher the accuracy of the map, the higher the cost of the sensor is required.
In the making , the collected sensor data will be processed to obtain the accurate global position and attitude of the sensor. The collected data will be fused to obtain accurate 3D point cloud and color information. Through AI automatic detection and manual labeling, rich information in the map is obtained.
Disadvantages of HD Map
The cost of road information collection vehicles and high-precision sensors, along with the processing computation and manual annotation costs brought by abundant elements, make the production cost of high-definition maps high.
In addition, high-definition maps require high timeliness, and the aforementioned map collection and production process needs to be repeated multiple times to maintain the freshness of the maps. The maintenance costs will also remain at a high level.
The high costs, complex processes, and limited freshness make high-definition maps a bottleneck for the further expansion and popularization of intelligent driving. It is precisely in this reality that high-definition maps gradually become the “crutch” that needs to be discarded in public opinion. Even so, the high-precision road semantic information still holds value for intelligent driving. Is it possible for vehicles to use their own “eyes” to see and understand the road semantic information in the three-dimensional world in real-time and reconstruct it online?
Strong Perception and Light Map
The perception ability of self driving vehicles mainly relies on vision-only algorithms based on the inertial navigation and stereo binocular technology. In order to make the original complex HD map “light”, the vehicle’s powerful perception ability can be used to reconstruct the drivable area (i.e., free space) in three dimensions, so as to generate online local “HD map” in real time. It does not rely on any prior information of the road, but on the scene “seen” by the vehicle itself, which provides reference for subsequent decision-making, planning and control.
This approach requires the implementation of two features similar to HD maps. They are environment perception prediction ability and local pose estimation ability. This enables the vehicle to know what is around it and where it is during self driving.
Environment Perception Prediction Ability
In terms of environment perception prediction ability, BEV (Brid’s Eye View) fusion algorithm is popular in the self driving algorithm framework. The main principle is to map the data of different sensors to the local vehicle coordinate system in the feature extraction stage, and then fuse them to perceive the surrounding environment of the vehicle through deep learning technology. The output environmental perception results are directly provided to the decision-making planning.
The processing flow of perception in the traditional algorithm mode
- Detect based on a single sensor output.
- Map the results detected in No.1 to 3D space.
- Do the correlation based on the lane detection of each sensor.
- Based on the correlation results and then based on the results of the previous time to do the fusion to get the final result of the current time.
However, in such an algorithm mode, the perception processing flow often faces the following challenges.
- For the mapping of the detection results on the image to the 3D space, small errors in the image detection of distant objects will cause large errors in the distance measurement in the 3D space.
- In the case of multi-sensor and multi-output, detecting the correlation between lane lines is also a big difficulty. If the correlation is wrong, the final output will deviate from the real world, and the output results will make the decision planning unusable.
- For multi-camera output correlation and time series fusion, more rule-based correlation/fusion strategies are needed, which leads to more complex correlation/fusion strategies and a sharp increase in the amount of code, which is not easy to maintain and iterate.
BEV algorithm
BEV algorithm can better solve the above problems. At the same time, because BEV algorithm uses deep learning neural network for feature fusion, it will be more concise in architecture. The BEV algorithm directly inputs the output results of all sensors into the deep learning neural network, and then converts the environmental features extracted by the neural network to the local vehicle coordinate system, and then directly outputs the 3D results in the local vehicle coordinate system through the deep learning neural network.
Since the environmental features output by the sensors are mapped to the local vehicle coordinate system, the road features can be aligned according to the position and orientation of the vehicle. The BEV algorithm is naturally friendly to temporal feature fusion, and the temporal detection effect will be more stable for occluded scenes. Therefore, under the BEV algorithm, the results can be predicted by converting the characteristics of the vehicle’s position and orientation at different times to the current vehicle coordinate system.
The BEV perception algorithm can not only predict rich semantic information (such as lane lines, ground signs, road edges, lane centerlines, diversion zones, road segmentation and other semantic information), but also predict topological information with higher semantic information (such as lane division and confluence, intersection topology, etc.). The online local “HD map” in the reachable space can be obtained by splicing and fusing these road semantic information according to the time sequence, which provides real time high precision road dynamic and static information for intelligent driving.
Based on the output of BEV perception algorithm, the vehicle can successfully complete basic driving assistance functions such as lane keeping on roads with missing and worn lane lines without relying on HD maps. It can also complete higher level intelligent driving functions such as on-ramp and on-ramp turns at urban intersections in high speeds, which greatly expand the use scenarios of intelligent driving.
The problem of environment perception prediction in the reachable space is solved, which can let the vehicle know “what is around me” in real time, and understand the dynamic and static information such as road topology, environmental characteristics and road users. However, it also needs to make stable and high-precision fusion positioning of the vehicle’s motion trajectory, that is, let the moving vehicle know “where I am”.
The local pose estimation (EgoMotion) capability is based on the inertial navigation stereoscopic binocular SLAM(Simultaneous Localization and Mapping) technology. It can maintain stable and high-precision vehicle trajectory estimation regardless of the congestion, rainstorm, night and underground parking scenes. At the same time, it also supports the expansion of other ambient monocular, fisheye and other cameras to further improve the performance.
The local pose estimation uses a multi-sensor fusion technology approach in order to take advantage of different sensors. However, in the actual mass production process, the installation position of each sensor is difficult to ensure accurately, and these positions will also be deformed over time. The overall sensor position and angle is a time-varying system. In order to solve these problems, in addition to outputting high-precision trajectory information of the vehicle, the local pose estimation will accurately model the external parameters of various sensors (that is, the position and angle of the sensor), fully considering their observability in various scenarios, and perform SLAM optimization together with the trajectory.
In the traditional local vehicle pose estimation, the input mainly includes Inertial Measurement Unit (IMU) and wheel speed. IMU mainly provides the yaw rate of the vehicle, and combined with the four wheel speed, the Ackerman steering model of the vehicle is constructed to calculate the vehicle trajectory.
However, the two types of sensors, wheel speed and IMU, are affected by wheel speed slip and IMU noise IMU temperature compensation, therefore their accuracy is limited. Although these two types of sensors have information complementary, the position, velocity, attitude and other information are obtained through the wheel speed integration, the acceleration and angular velocity measurement integration of the IMU, and the trajectory is calculated in the form of integration, so sensor noise and error will be accumulated in the final trajectory.
In order to avoid this situation, in addition to the input of the above two types of sensors, the vehicle trajectory is also online fused with the binocular depth estimation as the dominant visual information, that is, by using the image information, combined with the 3D vision principle, the relative position and attitude information of the previous and next frames can be directly obtained.
Summary
Use basic hardwares to implement core smart driving functions, rather than relying on expensive sensors or HD maps.
From the perception level, it is just like that human drivers can drive safely and stably by only relying on the limited information, such as signal signs, road features and lane space in the local spatial scene. However, for intelligent driving systems, it undoubtedly puts forward higher requirements for software algorithms and fusion technology.
Using low-cost sensor and the right computing power, the dynamic and static perception ability of BEV in driving and parking is realized. Based on the pure vision algorithm of inertial navigation stereo binocular, the information of binocular vision, inertial navigation and wheel odometer is fused to realize high-precision positioning of vehicles, and then the online local HD map technology is realized.
Online local HD map technology can replace the dependence on expensive sensors and high definition map, and can help car companies build intelligent driving functions with lower thresholds and better performance.
