In-Depth Analysis of Visual-LiDAR-IMU SLAM

In order to ensure that the SLAM algorithm can work effectively in scenarios with poor lighting conditions or degraded structures, integrating the LiDAR, camera, and IMU is a very good solution. The advantages of the several sensors complement each other. In environments with poor lighting and texture, LiDAR information can provide effective supplementation, and in degraded environments such as long corridors, camera visual information can provide effective supplementation. This article reviews the SLAM schemes based on multi-source fusion of camera visual and LiDAR and other sensors from two aspects: loose coupling and tight coupling.

1. Overview of Multi-Sensor Visual-LiDAR-IMU SLAM

The following table summarizes the current camera visual/IMU/Lidar fusion SLAM schemes. The DEMO scheme proposed by Zhang et al. uses the point cloud depth values from the LiDAR to provide depth information for the visual feature points, enabling higher-precision pose estimation and higher-quality point cloud maps ¹. Zhang et al. based on LOAM, proposed the V-LOAM (Visual-LiDAR odometry and mapping) scheme, which utilizes the high-frequency pose calculation results from vision to provide pose prior information for LiDAR point cloud scanning matching, and uses the low-frequency radar matching to further optimize the pose results. However, the algorithm execution process is frame-by-frame and lacks global consistency ². 

1.1 Addressing Limitations in Loosely-Coupled Schemes

To address this issue, Wang et al. optimized the global pose graph by maintaining a keyframe database, thereby improving global consistency ³. To overcome the motion degradation problem, Khattak et al. proposed another loosely-coupled method similar to LOAM, which uses visual-inertial priors for LiDAR point cloud scanning matching and can operate in tunnels ⁴.

1.2 Advancements in Fusion Techniques

Zhang et al. integrated IMU data based on the V-LOAM scheme, used high-frequency IMU measurements for motion prediction, updated the motion through visual-inertial odometry, and finally refined the pose through frame-to-map registration, achieving a process from high frequency to low frequency, and from low precision to high precision.The LIV-LAM scheme proposed by Radmanesh et al. uses unsupervised learning to discover and utilize target detection features, marking a milestone in the field of visual LiDAR fusion positioning for machine learning ⁵. Camurri et al. proposed the Pronto scheme for legged robots, which uses visual-inertial odometry to provide motion prior information for LiDAR odometry and can correct the pose between vision and LiDAR ⁶. Graeter et al. proposed a multi-source fusion SLAM scheme based on BA optimization method. This algorithm reprojects the depth information measured by the LiDAR into the image space and associates it with visual features, thereby maintaining accurate scale information ⁷. To cope with challenging environments, Shao et al. proposed a binocular/IMU/LiDAR fusion VIL-SLAM scheme. This scheme treats the visual-inertial odometry and LiDAR odometry as separate subsystems, using the high-frequency visual-inertial odometry to assist in de-distorting the LiDAR point cloud, and also incorporates visual-based loop closure detection to construct loop closure constraint estimation ⁸.

2. Camera visual/inertial/LiDAR fusion SLAM Schemes

3. Tightly-Coupled Fusion Approaches

To enhance the robustness of the SLAM system, researchers have explored the tightly-coupled approach of combining LiDAR, Camera, and IMU. Many scholars have performed joint state optimization for the three using MSCKF. Zuo et al. proposed LIC-Fusion, which utilizes the MSCKF framework to perform tightly-coupled operations on the edge features of LiDAR point clouds, IMU constraints, and visual features ⁹. In its subsequent work, LIC-Fusion2.0 introduced a plane feature tracking method based on sliding windows to handle the 3D point cloud of the LiDAR ¹⁰.

3.1 Factor Graph Optimization Frameworks

Zhao et al. proposed the Super Odometry scheme, which implemented a processing flow centered on the IMU, integrating the advantages of loose coupling and tight coupling. It utilized visual-inertial odometry and LiDAR-inertial odometry to constrain the high-frequency poses of the IMU, achieving a positioning mode from coarse to fine. Additionally, it adopted a point cloud organization method based on dynamic octrees, further improving the computational efficiency ¹⁶.

By integrating VINS-Mono ¹⁷ and LIO-SAM ¹⁸, the LVI-SAM (LiDAR-visual-inertial smoothing and mapping) proposed by Shan et al. consists of a visual-inertial subsystem and a LiDAR-inertial subsystem. Based on factor graph optimization, the coupling of the two subsystems is achieved. Unlike Super Odometry, in the LVI-SAM scheme, the depth information of visual features is obtained by projecting from the LiDAR point cloud, thereby achieving robust and high-precision state estimation and mapping ¹⁴. Wisth et al. proposed the tightly coupled LiDAR-visual-inertial system (VILENS), which uses the factor graph optimization framework to jointly optimize three sensors and directly extract the line and plane features in the LiDAR point cloud, achieving the purpose of real-time processing of LiDAR point cloud data ¹⁹.

3.2 Combining Filtering and Optimization Methods

The R2LIVE solution, open-sourced by the MARS Laboratory of the University of Hong Kong, combines filtering and optimization methods. This solution utilizes the error state iterative Kalman filter to fuse the IMU pose prediction information and the visual and LiDAR measurement information, achieving efficient pose output. To further enhance the measurement accuracy, factor graph optimization is used to further optimize the odometer results ¹². R3LIVE is based on R2LIVE and consists of two subsystems: the LiDAR Inertial Odometry (LIO) subsystem and the Visual Inertial Odometry (VIO) subsystem. The LIO subsystem (FAST-LIO) uses the measurements from the LiDAR and inertial sensors to construct the geometric structure of the map (i.e., the positions of 3D points). The VIO subsystem uses the data from the visual inertial sensors to texture-render the point cloud map (i.e., the colors of 3D points) ¹³. 

3.3 Evolution towards Higher Efficiency

This team also achieved very good results based on FAST-LIO2 and SVO2.0’s open-source FAST-LIVO ¹⁵, which compared to the R2LIVE scheme, eliminates the extraction of visual and LiDAR feature points, and the point cloud map constructed by LIO is supplemented with image blocks, which are solved for pose using the minimization of image photometric errors in VIO, thus having higher computational efficiency.

4. Further Developments and Conclusion

The Lvio Fusion scheme proposed by Jia et al. is based on graph optimization for integrating information from stereo cameras, LiDAR, IMU, and GNSS. It uses a lightweight deep reinforcement learning method to adjust the weights of each factor, achieving a high-precision and highly robust SLAM framework ²⁰. In addition to the common integration among LiDAR, cameras and inertial navigation systems, in some specific fields, it is also necessary to integrate with other types of sensors, such as millimeter-wave radar, geomagnetic sensors, UWB, etc. Currently, multi-source fusion SLAM based on tight coupling has gradually become the mainstream, and the problem of spatial-temporal calibration of multiple sensors has also received extensive research. Moreover, with the rapid development of deep learning technology, SLAM integrating deep learning is also an important direction for current and future development.

4.1 Conclusion and Future Directions

In conclusion, the robustness of visual-based SLAM solutions in dynamic scenes, low texture environments, and environments with drastic changes in lighting conditions, as well as the problem of large-scale dense mapping, still need to be further addressed. The SLAM schemes based on LiDAR do not have geometric elements, the same environment, and symmetrical buildings, and thus suffer from motion degradation and insufficient loop closure detection. The SLAM solution based on the fusion of vision/LiDAR/IMU can effectively alleviate the drawbacks of visual or LiDAR-based SLAM, but the research on its fusion is not yet sufficient.

1. Zhang J, Kaess M, Singh S. Real-time depth enhanced monocular odometry[C]. 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2014: 4973-4980.
2. Zhang J, Singh S. Visual-lidar odometry and mapping: Low-drift, robust, and fast[C]. 2015 IEEE International Conference on Robotics and Automation (ICRA), 2015: 2174-2181.
3. Wang Z, Zhang J, Chen S, et al. Robust high accuracy visual-inertial-laser slam system[C]. 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2019: 6636-6641.
4. Khattak S, Nguyen H, Mascarich F, et al. Complementary multi–modal sensor fusion for resilient robot pose estimation in subterranean environments[C]. 2020 International Conference on Unmanned Aircraft Systems (ICUAS), 2020: 1024-1029.
5. Radmanesh R, Wang Z, Chipade V S, et al. LIV-LAM: LiDAR and visual localization and mapping[C]. 2020 American Control Conference (ACC), 2020: 659-664.
6. Wisth D, Camurri M, Das S, et al. Unified multi-modal landmark tracking for tightly coupled lidar-visual-inertial odometry[J]. IEEE Robotics and Automation Letters, 2021, 6(2): 1004-1011.
7. Graeter J, Wilczynski A, Lauer M. Limo: Lidar-monocular visual odometry[C]. 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS), 2018: 7872-7879.
8. Shao W, Vijayarangan S, Li C, et al. Stereo visual inertial lidar simultaneous localization and mapping[C]. 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2019: 370-377.
9. Zuo X, Geneva P, Lee W, et al. Lic-fusion: Lidar-inertial-camera odometry[C]. 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2019: 5848- 5854.
10. Zuo X, Yang Y, Geneva P, et al. Lic-fusion 2.0: Lidar-inertial-camera odometry with slidingwindow plane-feature tracking[C]. 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020: 5112-5119.
11. Wisth D, Camurri M, Das S, et al. Unified multi-modal landmark tracking for tightly coupled<br>lidar-visual-inertial odometry[J]. IEEE Robotics and Automation Letters, 2021, 6(2): 1004-1011.
12. Lin J, Zheng C, Xu W, et al. R $^ 2$ LIVE: A Robust, Real-Time, LiDAR-Inertial-Visual Tightly-Coupled State Estimator and Mapping[J]. IEEE Robotics and Automation Letters, 2021, 6(4): 7469-7476.
13. Lin J, Zhang F. R 3 LIVE: A Robust, Real-time, RGB-colored, LiDAR-Inertial-Visual tightlycoupled state Estimation and mapping package[C]. 2022 International Conference on Robotics and Automation (ICRA), 2022: 10672-10678.
14. Shan T, Englot B, Ratti C, et al. LVI-SAM: Tightly-coupled Lidar-Visual-Inertial Odometry via Smoothing and Mapping[C]. 2021 IEEE International Conference on Robotics and Automation (ICRA), 2021: 5692-5698.
15. Zheng C, Zhu Q, Xu W, et al. FAST-LIVO: Fast and tightly-coupled sparse-direct LiDARinertial-visual odometry[C]. 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2022: 4003-4009.
16. Zhao S, Zhang H, Wang P, et al. Super odometry: IMU-centric LiDAR-visual-inertial estimator for challenging environments[C]. 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2021: 8729-8736.
17. Qin T, Li P, Shen S. VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator[J]. IEEE Transactions on Robotics, 2018, 34(4): 1004-1020.
18. Shan T, Englot B, Meyers D, et al. LIO-SAM: Tightly-coupled Lidar Inertial Odometry via Smoothing and Mapping[C]. 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020: 5135-5142.
19. Wisth D, Camurri M, Fallon M. VILENS: Visual, inertial, lidar, and leg odometry for allterrain legged robots[J]. IEEE Transactions on Robotics, 2022, 39(1): 309-326.
20. Jia Y, Luo H, Zhao F, et al. Lvio-Fusion: A Self-adaptive Multi-sensor Fusion SLAM Framework Using Actor-critic Method[C]. 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2021: 286-293.