In-Depth Analysis of Visual-Inertial SLAM

The visual-based SLAM scheme is prone to be affected by weak textures and drastic changes in lighting conditions, especially when the visual sensor is a monocular camera, there is also a problem of scale ambiguity. The IMU can perform high-frequency passive measurements independently of the environment, which not only increases the output frequency of the system’s pose, but also helps the monocular camera restore the scale. Therefore, the SLAM technology that integrates vision and IMU has also been widely studied. That is Visual-Inertial SLAM Technology.

1. The schemes of visual-inertial fusion SLAM

2. Detailed Explanation of Filtering and Optimization Methods

2.1 Filtering-based methods

1. MSCKF and its improved Variants
   • Visual-inertial fusion SLAM can be classified into two types based on the data fusion method: optimization-based and filtering-based methods. The Multi-State Constraint Kalman Filter (MSCKF) proposed by Mourikis et al. is a typical representative of the filtering-based method. The traditional visual SLAM scheme based on EKF adds feature points to the state variables. As the measurement environment increases, the dimension of the state variables will gradually increase, causing a significant increase in time consumption. MSCKF adds the camera pose to the state variables, uses the re-projection error of the feature points to construct the observation equation, and uses the feature points to constrain multiple camera poses and update the state variables. This approach effectively solves the problem of rapid increase in the dimension of the state variables and has high computational efficiency. However, this algorithm has inconsistent Jacobian points during the execution of the Jacobian calculation process, resulting in accuracy loss.
2. ROVIO Scheme
   • Bloesch et al. proposed the ROVIO scheme, which adds image block features to the state variables, uses block brightness errors as measurement residual terms for state update, and uses orthogonal triangular decomposition to reduce the scale of the least squares problem to accelerate the EKF solution speed ³.
3. Filter-Consistency Methods (e.g., RIEKF-VINS)
   • To solve the zero-space degradation caused by inconsistent linearization points, making the unobservable variables observable, Wu et al. proposed the filter consistency tightly coupled RIEKF-VINS ¹⁰. S-MSCKF, based on the MSCKF idea, introduces the extrinsic parameters of the camera and IMU into the state variables. Furthermore, the single-camera system was upgraded to a dual-camera system. By extracting key points and using the optical flow method for tracking, the robustness of the system was effectively improved without sacrificing efficiency ⁵.
4. Direct Method Fusion Based on IEKF
   • Bloesch et al. used the IEKF (Iterated Extended Kalman Filter) framework for the fusion of vision and IMU. Compared to the Extended Kalman Filter (EKF), IEKF has better convergence and numerical stability. Additionally, by using the direct method to construct the photometric error equation, it has higher computational efficiency ¹¹.

2.2 Optimization-based methods

1. The OKVIS Scheme
   • Leutenegger et al. proposed that OKVIS achieves dual-inertial odometry through an optimization method. This solution constructs the visual re-projection error and IMU constraints, optimizes using a sliding window approach of key frames, and controls the computational complexity through marginalization ¹². 
2. The ORB-SLAM and IMU Fusion Scheme
   • Raul et al. added an IMU sensor to the ORB-SLAM scheme framework, achieving an efficient monocular visual-inertial initialization process, restoring the scale of monocular vision, and adding loop detection based on the Bag of Words (Bow) model at the system backend to effectively reduce the cumulative error caused by long-term positioning of the visual odometry ⁴.
3. The VI-DSO Scheme
   • Stumberg et al. proposed VI-DSO, which uses the minimization of image pixel differences to construct visual constraints and builds IMU constraints based on the Euler angle pre-integration method for joint optimization to achieve pose state estimation and sparse scene reconstruction ⁶. After the VI-DSO initialization is completed, the scale and gravity vector are treated as variables and involved in the optimization process. Additionally, a dynamic marginalization strategy has been designed, which is one of the schemes for the direct method and IMU integration.

2.3 Advancements in Pre-integration Theory

1. Euler Angle-Based Pre-integration Method and Its Issues
   • Lupton et al. addressed the problem of inconsistent measurement frequencies of vision and IMU, which led to the post-processing fusion issue, and proposed an Euler angle-based pre-integration method. However, this scheme is prone to the problem of universal joint and has poor robustness ¹³.
2. Refinement of IMU Pre-integration Theory Using Lie Group Algebra
   • Forster et al. addressed the aforementioned universal joint deadlock problem and used the theory of Lie group algebra to conduct detailed derivations of the IMU pre-integration process from discrete time to continuous time, completing the pre-integration theory and improving the robustness of the pre-integration process ¹⁴.

2.4 Back-end Optimization and Global Consistency

1. The ICE-BA Scheme
   • In response to the current situation where the visual and IMU fusion based on optimization methods can only utilize a few measurements to construct constraints to achieve real-time performance, Liu et al. proposed the ICE-BA scheme ¹⁵. This scheme, when optimizing the system at the post-processing stage, proposed an incremental BA calculation method. Additionally, by ensuring the minimization of the re-projection error function and inertial constraint function during loop closure detection, it solved the global inconsistency problem of the visual SLAM system, making this scheme have good performance in terms of accuracy and computational efficiency.

2.5 Advances and Optimizations in Visual-Inertial SLAM Systems

1. Existing System Optimizations and New Proposals
   • Shen Shaojie’s team proposed that the VINS series is one of the most complete cases in the current visual-IMU fusion positioning systems. VINS-fusion adopts a pre-integration method based on quaternions, which can support single and dual vision-inertial fusion positioning. It uses the hand-eye calibration method to achieve online external parameter calibration and gravity alignment of vision and IMU, and then uses BA to fuse the information of the two ^{16 7 17}. The back-end of VINS-fusion is enhanced with a loop closure detection module, effectively avoiding the problem of pose drift in complex environments. Campos et al., based on ORB-SLAM and ORB-SLAM2, proposed a new ORB-SLAM3 system. This system not only improves the accuracy in pose estimation and map construction of single and dual vision and depth cameras, but also proposes a camera and inertial navigation fusion SLAM scheme with a relocalization module ⁹. To achieve more stable feature tracking at the front end, Minoda et al. proposed a VINS-Mask algorithm based on the Region of Interest (ROI) on the basis of VINS-Mono ¹⁸. They designed a point-of-interest-sensitive SuperPoint mask learning framework, obtained repetitive and reliable functional candidate regions, and significantly improved the overall pose estimation of the VIO system.
2. Feature Enhancement and Algorithm Adjustments
   • The Pl-VIO scheme proposed by He et al. extended the point features to include line features for the front-end extraction, and incorporated line feature constraints to participate in positioning optimization. This resulted in improved accuracy in indoor structured scenarios ¹⁹. Similarly, Fu et al. added line feature constraints to VINS-Mono and designed the Pl-VINS scheme ²⁰. This scheme adjusted hidden parameters and implemented length suppression strategies based on the line feature extraction method LSD. It significantly enhanced the efficiency of the algorithm, enabling Pl-VINS to run in real-time and achieving a considerable improvement in accuracy compared to VINS-Mono. Xu et al. aimed to improve the initialization accuracy of visual IMU fusion by using feature points-based SFM (Structure from Motion) in the initialization process of VINS-Mono, and initialized the acceleration deviation. After initialization, they switched to optical flow tracking for feature tracking, integrating the high accuracy of the feature point method and the better robustness of the optical flow method ²¹. 

3. Summary of Improvements and Remaining Challenges

In summary, visual-inertial SLAM has partially improved the accuracy and robustness compared to pure visual SLAM, but still has not completely solved the problems faced by visual SLAM.

1. Mourikis A I, Roumeliotis S I. A multi-state constraint Kalman filter for vision-aided inertial navigation[C]. Proceedings 2007 IEEE international conference on robotics and automation, 2007: 3565-3572.
2. Leutenegger S, Furgale P, Rabaud V, et al. Keyframe-based visual-inertial slam using nonlinear optimization[J]. Proceedings of Robotis Science and Systems (RSS) 2013, 2013.
3. Bloesch M, Omari S, Hutter M, et al. Robust visual inertial odometry using a direct EKF-based approach[C]. 2015 IEEE/RSJ international conference on intelligent robots and systems (IROS), 2015: 298-304.
4. Mur-Artal R, Tardós J D. Visual-inertial monocular SLAM with map reuse[J]. IEEE Robotics and Automation Letters, 2017, 2(2): 796-803.
5. Sun K, Mohta K, Pfrommer B, et al. Robust Stereo Visual Inertial Odometry for Fast Autonomous Flight[J]. IEEE Robotics and Automation Letters, 2018, 3(2): 965-972.
6. Von Stumberg L, Usenko V, Cremers D. Direct sparse visual-inertial odometry using dynamic marginalization[C]. 2018 IEEE International Conference on Robotics and Automation (ICRA), 2018: 2510-2517.
7. Qin T, Li P, Shen S. VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State<br>Estimator[J]. IEEE Transactions on Robotics, 2018, 34(4): 1004-1020.
8. Qin T, Cao S, Pan J, et al. A general optimization-based framework for global pose estimation with multiple sensors[J]. arXiv preprint arXiv:1901.03642, 2019
9. Campos C, Elvira R, Rodriguez J J G, et al. ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual–Inertial, and Multimap SLAM[J]. IEEE Transactions on Robotics, 2021, 37(6): 1874-1890.
10. Wu K, Zhang T, Su D, et al. An invariant-EKF VINS algorithm for improving consistency[C]. 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), 2017: 1578- 1585.
11. Bloesch M, Burri M, Omari S, et al. Iterated extended Kalman filter based visual-inertial odometry using direct photometric feedback[J]. The International Journal of Robotics Research, 2017, 36(10): 1053-1072.
12. Leutenegger S, Furgale P, Rabaud V, et al. Keyframe-based visual-inertial slam using nonlinear</li><li>optimization[J]. Proceedings of Robotis Science and Systems (RSS) 2013, 2013.
13. Lupton T, Sukkarieh S. Visual-inertial-aided navigation for high-dynamic motion in built environments without initial conditions[J]. IEEE Transactions on Robotics, 2011, 28(1): 61-76
14. Forster C, Carlone L, Dellaert F, et al. On-Manifold Preintegration for Real-Time Visual&#8211; Inertial Odometry[J]. IEEE Transactions on Robotics, 2017, 33(1): 1-21.
15. Liu H, Chen M, Zhang G, et al. Ice-ba: Incremental, consistent and efficient bundle adjustment for visual-inertial slam[C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 1974-1982.
16. Qin T, Cao S, Pan J, et al. A general optimization-based framework for global pose estimation</li><li>with multiple sensors[J]. arXiv preprint arXiv:1901.03642, 2019.
17. Yang Z, Shen S. Monocular Visual–Inertial State Estimation With Online Initialization and Camera–IMU Extrinsic Calibration[J]. IEEE Transactions on Automation Science and Engineering, 2017, 14(1): 39-51.
18. Sun J, Song F, Ji L. VINS-Mask: A ROI-mask Feature Tracker for Monocular Visual-inertial System[C]. 2022 International Conference on Automation, Robotics and Computer Engineering (ICARCE), 2022: 1-5.
19. He Y, Zhao J, Guo Y, et al. PL-VIO: Tightly-Coupled Monocular Visual–Inertial Odometry Using Point and Line Features[J]. Sensors, 2018, 18(4): 1159.
20. Fu Q, Wang J, Yu H, et al. PL-VINS: Real-time monocular visual-inertial SLAM with point and line features[J]. arXiv preprint arXiv:2009.07462, 2020.
21. Xu C, Liu Z, Li Z. Robust Visual-Inertial Navigation System for Low Precision Sensors under Indoor and Outdoor Environments[J]. Remote Sensing, 2021, 13(4): 772.