Online Spatial-Temporal Calibration Between Monocular Camera and IMU for Visual-Inertial Navigation Systems

1. Why Visual-Inertial Fusion Requires Precise Calibration

Inertial components are unaffected by the environment and require no external input signals. They autonomously output the real-time acceleration and angular velocity of the rigidly mounted carrier. Integrating acceleration once yields velocity, integrating velocity yields position, integrating angular velocity measurements yields attitude. Based on this principle, inertial components enable self-contained inertial navigation and positioning. By virtue of their sensor properties, inertial systems offer high short-term accuracy, strong real-time performance, and excellent autonomy.

A single sensor, constrained by its own hardware limitations, typically cannot satisfy the requirements of long-endurance, high-precision, and robust navigation and positioning. Multi-sensor integrated navigation has therefore become the primary approach in the field. Integrated navigation fully leverages each sensor’s strengths, allowing homogeneous or heterogeneous sensors to compensate for one another’s measurement deficiencies through collaborative operation, thereby reducing the limitations inherent in any single sensor.

2. Why Monocular Vision + IMU Is a Practical Combination

Low cost and compact size, a monocular vision system only needs one camera module as its sensor input and is widely used in autonomous driving, pedestrian monitoring, and object recognition. The monocular visual positioning system is based on the epipolar geometry constraint between the successive images and the relative position of the carrier is determined. Despite its inherent scale ambiguity problem, a monocular visual positioning system has received wide attention due to its many applications and low operational constraints and is appropriate for position estimation and attitude computation.

Vision sensors provide rich environmental texture and structural information with a strong scene understanding capability and at the same time, provide the capability for navigation, localization, place recognition and object detection. However, pure visual navigation fails to provide good stability in case of fast scene changes and strong carrier vibration. While not affected by environment, inertial navigation is susceptible to drift error, and thus to a degradation in long term positioning accuracy, but can be used to estimate position and attitude in real-time. Fusing visual image information with the acceleration and angular velocity output of inertial components therefore effectively mitigates the drawbacks of each single-sensor navigation system, enhancing the autonomy and reliability of navigation and positioning for unmanned systems.

3. The Calibration Problem — Why Plug-and-Play Matters

The key prerequisite for multi-source sensor fusion technology is the parameterization between heterogeneous sensors. In visual-inertial integrated navigation system, there are two categories of calibration methods: offline calibration and online calibration.

Offline calibration, which uses a target object with a known geometric shape to estimate the extrinsic parameters between sensors, prior to the equipment being placed in service. In the case of offline calibration, the user is normally required to move the rigid sensor assembly slowly and uniformly in front of a stationary target (usually a chessboard or fiducial marker) and repeat the entire calibration process several times until convergence. Although offline calibration has matured considerably and can achieve relatively high-accuracy intrinsic and extrinsic parameter calibration, it requires repeated calibration runs, and if the sensor position shifts, recalibration must be performed from scratch — a time-consuming process that significantly restricts the practical deployment scenarios of visual-inertial integrated navigation systems.

4. IMU Measurement Model and Pre-Integration Fundamentals

4.1 The IMU Measurement Model

The IMU sensor provides real-time vector data on three-axis angular velocity and acceleration of the carrier it is attached to. In most cases both accelerometer measurements and gyroscope measurements are corrupted by gaussian white noise, and also have slowly varying biases. The IMU model can be written as a mixture of the truth, a slowly varying bias, and white noise:

\[ \begin{aligned} \text{Gyroscope measurement:} \quad &\tilde{\omega}(t) = \omega(t) + b_g(t) + \eta_g(t) \\ \text{Accelerometer measurement:} \quad &\tilde{a}(t) = R^T(t)(a(t) – g) + b_a(t) + \eta_a(t) \end{aligned} \]Where \( (\tilde{\cdot}) \) denotes the IMU measurement value, \( (\cdot) \) represents the true sensor output without noise, \( b_g \) and \( b_a \) are the slowly varying gyroscope and accelerometer biases, and \( \eta_g \) and \( \eta_a \) represent Gaussian white noise.

4.2 Why Pre-Integration Is Necessary

The IMU pre-integration measurement model combines measurements taken by all IMU sensors over a period between two keyframes into a single measurement value, unifying the operating frequency of the IMU and other sensors (e.g. cameras). The integration process needs to be done in advance, in order to match data rates of IMUs with those of cameras for optimizing the data together, as the frequency of IMSUs are generally in the range of 100-1000 Hz, and the frequency of the cameras are generally in the range of 20-60 Hz.

The pre-integration measurement model for IMU can be obtained by defining the relative motion increments, which are independent of the starting time instant. The pre-integrated rotation, velocity and position measurements are relative changes between keyframes and are finally represented as “plus” random noise:

\[ \begin{aligned} \Delta\tilde{R}_{ij} &= R_i^T R_j \quad &\text{(rotation pre-integration)} \\ \Delta\tilde{v}_{ij} &= R_i^T (v_j – v_i – g\Delta t_{ij}) + \delta v_{ij} \quad &\text{(velocity pre-integration)} \\ \Delta\tilde{p}_{ij} &= R_i^T (p_j – p_i – v_i\Delta t_{ij} – \tfrac{1}{2}g\Delta t_{ij}^2) + \delta p_{ij} \quad &\text{(position pre-integration)} \end{aligned} \]

This formulation enables efficient re-linearization in the optimization procedure, while avoiding the need to re-compute the integration whenever linearization points are changed.

5. Analytical Dual-Stage Initialization Method

5.1Parameter Estimation as Maximum-a-Posteriori

Based on the IMU measurement model and the initial camera pose, the parameter estimation problem can be formulated as a Maximum-a-Posteriori (MAP) probability problem. All variables to be estimated are stacked into a state vector \( \chi \):\[ \chi = \{\, s,\ g_w,\ b_a,\ b_g,\ q_c^b,\ t_c^b \,\} \]Where: – \( s \in \mathbb{R} \): visual scale factor – \( g_w \): gravity direction in the world frame – \( b_a \in \mathbb{R}^3 \) and \( b_g \in \mathbb{R}^3 \): accelerometer and gyroscope biases – \( q_c^b \) and \( t_c^b \): rotation (quaternion) and translation from the body (IMU) frame to the camera frame, collectively called the camera-IMU extrinsic parameters

The MAP estimation problem is then:

\[ \chi^* = \operatorname{argmax}\ p(\chi \mid Z) \]

Where Z represents the set of IMU and image observations. Using Bayes’ rule, the posterior is expressed as the sum of a prior residual and observation residuals, each weighted by their respective uncertainties.

5.2 Fast Analytical Initialization in Milliseconds

During the system initialization phase, IMU biases are assumed to be constant. The gyroscope bias estimation equation can be obtained by making use of the relationship of the rotational transformation between the consecutive keyframes. The bias of the gyroscope is estimated by minimizing the rotation residual between all pairs of key frames:

\[ b_g^* = \operatorname{argmin} \sum \left\| \Delta\tilde{R}_{ij}\, \operatorname{Exp}\!\left(J_{\Delta R}^g\, \delta b_g\right) \right\|^2 \]In the first iteration, \( b_g \in \mathbb{R}^3 \) is initialized as a zero vector. After each iteration, the bias is updated via \( b_g^* = b_g + \delta b_g \). Once the gyroscope bias is successfully estimated, the IMU pre-integration values are immediately updated.Based on the estimated gyroscope bias, a relative rotation equation can be formulated to estimate the extrinsic rotation \( q_c^b \) between the IMU and camera:\[ q_c^b = \operatorname{argmin} \sum \left\| q_{b_i}^{b_j} \otimes q_c^b – q_c^b \otimes q_{c_i}^{c_j} \right\|^2 \]

This analytical method initializes very quickly, in the millisec range which is critical to systems with plug-and-play needs.

A parameter optimization method under noise perturbation is used to solve the accuracy issue caused by the limited accuracy of closed-form solutions. The parameters are then refined by use of a noise perturbation model, while simultaneously estimating the accelerometer bias, after obtaining coarse initial estimates by use of the analytical method. This two stage strategy combines analytic coarse initialisation with a perturbation approach to refinement to provide fast convergence while avoiding the accuracy limits of pure closed-form solutions.

6. Online Spatiotemporal Refinement Calibration

6.1 The Time Synchronization Problem

However, when deployed in the field, the time reference of the IMU sensors and the camera are not aligned due to the hardware trigger delay, transmission delay and clock asynchronization between the two sensors, resulting in a time-offset error. This time difference when not compensated directly impacts positioning accuracy of the integrated navigation system.

6.2 Online Time Delay Modeling

In order to solve the inconsistency of measurement time due to the time-offset errors between the sensors, an online time delay model is built. The time delay parameter \( t_d \) is introduced as a state variable to be estimated within the optimization framework:\[ t_{\text{cam}} = t_{\text{imu}} + t_d \]

The time delay between the camera and the IMU measurement is modeled and estimated along with the rest of the calibration parameters. This approach helps to compensate the information of heterogeneous sensors in real-time online without any requirement for hardware synchronization.

6.3 Global Spatiotemporal Optimization

A new spatiotemporal calibration and parameter refinement method is proposed, which applies to the whole globe. The 6-DOF extrinsic transformation between IMU and camera is combined with the inter-sensor time delay, which are jointly optimized using an iterative nonlinear optimization method. This is a full time delay compensation for heterogeneous sensors.

The integration of fast initialization and accurate spatiotemporal calibration error compensation is the basis of high accuracy integrated navigation system. The positioning accuracy is enhanced by about 15% compared to the traditional iterative and analytical calibration algorithms.

7. Practical Implications for Engineers

7.1 When to Use Online Calibration

The online spatiotemporal calibration is especially useful in the following cases:

1. Rapid deployment: If a visual-inertial system is needed immediately after the mounting of the sensors, without the possibility to perform offline calibration steps.
2. Sensor position changes: If the position of the camera and the IMU will vary relative to each other during operation (e.g., due to vibration, thermal expansion or mechanical adjustment).
3. Multi-sensor fleets: If a large number of the same visual-inertial sensors are to be calibrated and offline calibration isn’t feasible for each sensor.
4. Time-critical applications: In high dynamic maneuvering situations, positioning drift of micro-second order or even less is unacceptable.

7.2 Key Performance Factors

Several factors affect the success of online spatiotemporal calibration:

• Sufficient excitation: Enough rotational and translational motion of the sensor platform is needed to observe all the calibration parameters. The observability is poor in straight-line constant velocity motion: the system requires multi-axis rotation to separate out IMU biases from the extrinsic parameters.
• IMU noise characteristics: Lower-noise IMUs (IMUs with a lower Angle Random Walk and Bias Instability specification) will converge to accurate calibration parameters quicker and more reliably. The pre-integration noise covariance \( \Sigma_{ij} \) directly depends on the IMU’s noise density and bias instability values, meaning that selecting an IMU with well-characterized noise parameters is critical for calibration quality.
  
• Initialization quality: The analytical initialization stage needs to give a relatively good approximation as a coarse-sketch of the global optimum before the iterative refinement stage can converge to the global optimum. Inadequate initialization can cause the optimizer to get stuck in local minimas.
• Environmental texture: Enough visual details in the environment are still needed since the calibration process involves estimating the pose of the environment correctly to constrain the parameters of the IMU. If the textured environment is lacking in some areas, then more types of features (lines, planes) may be required to keep the image from moving while the sequence is being conducted.

Summary

For the visual-inertial odometry and SLAM systems, the spatial and temporal calibration between the camera and the IMU is critical. Chessboard targets are used for offline calibration, which is a well developed technique but impractical for many deployment situations. The analytical dual-stage initialization approach is capable of a millisecond initialization of IMU biases, extrinsic parameters and scale factor. The system, when used in combination with spatiotemporal refinement on-line, which simultaneously estimates the 6-DOF extrinsic transformation and the time delay between the two sensors, is able to perform in plug and play with positioning accuracy increased by about 15% compared to traditional methods. The correct online calibration of DAISCH IMUs is crucial for maximizing the achievable accuracy from the sensor fusion architecture for engineers who are working on integrating DAISCH IMUs into monocular visual-inertial navigation systems.