CEVA Helps Customers Efficiently Implement Sensor Fusion with SensPro Sensor Hub DSP

It is well known that sensors work best when used together. This is especially true for simultaneous localization and mapping (SLAM). SLAM plays an important role in the AR/VR space, adjusting the scene based on the user's posture and avoiding collisions during use in applications such as drones or robots. The SLAM market is expected to grow to $465 million by 2023, with a compound annual growth rate of 36%, providing a great opportunity for most developers. Applying SLAM to indoor navigation on mobile phones may dominate the large-scale basic platform market. GPS does not work indoors, and beacon-based navigation only works in areas with beacon infrastructure. SLAM, on the other hand, works anywhere an indoor map is available, which meets the low-cost expectations of most building automation management systems. The scene of the area is fused with the posture and movement of the user walking in the area, making the application of SLAM here a great example of applying sensor fusion.

Out-of-the-box solutions

I will first describe how to integrate and test this solution based on our CEVA SensPro Sensor Hub DSP hardware, combined with our SLAM and MotionEngine software modules to regulate and manage motion input. We need a camera and inertial measurement sensors, a CPU and DSP. We use the CPU to host the MotionEngine and SLAM framework, and use the DSP to execute the SLAM algorithm to reduce the task burden.

For a simpler explanation, I will first start with the OrbSLAM algorithm, a widely used open source implementation. It will perform three main functions. Tracking does the (visual) frame-by-frame registration and localizes the new frame on the current map. Mapping adds points to the map and does local optimization by creating and solving a complex set of linear equations. Loop closure does global optimization by making corrections on returning to points that have been visited before. This is achieved by solving a large set of linear equations.

Some of these functions can be run very efficiently within the host application on a CPU core, along with control and management functions that are specific to your application. Some functions must run in a DSP processor to be practical or to gain a competitive advantage. For example, tracking might manage 1 frame per second (fps) in a CPU, with feature extraction accounting for 40% of the algorithm's runtime. In contrast, a DSP implementation might manage 30 fps, a resolution that is important for fine-grained correlation between video and the IMU.

The reason for this advantage is easy to understand. The DSP implementation provides very high parallel processing capabilities, fixed-point/floating-point support, which is critical in tracing and solving linear equations. In addition, there is a special instruction set to accelerate feature extraction. A simple link between the host and the DSP allows the DSP to be used as an accelerator, offloading intensive calculations to SensPro.

Vision Fusion IMU

We provide two key components: visual SLAM using the CEVA-SLAM SDK product and CEVA MotionEngine software, which can process IMU inputs for three of the six degrees of freedom very accurately. The fusion of IMU and video information depends on an iterative algorithm that is usually customized to the application requirements. The last step links the visual data with the motion data to generate accurate positioning and mapping estimates. CEVA provides mature visual SLAM and IMU MotionEngine software as a solid foundation for developing fusion algorithms. Algorithm-intensive functions such as this will run fastest on a DSP, such as our SensPro2 platform.

Testing the prototype

Once you have built your prototype platform, how will you test it? There are multiple SLAM datasets available. Kitti is one, EuroC is another. In the example below, I show the accuracy comparison between an OpenCV implementation and our CEVA-SLAM SDK implementation. You will want to do a similar analysis for your product.

Incorporate your own approach

As I mentioned earlier, there are many ways to build a SLAM platform. Maybe you don’t want to start with OrbSLAM, or you want to integrate your own algorithms or differentiated algorithms. SensPro Sensor Hub DSP supports both.