aiSim5 expands simulation testing scope based on generative AI (final)

In the previous chapters, we discussed the confidence of aiSim simulation synthetic data. In addition, in the process of scene reconstruction and closed-loop testing, we will inevitably face problems such as time-consuming and high-cost 3D scene production and reconstruction, low scalability, and unsatisfactory traffic conditions. The current main challenge is how to automatically generate 3D static scenes and add dynamic instance editing, thereby effectively shortening the test process and expanding the scope of simulation testing.

Figure 1: Actual image

Figure 2: NeRF reconstruction scene

For 3D reconstruction, the two main solutions are NeRF and 3DGS.

1. NeRF

1. Neural Radiance Fields

NeRF encodes the color and density information of each point in three-dimensional space into a continuous function and parameterizes it by MLP. Given a viewpoint and a point in three-dimensional space, NeRF can predict the color of the point and the density distribution along the line of sight. By performing volume rendering on this information, NeRF can synthesize images from new viewpoints.

2. Advantages

High-fidelity output.

• Based on NerFStudio, a more friendly code library is provided.

• Relatively fast training time.

• It is scalable for areas to be rebuilt.

3. Shortcomings and main challenges

Slow rendering. NeRF needs to perform a lot of sampling and calculations along each ray from the camera to the scene to accurately estimate the volume density and color of the scene. This process is computationally intensive, and it takes about 10 seconds to render an image at full HD resolution on an NVIDIA A100.

The scene depth estimation effect is not ideal. NeRF implicitly learns the depth information of the scene through volume rendering, but this depth information is usually coupled with the color and density information of the scene. This means that if there are complex situations such as occlusion or non-Lambertian reflection in the scene, NeRF may have difficulty accurately estimating the depth of each pixel.

Reconstruction quality of close objects may be low. This may be caused by insufficient view angle and resolution, inaccurate depth estimation, and motion blur occlusion.

Ghosting artifacts caused by imperfect calibration of high FOV cameras.

Of course, in order to solve these problems, researchers introduced depth regularization to improve the accuracy and stability of NeRF depth estimation, and improved the rendering speed by optimizing the structure and algorithm of NeRF.

2.3DGS

1. 3D Gaussian Splatting

3DGS uses three-dimensional Gaussian distribution to represent point cloud data in the scene. Each point is described by a Gaussian function with mean and covariance. The Gaussian function is rendered by rasterization to generate realistic 3D scene images.

2. Advantages

The training time is short.

Near real-time rendering.

Provides high-fidelity output.

3. Shortcomings and main challenges

The code base is less friendly. Compared with NeRFStudio, the documentation is less complete and easier to use.

The initial point cloud acquisition has high requirements, requiring precise sensors and complex data processing procedures, otherwise it will have a significant impact on the performance of 3DGS.

Depth estimation is also insufficient, which may be due to several reasons: the optimization process tends to optimize each Gaussian point independently, resulting in overfitting in a small number of images; the lack of global geometric information leads to inaccurate depth estimation in large scenes or when reconstructing complex geometric structures; the depth information of the initial point cloud is not accurate enough, etc.

Camera model support is limited. Currently, 3DGS mainly supports the pinhole camera model. Although 3DGS versions of other camera models can be derived in theory, subsequent experiments are needed to verify their effectiveness and accuracy.

The scalability of the reconstruction area is limited, mainly due to the incomplete reconstruction caused by the lack of geometric information outside the LiDAR coverage area and the large amount of computation required to reconstruct large urban scenes.

Integration and resource-intensive challenges,Currently 3DGS integration usually relies on Python interfaces; 3DGS may occupy a large amount of VRAM when running.

Optimizing hyperparameters and adopting new methods, such as Scaffold-GS, may help reduce memory requirements and improve processing capabilities on large scenes.

3. Operation method

1. Training process

Step 1: Input - Camera video data; Vehicle motion data; Calibration data; LiDAR point cloud data for depth regularization;

Step 2: Remove dynamic objects: Create segmentation maps to identify and mask different objects and regions in the image; automatically annotate dynamic objects* (Kangmo aiData toolchain);

Step 3: Perform NeRF or Gaussian splatting.

NeRF:

Any camera model can be used, the example uses the MEI camera model;

Large-scale reconstruction using Block-NeRF;

Embedded in different climate conditions.

Gaussian splatting:

Convert the input camera into a pinhole camera model;

The initial point cloud can be obtained from COLMAP or LiDAR;

Large-scale reconstruction using Block-Splatting.

2. Add dynamic objects

After NeRF and 3DGS generate static scenes, aiSim5 will further add dynamic elements based on the external rendering API, which can not only reconstruct the original scene but also construct different traffic conditions according to test requirements.

NeRF/3DGS-based scene details in aiSim5.

Figure 13: Mesh casting shadows

Figure 14: Ambient occlusion under the car

3. Effect display

After adding dynamic objects in aiSim5, you can freely change the traffic status in the map scene for SiL/HiL testing of perception/regulation systems.

Figure 15: aiSim5 running NeRF city scene 1

Figure 16: aiSim5 running NeRF city scene 2