NIO's Big Model Journey: Self-developed Chips + World Model + Swarm Intelligence

Last weekend on July 27, NIO held the second NIO IN.

Li Bin said that the first NIO IN in 2023 is like an outline, which is the first time that NIO has fully displayed to the outside world the 12 major technology fields it has laid out.

This session is more like the first delivered chapter. It highlights the progress made in five phases:

• 
  

• Self-developed intelligent driving chip Shenji NX9031 was successfully taped out;

• 
  

• The vehicle operating system SkyOS is fully launched;

• 
  

• Banyan 3.0 will be launched this year, launching multiple NOMI agents;

• 
  

• The Intelligent Driving World Model (NWM) will start mass production in the fourth quarter;

• 
  

• The second generation NIO Phone was released.

Before NIO’s second and third brand models were launched, NIO’s years of investment in self-research finally began to show some results.

In the field of intelligent driving, the switch from modular intelligent driving solutions to end-to-end large models has become the most important technological trend this year. In the next week, Xiaopeng, Ideal, etc. will announce their respective progress in end-to-end intelligent driving, and NIO will be the first to make a move.

NIO CEO Li Bin and NIO Vice President of Intelligent Driving R&D Ren Shaoqing fully explained to us NIO's end-to-end thinking on intelligent driving.

In general, self-developed chips, world models, and swarm intelligence are the core framework of NIO's intelligent driving model.

1. NIO's self-developed high-computing intelligent driving chip, the Shenji NX9031

Shortly after the opening, Li Bin took out the Shenji NX9031 that had already been taped out from his pocket, and the audience burst into applause.

NIO said that this is "the world's first automotive-grade 5nm high-performance intelligent driving chip."

In fact, the Shenji NX9031 was taped out a few months ago, and after the team tested it, "the results were better than expected."

NIO has previously mentioned that the positioning of the Shenji NX9031 is "one chip for four OrinX chips". This time, they announced more parameters and features of the 9031:

• 32-core CPU, using big.LITTLE core architecture, CPU computing power reaches 615K DMIPS;

  
  

• NPU acceleration unit, Transformer algorithm performance increased by 6.5 times, LiDAR algorithm performance increased by 4 times, BEV algorithm performance increased by 4.3 times;

  
  

• The ISP can process images at up to 6.5G pixels per second.

  
  

• Sharing of computing power and integration of car and cloud.

At the event, NIO also showed off its self-developed ISP on Digi-Core to handle harsh lighting conditions.

The high processing bit width of the ISP on the 9031, combined with the noise reduction algorithm, can provide a very high dynamic processing range, making the image details richer in low light and making it easier to see signs and small objects hundreds of meters away.

A live comparison of image processing between the Shenji NX9031 and the industry's flagship intelligent driving chip, based on an 8-megapixel camera

What NIO mentioned about "one chip is equivalent to four" refers to at least a 4-fold performance improvement in key indicators such as AI computing power and ISP.

The first model officially announced to be equipped with the Dimensity NX9031 will be the ET9, which will be delivered next year.

However, Li Bin also mentioned that "(the chip) will not be effective until the first quarter of next year. Even if the chip is installed in the car, it will take time to implement the new architecture experience, and expectations must be managed well."

This may also mean that on the NT3 generation platform, NIO's self-developed Shenji and third-party computing chips may have to run in parallel for a period of time to achieve a smooth transition.

2. NWM world model allows intelligent driving systems to learn to imagine

The Shenji NX9031 was jointly defined by NIO's internal chip team and intelligent driving team over several years. One of its important features is that it is natively built for the world model.

The world model is the next major direction of NIO’s intelligent driving research and development.

NIO believes that a sufficiently smart intelligent entity should have the ability to imagine and reconstruct (spatial understanding) and imagine and deduce (temporal understanding), but an end-to-end model does not necessarily have these two core capabilities.

Therefore, it is not enough for intelligent driving to have an end-to-end model. The core is to build a world model.

The NIO World Model (NWM) was already being planned last year, but NIO IN was not yet mature at that time.

Currently, NWM has the ability to generate 2-minute videos, which exceeds most of the AIGC video generation software in the industry.

A 2-minute video can be used to predict driving scenarios that will occur in the next 2 minutes. In terms of trajectory planning, MWM can currently generate 216 possible trajectories every 0.1 seconds, and then regenerate 216 trajectories every 0.1 seconds based on environmental dynamics, allowing the system to select the best driving strategy.

Li Bin said that NWM has made rapid progress in the past few months, which is "leap-forward, and a few months ago it could only generate a few tens of seconds." 2 minutes means that even if you drive very slowly (30 kilometers per hour), you can cover all kinds of driving situations that will occur in the next 1 kilometer.

NWM is a multivariate autoregressive generative model. Shaoqing summarized that the differences between NWM and common end-to-end models include:

Spatial understanding ability: NWM reconstructs the generalized information of sensor input through generative models, while the end-to-end model has a single learning task and extracts information with loss;

Time understanding ability: NWM autoregressive model automatically models long time series environment, while the latter does not have the ability to model long time series;

Data requirements,NWM uses unlabeled data for self-supervised learning,,which relies on trajectory signals with low information density,perception annotations to assist training, which is costly and inefficient.

However, training an ideal world model is also very challenging: it requires more than tens of millions of clips of real data for training, the data must be rich, the timeline of imagination and reconstruction must be coherent, and there is also a lot of engineering work to be done.

The next step for NIO is to deploy NWM on the vehicle side. Shaoqing revealed that there will be a chance to "give everyone some experience" in Q4 this year.

To match NWM, NIO has developed a simulator called NSim (NIO Simulation). In the entire data link, the swarm intelligence on the vehicle side + NSim can theoretically provide NWM with a steady stream of data.

3. Swarm Intelligence: A Unique Verification Path for Intelligent Driving Systems

There are two challenges in fully modeling the intelligent driving system:

• First, the world model requires tens of millions of clips of real data. Where can we get this real data?

  
  

• Second, in the past, bugs in intelligent driving systems might only require locating the problem and verifying 1% of the modules, but now with the iteration of the model, the workload of testing and verification has skyrocketed.

  
  

In the past, NIO's "arrogant" use of four OrinXs was controversial in the industry. In the era of large models, the design of swarm intelligence seems to be beginning to play a more obvious role.

Take data collection as an example. Without a mass-produced fleet, the average intelligent driving company would only have a few hundred test vehicles, and the cost of owning a few hundred true value collection vehicles alone would be several hundred million yuan. NIO collects data through a mass-produced fleet, and currently has more than 200,000 models on the NT2 platform.

The swarm intelligence uses an OrinX design, which enables the vehicle to obtain valid data not only in intelligent driving state, but also in non-intelligent driving state.

NIO launched end-to-end AEB in Banyan 2.6.5, which collected 10,000 collision incidents from 2 billion kilometers of data. In fact, the total mileage of NIO users' intelligent driving announced on NIO IN is only 1.1 billion kilometers (although it is already the highest total mileage among all companies).

The core challenge of improving AEB capabilities is to significantly increase AEB scenario coverage without increasing AEB false triggering. Verification coverage is a difficult point in development. End-to-end AEB verification also uses swarm intelligence, with 400 million kilometers divided into 10 rounds of mileage verification.

Shaoqing said, "Swarm intelligence and generative models are killer features that can meet the needs of upstream training data."

In terms of testing and verification, swarm intelligence allows the new model version to be compared with the state of human driving and the past steady-state version, forming a feedback loop. The NIO Intelligent Driving Team told us that "the verification of swarm intelligence is very close to real-car verification and is much better than simulation verification."

Over the past four years, the architecture of intelligent systems has undergone tremendous changes every year.

Whether it is electronic and electrical architecture, software architecture or chip design, highly dynamic technological changes are always a huge challenge.

NIO has blazed a unique path through long-term investment.