Thinking about the development and testing of on-board ECU in autonomous driving

As the project progresses, I have recently started to undertake some test and verification work. Seeing our company's testing capabilities and processes, I couldn't help but sigh deeply.

0. Introduction

I think back to the last time I wrote about testing. It was because two engineers from NIO had an accident at work. I felt sad and sorry for my two colleagues. Time is like a rearview mirror when driving. It goes farther and farther, but it also reminds us of the direction of the road ahead. Since I entered the automotive industry, my job role has been constantly changing, from testing to system to architecture, but I can't avoid the word "testing".

Recently, I was inspired to say that the fate of system engineers is testing. This sentence may seem like a joke, but it is not groundless. Looking at the ASPICE development process, system requirements design and system testing are at the two ends of the V model, at the positions of ASPICE SYS.1 and SYS.5 respectively. From the development perspective, the person who proposes the function requirements must be responsible for the performance of the function, because only the proposer can know the expected product form most clearly. Therefore, every system engineer should have a set of testing skills.

Figure 1 ASPICE Overview (from ASPICE)

1. Test closed loop

Yes, the test discussed above is based on the system test of ECU. System test in this sense is generally performed by the supplier's system engineer when delivering the product. The software and hardware of ECU have been formed, and the supplier completes the test in a simulation environment or when the counterpart is online; or the system engineer of the OEM performs system-level test on the parts when accepting the product. In the actual R&D process, this kind of system test has actually reached the end stage of R&D, and is mostly used to verify the performance of the product under complex working conditions. When I was doing testing and system in the OEM, I was mainly responsible for this kind of system test.

So what does the complete test closed loop look like during the development of our in-vehicle ECU?

Starting from the software, engineers of each module should conduct self-tests on the module. For example, application modules such as perception, positioning, and regulation and control, as well as updates to middleware, need to be self-tested and reported before submission. After the module self-test passes, the leader of the software department is responsible for integration testing. A complete integration test should include functional level, performance level, basic communication, robustness, etc., which can be achieved through the pipeline. After each software release, you need to pay attention to the generated version number to facilitate troubleshooting.

The above tests are all run in a simulation environment. Next, the software version is flashed into the ECU board that has passed the DV/PV test to verify the performance of the software in the hardware. This link generally has the most problems, and it is difficult to locate whether it is a software problem or a hardware problem. It often requires the experience of engineers to analyze and solve it.

Of course, there are many automated test equipment on the market that can help us perform iterative testing. This type of test equipment supports some companies. Vector is a leader in this field. In recent years, Kunyi Electronics in China has also developed very well. The first HIL test bench I used was from Kunyi. It’s interesting to say that it was originally to test ECU, but it turned out to be "free" to help suppliers test their benches... Hehe, of course, that was a few years ago. Now domestic manufacturers are developing very well. Recently, I have also come into contact with Beihui's test equipment and training. They have undertaken many projects within the Geely system.

Finally, the ECU can be put into the whole vehicle environment for testing. The whole vehicle environment here can be a simulation environment composed of multiple counterparts connected to the test bench, or it can be a real vehicle environment. The real vehicle environment is more realistic and closer to customer use, but after all, the test environment on the vehicle is relatively poor and the work efficiency is relatively low. Generally, after multiple ECUs are installed together at the key node of mass production, professional test engineers from the OEM will conduct road tests, and then report the problems of each ECU, which will be handled by different responsible persons.

Figure 2 V model test process (Source: Beihui Information)

The above paragraph actually describes the MIL-SIL-HIL-VIL test chain. When I was in charge of the system at my old company, I also brought several fresh graduates to do ADAS product testing. I have repeatedly thought about how to do the test, what tests to do, what tools and environment we need, and how to achieve rapid iterative verification. Now I think it still depends on the current situation. Each company has different needs, different resources, different R&D goals, and naturally different test conditions that can be provided.

2. Testing in autonomous driving

In the field of autonomous driving, testing requires more professionalism and meticulousness, just like entering the "hardcore" mode. At present, one of our high-end models under development is equipped with 12*Camera, 3*Lidar, 5Radar, 12Uss, 1*high-precision positioning box, and connected to the domain controller. Think about it, there are so many sensors on our car, can you imagine the complexity of the test? But this is the charm of the challenge, right? In such a complex network topology, how to accurately, quickly, stably and comprehensively detect and locate problems requires continuous thinking.

Figure 3 Smart car sensors (from the Internet)

There are significant differences between testing in the field of autonomous driving and general software and hardware testing. These differences mainly stem from the complexity and diversity of autonomous driving systems and their special requirements for safety.

3. Think

When it came to our company, we redeveloped the algorithm. At this stage, we did not have a professional testing team or a complete testing process, and the closed-loop link of the testing problem needed to be improved. Often, the algorithm engineer could merge the problem into the main line after a simple verification on the car, while the problems reported by others could be closed only after local verification. These problems could easily lay the groundwork for subsequent development. Sometimes, the results of the actual car test and the simulation test were completely different.

System engineers are key players in project development. They not only need to have deep system knowledge and a deep understanding of architecture and design, but also a series of testing capabilities to ensure the quality and reliability of the system. As system engineers, we should stay calm, record, analyze, and track problems. The following are some suggestions based on our company's actual situation.

Test tools and automation: As the complexity of software and hardware increases, the efficiency and accuracy of manual testing will be challenged. Therefore, with the help of automated testing tools, such as Vector's CANoe, CANalyzer and other tools, the testing efficiency can be greatly improved. In addition, it is also crucial to write perfect automated test scripts and test cases.

Continuous Integration and Continuous Testing: By establishing a continuous integration and continuous testing (CI/CD) environment, you can ensure that the software can be quickly verified after each change, so that problems can be discovered and fixed as early as possible.

Fault injection and robustness testing: In order to ensure the stable operation of the ECU under various abnormal and fault conditions, fault injection testing can be performed to simulate various possible hardware failures, communication interruptions, etc.

Differences between simulated environment and actual environment: In a simulated environment, the test conditions are controllable, while the conditions of real vehicle testing are more complex and changeable. In order to bridge the gap between the two, you can consider using more realistic simulation tools, such as IPG CarMaker.

Documentation and tracking: It is crucial to record the test process, test results and issue tracking. Consider using an issue tracking tool such as JIRA, and clearly define the conditions and responsibilities for closing issues to ensure that each issue is properly handled.

Team training and knowledge sharing: Regularly organizing technical sharing and training within the team can ensure that team members keep their understanding of testing methods and tools up to date.

Safety and regulatory considerations: As the global automotive industry places increasingly stringent requirements on functional safety and related regulations, standards such as ISO 26262 have put forward stricter requirements for the testing of in-vehicle software. Considering these factors will make the testing more rigorous, but also more challenging. For example, the NOP function needs to consider the regulatory requirements of R79, while functions such as ACC/LCC have long had complete regulatory definitions.