Implementing ISO21434 safety testing based on CANoe and TESTstudio

Vector provides a professional open development platform for OEMs and suppliers in the automotive and related industries. In the field of security, it provides a variety of security development and testing tools, security software components and security consulting services, which are applied to the creation of embedded systems. The basic protocol station plays a key role in the network security deployment of MICROSAR.

Fan Kefa, Business Development Manager of Victor Automotive Technology (Shanghai) Co., Ltd., gave a speech on "One-stop Solution for Vehicle-side Cybersecurity". The following is a summary of the speech:

Fan Kefa, Business Development Manager, Victor Automotive Technology (Shanghai) Co., Ltd.

Vehicle-side safety testing under ISO 21434

Today's topic mainly focuses on in-vehicle secure communication and the application of corresponding testing technologies. For example, Ethernet, which has a seven-layer architecture, needs to communicate between each layer. Each layer will have potential network attack points, which requires us to deal with various intrusion methods accordingly. Specifically, the automotive industry has given a clear answer, and there are corresponding technical means at each layer to ensure that communication can be carried out safely, such as SecOC, TLS, IPsec and MACsec.

My colleague has just introduced the security solutions provided by Victor in the design and implementation phase. I will now talk about the security testing on the test side. Later, I will mention how to decrypt the encryption technology used by the customer from development to debugging. The ISO 21434 security specification system has some methodologies, which are somewhat different from the functional safety ISO 26262 test specifications, but most of them are similar.

Vehicle-side security testing mainly includes functional testing involving security, which mainly tests the correct behavior of security-related functions and the robustness of the system. It is a test of product security (mechanism), followed by vulnerability scanning (used to test known vulnerabilities), fuzz testing (trying to discover new vulnerabilities by sending invalid, undefined or random inputs to the target system, which is the preferred technology for finding vulnerabilities from a hacker's perspective) and penetration testing (highly personalized and creative testing of SW+HW systems).

Application of CANoe in SIL/HIL network security testing

Victor continues to deepen its research in the field of testing and builds a complete testing solution for the introduction of network security into vehicle systems. Generally speaking, suppliers will be exposed to a large number of different OEM encryption and decryption project requirements, so the selected tools also need to support different OEM encryption and decryption systems to connect the subsequent production, testing, DV, PV and other stages. It is far from enough for the tool to simply capture data, but also to convert the data into recognizable messages, and then serve the development, testing and production links.

From the tool level, in recent years, OEMs may use keys or certificates to encrypt and decrypt communications. The encryption and decryption algorithms used and the encryption and decryption technologies used at different levels are different, especially in the HPC and regional controller fields. Therefore, suppliers need to spend a lot of time in engineering to understand different encryption and decryption technologies in advance, and finally debug and test them in the development environment, which is a very troublesome thing.

In addition, the status of security, freshness, certificates, etc. are different in the three stages of ECU development, mass production delivery, and after-sales maintenance. This brings up a question: Can the tool system deployed by suppliers on the production line support applications at different stages? CANoe, launched by Victor as an old tool, also hopes to better meet engineering applications in the field of network security.

Image credit: Victor

So how to deal with the above problems? Victor provides a systematic security management engineering solution, including HSM, Secure Boot, SecOC, Secure Diagnostics, etc. The security management solution (Security Manager) can provide a consistent interface for all Victor tools and the source of encryption materials, reducing the security complexity of nominal function testers. It is a highly integrated solution (no need to adapt to the existing tool chain).

Image credit: Victor

In addition, Victor CANoe can provide the required services for specialized security plug-ins. What our department mainly does is to present encrypted messages as plain text, support SecOC communications of the AUTOSAR standard and specific variants required by OEMs, realize message encryption and fresh value management, and provide corresponding support for Ethernet communication security.

vTESTstudio: A powerful tool for automotive network security fuzzy testing

The next step is fuzz testing. Through CANoe, the encrypted messages on the communication test system can be converted into plain text, so that subsequent tests can be carried out according to the steps of traditional tests. After completing the tests of security functions and security communication modules, it is necessary to do corresponding fuzz testing. Fuzz testing is mainly divided into two aspects: the first is to fuzzify the signal application layer, and the other is to fuzzify based on communication and messages.

The core concept in fuzz testing is: what should be monitored during fuzz testing, and what problems do you hope to find? This is still not clearly defined in the industry. It may be to test applications or ECUs. Different levels and different engineers focus on different things.

Next, let's talk about what fuzz testing is. Fuzz testing consists of three parts: the test environment, the fuzz test case generator, and the monitoring data flow. Easy-to-use test tools are needed to simulate various attacks, that is, automatically generate fuzz test cases, so as to provide high returns for the test system. Fuzzers used for fuzz testing are divided into two categories: mutation-based fuzzers, which create test cases by mutating existing data samples; generation-based fuzzers, which use the protocol or file modeling used by the system under test, generate inputs based on the model and create test cases accordingly.

In the above, the most important part of the fuzz test definition is actually the fuzz tester. Simply put, as long as the tester modifies the fuzz tester in the tool system and can generate the corresponding random numbers according to the algorithm rules and give them to the ECU, it can be defined as a fuzz test, but the data required by the generator is different.

Victor's solution reuses the automated script development tool vTESTstudio for HIL and SIL, embeds the fuzz test engine used for fuzz testing, and configures the corresponding test data in it, mainly the message level of Ethernet and CAN communication, the signal level of Ethernet and CAN transmission, the I/O or A2L signal level, etc. Taking the three signals defined in DBC as an example, these three signals are configured in the tool through the fuzz engine in vTEST studio, and each signal will generate fuzz test data according to the configured rules. When these three signals are generated, the signals will be configured and combined according to the Sequential, Pairwise, and Combinatorial rules. After the combination, the tester will input these data into the test environment and ECU.

The most difficult part of fuzz testing is that the test requires monitoring of observation items. For example, a set of data may be given to the ECU, which makes the CPU usage of the corresponding task very high. If the observer wants to monitor the CPU usage of the data input, what should he do? There is no doubt that a measurement function must be set in the board to monitor the CPU usage within a certain period of time. This requires using the debug port of the controller itself to pass its CPU usage to CANoe during the test.

In addition to being able to do fuzzy at the message level, Victor can also do fuzzy at the I/O level, because CANoe itself is a test tool for the HIL environment. CANoe, combined with vTESTstudio, provides embedded system development engineers with similar attack injections from the attacker's perspective, and generates fuzzy test cases by reusing existing configuration engineering data. CANoe's powerful functions help strengthen the fuzz tester environment in the automotive electronics industry: it can adapt to different OEMs, different network protocols and databases; it supports data recording, data playback, and provides detailed automated test reports.

Image credit: Victor

In addition, the dynamic test solution can cover both the real ECU in HIL and the virtual ECU in SIL. Victor can provide both, but it is necessary to insert the stub in advance in the real board test and get the data through the debug port. Every test rate given by Victor in the tool can get the coverage during the execution process.

Next, let's talk about the advantages of using CANoe for fuzz testing. In short, testers can directly use the CANoe project developed by the R&D department, embed the fuzz tester on it, and then directly perform the corresponding fuzz testing.

Usually, fuzz testing engines are based on "black box" testing technology. However, in the automotive electronics industry, reusing existing configuration engineering (such as network communication protocols and controller logic state functions) and tools to develop verification systems can achieve "white box" testing technology. In addition, when software functions change, rapid iteration of verification can be achieved in conjunction with a continuous integration environment.

Because of the existence of CANoe, Victor will use gray-box fuzz testing more often. Black box is to do fuzz testing on communication messages. It uses existing databases and its internal signals and variables to directly manipulate the signals transmitted in the messages, so as to obtain results more efficiently. This is because the gray-box fuzz testing efficiency is better than the "brute force" black box method and can better determine the problems in the system.