Implementing vehicle communication testing based on fuzzy testing method

Fuzz testing is a type of testing method that is bound to be mentioned in the field of network security testing. It has extremely distinctive characteristics, including extremely low demand dependence, reversible testing concepts, etc., which are so "different in style" from conventional testing methods. But at the same time, this testing method can often find some core and serious BUGs, so it has an important position in the entire testing system, and this testing method has also been "silently" integrated into our testing activities. This time, I will introduce to you how to use fuzz testing in the field of vehicle communications.

1

What is fuzz testing?

The overall testing idea of ​​fuzz testing is very simple. It is to input "random values" into the system to be tested and then monitor whether any abnormalities occur.

Figure 1 Schematic diagram of fuzz testing This process has the following characteristics:

1

/ Random value

Random values ​​can be normal data or unexpected data, and are usually obtained through mutation (tampering with the original data) and generation (forging data through rules). The entire test is not to find a few random values ​​for testing, but to continuously input a large number of random values ​​(which can be understood as endless) to perform long-term continuous testing.

Figure 2 Comparison between functional testing and fuzz testing

2

/ Monitoring abnormalities

Because the system under test may receive and process unexpected data, it may have various behaviors. Abnormal behaviors may include service shutdown, program crash, resource overload, and data loss.

3

/ Automation

Each test round is to monitor anomalies after given input. For fuzz testing, such rounds need to be executed a large number of times, even endlessly, so automation is a basic requirement.

2

Fuzz testing applied to vehicle communications

Although fuzz testing is rarely used in vehicle communication testing, we actually use this method in many tests. For example:

High load test in CAN communication test

Inject specific messages into CAN communication to form the expected load conditions, and then monitor whether there are any anomalies. The random value variables in this test are the load rate and the message ID that generates the load. The judgment of anomalies may include error frames, communication recovery, etc., which forms a high-load fuzz test.

Sleep and wake-up test in network management

The wake-up event is triggered at each moment when the controller is in sleep mode, and then the normal wake-up is monitored. The random value variable in this test is the timing, that is, when the wake-up event is triggered, and the abnormal judgment is the wake-up behavior, which forms a sleep-wake-up timing fuzzy test.

The two examples given here are quite special. What we want to illustrate is that the random values ​​we mentioned are not just data that are easily associated with protocol fields or signal values, but also include various state values ​​(such as load rate, throughput, frequency, etc.), timing (such as sleep wake-up process, three-way handshake of TCP connection, unlocking interaction of diagnostic service, etc.) and other data. Especially the timing, which usually has a series of steps. In this process, not only can the interval time of each step be used as a random value, but also each step itself can be "made a fuss", such as coupling a random signal value to each step.
In summary, everything can be digitized, and data can be randomized. This may be the core idea of ​​fuzz testing. Combined with the characteristics of in-vehicle communication itself, we can get some typical fuzz testing content:

Fuzz testing of data formats in protocols

For example, when receiving different IDs and DLCs in the CAN bus, whether an error frame appears. This test may be more targeted at CANFD nodes or OBD nodes, and may even include special data such as extended frames and remote frames. Another example is when receiving different source MACs, target MACs, source IPs, target IPs, source ports, and target ports in Ethernet, whether an exception occurs. The exceptions here will vary depending on the firewall policy, such as no response, restart, and sending unreachable error messages.

Figure 3: Fuzz testing example for the data format in the protocol - CAN ID fuzz testing

Fuzz testing for signal values

For example, when different signal values ​​(including illegal values, invalid values, etc.) are received in the CAN bus, whether restart or freeze occurs. Another example is when various requests, responses, subscriptions and other data (including unexpected data types, encodings, byte sequences, etc.) are received in SOME/IP, whether service suspension, service crash, freeze occurs, etc.

Figure 4: Fuzz testing example for signal values ​​- CAN signal fuzz testing

Fuzz testing for timing

For example, the network management sleep wake-up test mentioned above. Another example is that in SecOC or time synchronization mechanism, a specific benchmark needs to be synchronized periodically, and other data will use the synchronized benchmark. If various offsets occur in the synchronization timing (including the offset of the benchmark itself, the offset between the benchmark used by the application data and the benchmark update time, etc.), it is necessary to determine whether the synchronization mechanism has collapsed.

Figure 5: Timing-based fuzz testing example - sleep-wake fuzz testing

Fuzz testing of traffic

For example, in the high-load test mentioned above, or in the performance test of the gateway node, different data streams are received in parallel at each port to form different throughputs to determine whether the routing mechanism has collapsed.

Figure 6: Traffic fuzzing example - throughput fuzzing

3

Issues that need to be paid attention to in fuzz testing

Although there are a large number of tests that can use fuzz testing methods, fuzz testing also has its limitations or shortcomings. How to maximize its advantages and avoid its shortcomings may require consideration of the following:

abnormal

The judgment of abnormalities is a very difficult but very important point. The difficulty lies in the fact that the abnormal situations are relatively complex and the means of monitoring are often limited, especially the need to realize automatic monitoring of abnormal situations. In the vehicle network, the abnormalities can be judged including specific signal values, periodic fluctuations of periodic messages, current values ​​of controllers, DTC, etc. For each different fuzzy test, it should be necessary to select the appropriate abnormal judgment method according to the test content. Another effective method is to perform "insertion" to break through the limitations of the controller itself and the limitations of conventional "black box testing".

Figure 7 Schematic diagram of pile insertion

balance

Since fuzz testing is a method that approaches infinite testing, the inevitable impact is the increase in time cost. However, our actual output does not increase linearly with time, so the test efficiency will become lower and lower. In order to improve the test efficiency, we must use some rules to screen random values, but this will inevitably lead to a decrease in randomness. As described in many testing theories, "testing can prove the existence of defects, but it cannot prove the absence of defects." Many of our tests are to seek a balance between input and output. Fuzz testing is more obvious due to its characteristics, so that we have to define whether the test is completed. There will be some disputes and games. To deal with such a balance, the difficulty may be greater than the test itself.

Positioning and repair

Testing will find problems, which brings another problem: how to locate them. The characteristics of fuzz testing will make it very difficult and uncertain to locate the problem (including problem reproduction), especially when the random values ​​in fuzz testing are more dimensional, things will become more complicated. Another point to consider is the retest after the problem is fixed, which may be more troublesome than doing a complete fuzz test. At this time, the points that need to be solved are how to quickly verify that the problem has been fixed and how to verify whether the repair plan will cause new problems. Although these two points exist in all tests, they are more difficult due to the randomness of fuzz testing. Based on the above considerations, I personally think that for current in-vehicle communications, fuzz testing needs to be regular and targeted in the early and middle stages of development. Although we have reduced the scope of problem discovery, we have effectively improved the test efficiency, problem repair efficiency, and resource utilization. As the software matures, the original rules can be gradually relaxed to increase randomness. In this process, it is also necessary to make adjustments based on your own situation. In short, the key point is actually two words: balance.

4

Summarize

Fuzz testing is a very effective testing method. Its randomness allows us to find problems that are difficult to think of through logic. This expands the scope and means of our problem discovery and can effectively improve testing experience, which is similar to scenario testing. But at the same time, its high investment and difficult repair problems are also issues that cannot be ignored. How to achieve a balance is a point we must consider. In general, this is a multiple-choice question of whether to exchange time for experience or experience for time. In addition, the in-vehicle communication test in this article is a broad concept, which also includes tests such as safe communication, safe diagnosis, and safe flashing using in-vehicle communication technology as a carrier.