Debugging Methods of Embedded Systems

    The article introduces that the embedded system development process is actually a debugging and diagnosis process, and debugging and diagnosis will always accompany a product throughout its life. Even the most mature products will occasionally have problems of one kind or another, which require developers to diagnose and troubleshoot.

    Embedded system debugging includes hardware debugging, software debugging and comprehensive debugging. Hardware debugging generally refers to the inspection before and after the system is powered on when it is just developed, including:

    1) Before powering on, check whether the power supply and ground are short-circuited, and visually check whether there are cold solder joints or leaking solder joints;

    2) After power-on, check whether the frequency, waveform and amplitude of the clock line are normal, whether the power supply voltages are stable and normal, whether the temperatures of each chip are normal, and whether the indicator lights are normal.

    Software debugging generally refers to verifying whether the timing of program execution is correct, whether the logic and results are consistent with the design requirements, and whether the functional and performance requirements can be met while ensuring that the hardware is normal. There are many methods for software debugging, including:

    1) Use indicator lights to track and debug;

    2) Use serial port to print and debug;

    3) Use a simple debugger to debug at the assembly code level;

    4) Use a relatively high-end debugger for source code level debugging;

    5) Use an emulator to perform hardware simulation.

    The above simple hardware debugging or software debugging is relatively simple, and the difficult part is comprehensive debugging. Below I will first give some difficult problems I have encountered in my work, and then summarize some general debugging methods and precautions from them.

    Example 1: The debugging of the PPC860 (Motorola) network engine platform designed and manufactured by us is nearing completion. All four boards produced in the same batch have passed all software tests, so we went to solder the second batch. However, the FEC of one board in the second batch did not work properly. Several of our software and hardware engineers used various means and re-read the chip manual many times, but still could not find the cause. So we sent the board back to the factory to re- solder the BGA , but the problem still existed. We had no choice but to treat this board as a sample and solder all the chips on the board. According to common sense, this approach is very logical, because the components are the same and the boards are also from the same batch, so it may be that the welding of a certain part of this board is not good, but it is difficult to check, and repeated re-soldering may damage the motherboard. Later, someone found out from the characters on the PPC860 chip that one had to open their eyes wide to see clearly with a magnifying glass (it is said that China's first-generation domestic high-end processor chip "Hanxin" was made by a "scientist" who ground off these letters on the same type of chip as "Moto" and engraved "Hanxin No. 1" on it!!!) that the CPU version number of this board was "D4", while the CPU version number of other boards was "D3", but there was no comparison between the two versions in the chip manual. The errata manual of PPC860 was found on the Internet, and it was discovered that in the PPC860TZP50D4 version, a bit called FEC_ PIN _MUX (bit2) was added to the ECNTRL register to control the multiplexing function of the FEC pins. If FEC is to be used, this bit must be set to 1, so the statement line ECNTRL |= 0x00000004 must be added to the FEC-related program.

    Example 2: When I debugged the amateur homemade MC68VZ328 board, the circuit board hardware check was fine, and using Code warrior to burn the compiled uClinux program into the flash through the serial port was also normal. However, when I powered on again, I found that there was no data in the serial port. I checked with a multimeter (I didn't have "advanced equipment" such as an oscilloscope at that time) and found no results. Then I put this board in my bag every day when I went to and from get off work, and I took it out to look at it with wide eyes. I couldn't help but feel angry when I looked at it, but one day I found that a capacitor was welded on a place marked with a resistor symbol. When I got home, I replaced the resistor and powered on again. The serial port printed out the startup information of uClinux. Oh, that taste was sweeter than drinking honey. Of course, it was because I didn't have much experience at that time. If this problem was put now, it would be estimated that it would be solved within a day. In addition, when I first debugged the homemade S3C4510 development board, I couldn't output characters from the serial port. It took me half a day to find that the polarity of the capacitor on the sixth foot of the serial port level conversion chip max3232cse was welded in reverse.

    Example 3: When debugging the SB1250 embedded server motherboard, since DDR1 generation memory sticks were used, there were quite a lot of decoupling capacitors and resistors in series and parallel on the data and clock lines. Almost none of the first batch of soldered boards could successfully enter the CFE (BIOS) menu interface. The clock waveform and power supply were both very good compared with the borrowed DEMO board. I re-tinned all the decoupling resistors and capacitors around the DDR DIM slot on the motherboard with a soldering iron. Hehe, it really worked. This method has been tried and tested, and it is also very useful when debugging the PCI and HT buses later. It may be because the SB1250 system is a high-frequency circuit with relatively high requirements for welding. Even the slightest leakage or cold soldering will not work. I think so.

    From the above examples, we can summarize the following debugging methods and precautions:

    1) Deepen understanding: Deepen understanding includes deepening understanding of hardware and software. Deepening understanding of hardware mainly involves reading relevant chip data manuals in detail, and deepening understanding of software is because not all programs need to be written by ourselves when developing embedded systems. Many are already prepared and directly obtained from the Internet or purchased, but these software are not necessarily completely targeted at our own target boards, so some problems are often found during use, especially in the underlying software. Once a problem occurs, the developer must first understand the code where the problem occurs. Only on the basis of a deep understanding of the relevant hardware and software can we make more realistic judgments and better solve the problem.

    2) Comparison method: There are many comparison methods, such as running the same software on two similar but different hardware platforms to compare the phenomena; running two different versions of software on the same hardware platform to compare the phenomena; running the same software on two different hardware platforms from the same batch to compare the phenomena. For some not very hidden problems, the comparison method can usually get good results.

    3) Decomposition method: When you encounter a problem that is complex to analyze and may have many factors, you can divide the problem into several small problems to test and diagnose. For example, you can write several separate small test programs to check and test various possible factors, and then make scientific judgments based on these test results.

    4) Combination of software and hardware: This method requires a certain amount of inspiration and understanding. For example, in Example 5 above, during the test, you can temporarily change the state of the hardware without damaging the hardware (such as short-circuiting the data line and the clock line in this example) to see if the problem phenomenon will change. If so, do more similar tests to find out the change rules and key factors, and then analyze and solve them. In the development of low-level software, special attention should be paid to the software programming of hardware modules with strict timing requirements. Once there is a problem with the timing of the program, and this part of the software has been integrated with other system software, it will be difficult for others to find the problem when checking this software.

    5) Diagnosis and troubleshooting should be based on a large number of experiments. We should do more hands-on work. We should not just imagine and be unwilling to actually operate, and call it "good at thinking and analysis". Embedded system development is a very practical science. We need to summarize the objective laws of things in practice, so as to better understand and use them and make them work better according to our intentions.

    6) Embedded system development and debugging requires developers to have a rigorous and meticulous work attitude and never miss any clues found during the debugging process, because it may be the key to open Pandora's box.

    7) We must have a pragmatic work style and have the spirit and courage to doubt everything. We should respect authority and the scientific and technological achievements of our predecessors, but when contradictions arise, we should believe in the experimental results even more, so that science can progress.

    8) Be brave enough to challenge yourself, put aside habitual thinking and prejudices, broaden your thinking, and analyze problems from multiple angles.

    9) Embedded system development, especially low-level software and operating system kernel development, is a relatively difficult and challenging job because it requires dealing with both software and hardware. Even if we do everything very well and consider everything very carefully, the target system may still play some small jokes on us. We often encounter a very small problem that bothers us for days or even weeks, during which we may lose our appetite and sleep at night. Therefore, embedded system low-level software developers must not only have a peaceful mind, but also have a certain amount of patience and perseverance, as well as the courage and confidence to overcome all difficulties! As long as we do well enough, the process of solving a specific problem may be somewhat accidental, but we will eventually eliminate all obstacles!

    Therefore, the embedded system debugging process is a process of gaining a deeper understanding of our target system and the characteristics of each unit module in the system, a process of exercising our logical thinking and analytical reasoning ability, a process of developing new ideas, challenging habitual thinking and authority, a process of cultivating a rigorous and meticulous work attitude and a pragmatic work style, a process of exercising our endurance and perseverance, and ultimately a process of learning and progress!

    Improving the debugging and diagnostic capabilities of embedded systems is a long-term process of practice, accumulation, and improvement!