Three things engineers must do when troubleshooting

Recently, my students frequently get stuck: seemingly simple designs cannot be adjusted no matter what, and they are almost going crazy. About a week ago, when Xiao Chen came to see me, he looked like he was about to let go and jump off the cliff because he couldn't hold on to the branch. He was so anxious that he wanted to say something unpleasant to me. In the past two days, Xiao Chen found his problem, solved it, and became happy again. But Chang Wei's problem came again.

　　Using the MSP430F169 microcontroller to implement SPI control for the programmable gain amplifier PGA280 works fine, and the same microcontroller to implement control for a 24-bit ADS1259 works fine too. But when both are soldered together and controlled separately using the CS chip select, it doesn't work. The problem is so simple, but it makes him very anxious.

　　It is only a matter of time before they solve the problem. I am not worried. I am happy that they can find and solve the problem. However, I found that they all fell into a state of confusion without exception: when a problem appeared, they would try left and right, sometimes they succeeded and were happy, but when they came back from dinner, they failed again and tried again. They kept going back and forth like this, and when they broke down, they would start to get angry and anxious, and then look for help everywhere, just like looking for straw after drowning. If this state lasted for more than 3 days, they would start to get angry with me.

　　I told them: Failures are a good thing. First, they train you. Second, they eliminate hidden dangers. I will not intervene unless it is absolutely necessary. I only need to teach them the three musts for troubleshooting.

　　Troubleshooting is a profound subject. But generally speaking, there are three things that are necessary. With these three things, there is no fault that cannot be solved: First, mentality. Second, strategy. Third, patience.

　　The first mentality. You must have a strong sense of gratitude for the failure. Thank God for giving me this opportunity, I must seize it firmly. You can imagine yourself as Sherlock Holmes. You haven't received any work for several months. It's boring to chat with Watson every day. You are eager to have a case. Scotland Yard is helpless and waiting for you to take action. Only this mentality can enable you to maintain an excited mind, careful thinking and sufficient patience in the subsequent long-term struggle.

　　This is my greatest characteristic. When students report a problem to me, I am usually very excited and listen to every word, just like listening to a test question. They are careless, but my eyes are as sharp as a knife. I especially hope that my students can learn this.

　　The second strategy. This is a technical job. There are too many details to explain, so I will just briefly talk about it.

　　1) Allow the failure to recur and avoid random failures. I will talk about random failures when I have the chance.

　　2) Protect the fault site and do not tamper with it easily. Whether it is software or hardware, it should be guaranteed to be recoverable. Therefore, do not solder off the chip at will, and the soldered chip should be placed properly so that it can be retrieved. In addition, the software must be backed up according to the serial number.

　　3) Don't make more than two changes at a time.

　　4) Develop a habit of recording all actions and facts in a small notebook. Changing a power supply may seem like a small matter, but it may turn fault A into fault B, and your mind will be confused. Therefore, if you need to change the power supply, you should also record it.

　　5) Pay attention to instruments and operating methods. Every time you record facts, make sure they are true.

　　6) Learn to use logical thinking. Mainly, how many possibilities are there to cause this fault phenomenon, list them one by one, from the most likely to the least likely.

　　7) Learn the troubleshooting order. There are two main factors that affect the troubleshooting order: the first is the possibility of failure, and the second is the difficulty of troubleshooting. Of course, we should first try the most likely and least difficult to troubleshoot. But the two are not always so coincidental. For example, you suspect that chip A is broken, which is the most likely. But it is very difficult to solder it off, and the troubleshooting is difficult to implement, so you can first check other possibilities. This item has a bit of luck and a bit of experience.

　　8) Learn the dichotomy and use it skillfully. The dichotomy is to divide the fault into two parts (or three parts, not too many, otherwise it will be chaotic), and then create some situations to find out which part it is, and then divide it again to gradually narrow the encirclement. In the past, the Japanese devils used this method to find out where the telegraph was sent in the city: one area after another had a power outage, and saw where the power outage caused the telegraph signal to disappear, so they could determine which area the sender was in, and then narrow the area of ​​power outage, and finally find our underground workers. It's still not clear in a few words, I'll find a chance to talk about it.

　　When you list all the possible faults according to this meticulous way of thinking and find no results after checking them one by one, you should be even more excited. Just like adding a 1V DC voltage to a 1k resistor, but the measured current is not 1mA, you should have this mentality: What the hell, does Ohm's law not hold true?

　　At this point, you can find a teacher, a friend, or anyone else. But who can do this? Most students completely collapse at this stage.
Therefore, the third rule is to have enough patience.

　　I once encountered a fault, that is, the power supply current was too high. There were a lot of chips densely packed together, and they were working normally, but the power supply indicated that the current was too high. I knew there must be a short circuit or a short circuit nearby. But how to check? The key is that the whole system is working normally.

　　At that time, I warned myself to be patient. I was different from ordinary people. I had enough patience and I would definitely find it. So, I first used a magnifying glass to check all the positions on the board to see if there was a short circuit or solder splashing. It took a long time and I can't remember it. Then, I looked at the PCB diagram on the computer and checked all the spacings around 10mil with a multimeter, but still couldn't find it. I was a little annoyed, so I warned myself again that I was not an ordinary person, not an ordinary person, and continued.

　　What did I do this time? No one would have thought how determined I was: I planned to measure every pin of the chip, or every node in the circuit board, and every node that should not be connected. How many times would it take? But I was determined. So I started to do it. Fortunately, the chips at that time were all DIP packages, and there were not many pins. I checked them one by one and finally found them. In fact, it didn't take too long, only one or two hours.

　　The result was that two completely unrelated output lines were short-circuited, and the resistance was not 0, I can't remember how much it was, it was about a few ohms. I looked left and right, they were not connected, there was only a section of about a few centimeters where the two lines ran parallel. I cut it, and it was no longer short-circuited, the short circuit was limited to about 10cm, cut it again, and finally it was limited to a space of about 1cm, the two lines were parallel, but short-circuited. The distance between the two lines was about 3mm.

　　I have messed up this board, but it left me with an eternal question: two wires with a distance of 3mm shorted within a length of 1cm. I looked at it with a magnifying glass, but there was no trace. I held it up to the glaring desk lamp and looked carefully, and a thin trace appeared, so thin, so twisty, and there was an opaque thin line.

　　I raised the cutting knife and made several deep cuts at 3mm intervals, and the short circuit disappeared.

　　This is a story from the mid-1990s and I still remember it vividly.

　　Without my patience, such a fault would be difficult to find.

　　Some people may say that this is an isolated case. The quality of the printed circuit board is not good. What's the point of finding it? The board is already scrapped. But I have a different view. If I find it, I can pat my chest. Self-confidence is more important than anything else. I still have enough confidence. When students encounter problems, I am not anxious. I will check them slowly. I will never believe that I can't find the problem after my deadline.

　　A good attitude, a careful strategy, and enough extraordinary patience are the three musts for troubleshooting. Many people may focus on technical strategies, which are of course very important. But I found that the most important ones are the first and third ones. They are not something you can learn if you want to, but something you need to understand and cultivate.