This review mainly explains the commonly used kernel debugging methods. Because deadlock detection and memory detection are inevitable for programmers who develop kernels and drivers, this review will focus on the kernel debugging methods mentioned in this book.
1. What is deadlock?
Deadlock refers to the phenomenon that two or more processes are waiting for each other due to competition for resources. For example, process A needs resource X, and process B needs resource Y, but both parties have the resources needed by the other party and neither releases them, which will lead to deadlock. In kernel development, concurrent design must always be considered. Even if the correct programming ideas are used, deadlock is inevitable. In the Linux kernel, there are two common deadlocks:
- Recursive deadlock: If a lock is used in a delayed operation such as an interrupt, it will cause a recursive deadlock with an external lock.
- AB-BA deadlock; multiple locks cause deadlock due to improper processing, and inconsistent lock processing order on multiple kernel paths can also cause deadlock.
The following is an example of a deadlock program. The function tries to acquire the lock again when it has already acquired the lock, causing a deadlock.
#include <linux/init.h>
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/kthread.h>
#include <linux/freezer.h>
#include <linux/delay.h>
static DEFINE_SPINLOCK(hack_spinA);
static struct page *page;
static struct task_struct *lock_thread;
static int nest_lock(void)
{
int order = 5;
spin_lock(&hack_spinA);
page = alloc_pages(GFP_KERNEL, order);
if (!page) {
printk("cannot alloc pages\n");
return -ENOMEM;
}
spin_lock(&hack_spinA);
msleep(10);
__free_pages(page, order);
spin_unlock(&hack_spinA);
spin_unlock(&hack_spinA);
return 0;
}
static int lockdep_thread(void *nothing)
{
set_freezable();
set_user_nice(current, 0);
while (!kthread_should_stop()) {
msleep(10);
nest_lock();
}
}
static int __init my_init(void)
{
lock_thread = kthread_run(lockdep_thread, NULL, "lockdep_test");
if (IS_ERR(lock_thread)) {
printk("create kthread fail\n");
return PTR_ERR(lock_thread);
}
return 0;
}
static void __exit my_exit(void)
{
kthread_stop(lock_thread);
}
MODULE_LICENSE("GPL");
module_init(my_init);
module_exit(my_exit);
2. Kernel debugging method
(1) printk
When I was developing simple drivers, the most commonly used debugging tool was printk, because I think this method is the simplest and most direct. This book also mentions this debugging method in detail. The printk() function is similar to the printf() function in the C language. One of the most important differences between them is that the printk() function provides an output level, and the kernel can use this level to determine whether to output to the terminal or serial port. The following are the output levels of the printk() function:
The Linux kernel defines 8 output levels for printk, with KERN EMERG being the highest and KERN DEBUG being the lowest. When configuring the kernel, a macro is used to set the system's default output level CONFIG MESSAGE LOGLEVEL_. The Linux kernel defines 8 output levels for printk, with KERN EMERG being the highest and KERN DEBUG being the lowest. When configuring the kernel, a macro is used to set the system's default output level CONFIG MESSAGE LOGLEVEL_DEFAULT. Usually, this value is set to 4, so only when the output level is higher than 4 will it be output to the terminal or serial port, that is, only KERN_EMERG to KERN ERR meet this condition. Usually, during the product development phase, the system default level is set to the lowest so that more problems and debugging information can be exposed during the development and testing phase. When the product is released, the output level is set to 0 or 4.
(2) Dynamic output
Dynamic print is one of the favorite output technologies of kernel subsystem developers. When the system is running, the system maintainer can dynamically choose which kernel subsystems to open for output, and can selectively open the output of certain modules, while printk is global and can only set the output level. To use dynamic output, you must turn on the CONFIG_DYNAMIC_DEBUG macro when configuring the kernel. The kernel code uses a large number of pr_debug()/dev_dbg() functions to output information, which use dynamic output technology. In addition, the system needs to mount the debugfs file system. Dynamic output has a control file node in the debugfs file system, which records the file name path, line number of the output, module name, and statement to be output for all files in the system that use dynamic output technology. You can use the following command to view it.
cat /sys/kernel/debug/dynamic_debug/control
The following examples illustrate how to use dynamic output technology:
(3) Oops analysis
When writing drivers or kernel modules, pointers are often explicitly or implicitly taken illegally or incorrectly used, resulting in an oops error in the kernel. When the processor accesses an illegal pointer in kernel space, a page fault interrupt is triggered because the mapping relationship between the virtual address and the physical address is not established. In the page fault interrupt, the address is illegal and the kernel cannot correctly establish a mapping relationship for the address, so the kernel triggers an oops error.
The following example demonstrates how to analyze a kernel oops error.
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/init.h>
static void create_oops(void)
{
* (int * )0 = 0; //人为编造一个空指针访问
}
static int __init my_oops_init(void)
{
printk("oops module init\n");
create_oops();
return 0;
}
static void __exit my_oops_exit(void)
{
printk("goodbye\n");
}
module_init(my_oops_init);
module_exit(my_oops_exit);
MODULE_LICENSE("GPL");
Compile the above into a kernel module and load the module with the insmod command, the following error will occur
The PC pointer points to the address of the error. In addition, "Call trace" also shows the calling relationship of the program when the error occurs. First, observe the error information create_oops+0x14/0x24, where 0x14 means that the instruction pointer is at the 0x14th byte of the create_oops() function, and the create_oops() function itself is 0x24 bytes. Continue to analyze this problem, assuming two situations: one is that there is source code for the error module, and the other is that there is no source code. In some actual work scenarios, it may be necessary to debug and analyze oops errors without source code.
First look at the case with source code, which is usually added to the symbol information table during compilation. Add the following statement in Makefile and recompile the kernel module.
KBUILD_CFLAGS +=-g
The following two methods are used for analysis. First, use the objdump tool to disassemble.
aarch64-linux-gnu-objdump -Sd oops.o
The disassembly tool objdump can be used to see the assembly of the error function create_oops(). The instructions of bytes 0x10 to 0x14 are used to assign 0 to the x0 register, and then write 0 to the x0 register. wzr is a special register with a value of 0, so a null pointer write error occurs here. Then, use the gdb tool. In order to quickly locate the specific location of the error, use the "list" command in gdb plus the error function and offset.
aarch64-linux-gnu-gdb oops.o
Let's look at the case where there is no source code. For binary files without compiled symbol tables, you can use the objdump tool to dump assembly code, for example, use the "aarch64-linux-gnu-objdump -d oops.o" command to dump the oops.o file. The kernel provides a very useful script that can quickly locate the problem. The script is located in the scripts/decodecode folder of the Linux kernel source code directory. We save the error log to a .txt file.
export ARCH=arm64
export CROSS_COMPILE=aarch64-linux-gnu-
./scripts/decodecode < oops.txt
The decodecode script converts the oops log information into intuitive and useful assembly code, and tells you which assembly statement has an error, which is very useful for analyzing oops errors without source code.
Summary: This review mainly introduces the deadlock situations and some common kernel debugging methods, and introduces the steps of oops analysis of kernel errors in detail. Kernel debugging methods cannot be fully mastered if you only read them. Therefore, when writing kernel drivers in the future, you should practice these methods more and truly transform them into your own skills.
提升卡
变色卡
千斤顶
京公网安备 11010802033920号
