A brief introduction to Linux kernel tracing technology

jobszheng5

A brief introduction to Linux kernel tracing technology [Copy link]

<div class='showpostmsg'># A brief introduction to Linux kernel tracing technology. The embedded Linux programs we usually write are mainly divided into two categories: Linux applications, that is, programs running in the user layer; Linux kernel drivers, that is, programs that need to be compiled into the kernel. The former runs under the framework of the Linux kernel. When a bug occurs, the kernel will issue an alarm or even directly kill the running program; while the latter, if it occurs during operation, will directly crash the kernel. Today's learning and sharing content is kernel debugging technology. ## Ftrace Ftrace, whose name comes from Function Trace, is based on static code stub technology and does not require users to define trace behavior through additional programming. Static code stub technology is more reliable. It will not cause kernel crashes due to improper user use. Ftrace uses the profile feature of GCC to add a stub code at the entrance of all functions, and ftrace overloads this code to implement the trace function. Ftrace has written its own mcount stub function. When using GCC to compile the kernel, add the -pg option, and then add the mcount call code at the entrance of each function to implement the trace function. After configuring the Ftrace related compilation options, ftrace provides an access interface to the user space through the debugfs file system. Commonly used trace tracers include Function, Function-graph, and blk. **Ftrace Function Trace** The function tracer will record all functions in the current system operation process. Note that this is all functions. Therefore, we have to introduce a dynamic ftrace function. Since the Function function will print a lot of debugging information, it is recommended to use dynamic ftrace in this article. Dynamic ftrace uses two files, set_ftrace_filter and set_ftrace_notrace. The former sets the function to be traced, and the latter filters out the functions not to be traced. ## Perf tool The famous perf tool is a Linux performance analysis tool that uses CPU, performance monitoring unit (PMU) and software counting for performance analysis. In the actual debugging process, the perf record command is generally used, and a number of options are added: -e: select an event -p: specify a process ID to collect data for a specific process -o: write the data file of the collected data After that, the perf report command is used to parse the data generated by perf record and give the analysis results. I was fortunate to use the perf tool to debug the process and observe perf record and perf report in person. Frankly speaking, it is really beyond my ability. ## eBPF tool eBPF is essentially a kernel code injection technology. The general steps of injection are as follows: 1. The kernel implements an eBPF virtual machine 2. Use high-level languages such as C language to write code in user mode, and compile it into BPF target code through the LLVM compiler 3. In the user mode, the BPF target code is injected into the kernel by calling the bpf() interface function 4. The kernel converts the BPF target code into native code through the JIT compiler 5. The kernel provides a series of hooks to run BPF code 6. The kernel mode and user mode use a mechanism called map to communicate The advantage of eBPF is that it can flexibly modify the kernel processing strategy without modifying the kernel code. It is also the mainstream kernel injection technology and the fastest growing debugging technology. Keep paying attention and keep learning.</div><script> var loginstr = '<div class="locked">To view all the contents of the post, please <a href="/bbs/member.php?mod=logging&action=login" style="color:#e60000" class="loginf">log in</a> or <a href="/bbs/member.php?mod=register_eeworld.php&action=wechat" style="color:#e60000" target="_blank">register</a></div>'; if(parseInt(discuz_uid)==0){ } </script><script type="text/javascript">(function(d,c){var a=d.createElement("script"),m=d.getElementsByTagName("script")[0],eewurl="//counter.eeworld.com.cn/pv/count/";a.src=eewurl+c;m.parentNode.insertBefore(a,m)})(document,523)</script>

chejm

Thanks to the host for sharing the Linux kernel tracing technology. I have benefited a lot from it. I hope to have the opportunity to practice the technology provided by the host.

hellokitty_bean

I have heard of this Berkeley packet filtering technology. Wow, it is really advanced. . . . . I don’t understand the kernel. . . .

jobszheng5

hellokitty_bean posted on 2024-5-2 20:47 I have heard of this Berkeley packet filtering technology. Wow, it is really advanced. . . . . I don’t understand the kernel. . .

eBPF technology has only been popularized for a few years, but it is very convenient. If you want to play with the kernel, it is recommended to learn about eBPF.

hellokitty_bean

Hmmmm. I learned it from the master! But if I haven’t even learned Linux well, can I play EBPF?

jobszheng5

hellokitty_bean posted on 2024-5-5 21:23 Yes, yes. I learned from the great god! But if I haven’t even learned Linux well, can I play EBPF?

I am ashamed, ashamed, my current Linux level is still very average, and I am still learning.

eBPF is a kernel observation technology. I am currently mainly studying the data structure and algorithm application in the kernel.

hellokitty_bean

The host is being modest!

My current feeling is: Only after understanding the Linux kernel can we learn to observe. Otherwise, even if we observe it, we cannot recognize its origin. Isn’t it easy to become a blind man riding a blind horse?

jobszheng5

hellokitty_bean posted on 2024-5-6 09:17 The author is too modest! My feeling now is: only after understanding the Linux kernel can you learn to observe, otherwise even if you observe it, you can't recognize it...

eBPF is a tool that walks on two legs.

Using a magnifying glass can help you observe and understand the world, and using a microscope is another way to observe and understand the world.

hellokitty_bean

Hmm, that makes sense!!! Walking on two legs, absorbing both... It's a good idea!

freebsder

Learn about dtrace. It was developed by Sun in Solaris 20 years ago. Now it seems to be usable in Linux. Linux particularly wants one thing: dtrace and another thing: zfs

nian1206

Mark it and I will use it later. Thank you very much for sharing. May good people have a peaceful life!