Use of eBPF on Android

Latest update time：2021-01-29

Reads：

1. What is eBPF

eBPF is the abbreviation of extended BPF, and BPF is the abbreviation of Berkeley Packet Filter. Friends who are familiar with Linux networks should be familiar with BPF. It uses a register-based virtual machine to describe the behavior of packet filtering through specific syntax rules. The more commonly used function is to count traffic through filtering. The tcpdump tool is implemented based on BPF. eBPF extends it to achieve more functions.

The main differences are as follows:

1) Allow code snippets to be written in C and compiled into eBPF bytecode through LLVM;

2) cBPF only implements SOCKET_FILTER, while eBPF also has KPROBE, PERF, etc.

3) BPF uses socket to implement interaction between user mode and kernel, while eBPF defines a new system call dedicated to eBPF, which is used to load BPF code segments, create and read BPF maps, and is more general.

4) BPF map mechanism, which is used to temporarily store the data generated by the BPF code in a key-value manner in the kernel.

eBPF can be simply understood as the kernel implementing a virtual machine mechanism, compiling C-like code into bytecode (detailed explanation later), hanging it on the kernel hook. When the hook is triggered, the kernel runs the bytecode in the "sandbox" of the virtual machine. This can not only conveniently implement many functions, but also ensure the security of the kernel through the sandbox.

2. What can eBPF do?

If BPF focuses on traffic monitoring, then eBPF focuses on the performance field. Through various hooks, it can obtain various performance indicators of the system in user space. It can be as large as the overall statistical indicators of the monitoring system, or as small as the running time of a system function.

Here we need to mention the open source project BPF Compiler Collection (BCC), which is a very convenient system monitoring tool based on eBPF. The following BCC diagram can well illustrate what we can do with eBPF. BCC can also run on the Android system, but the system needs to be modified to a certain extent. I may write a separate article to explain it later. For kernel developers, I am more concerned about how to implement the monitoring function by themselves, which will also be briefly explained below.

From the above picture, we can see that eBPF can monitor almost all aspects of the system:

1) Various indicators of applications and virtual machines

2) System library performance monitoring

3) Kernel system call performance

4) File system performance

5) Network call performance

6) CPU scheduler performance

7) Memory management performance

8) Interrupt performance

3. eBPF Framework

Before we start, let's explain the terms in eBPF to help you understand it better.

1) eBPF bytecode: The hook code written in C language is compiled into , loaded into the kernel through the program, and runs in the kernel "virtual machine" after the hook is triggered.

2) JIT: Just-in-time compilation, compiles bytecode into native machine code to increase running speed, similar to the concept in Java.

3) Maps: The hook code can store some statistical information in a map of key-value pairs to communicate with user space programs and transfer data.

There are many detailed explanations of the eBPF mechanism on the Internet, so I won’t go into detail here. Here is a picture that includes everything involved in using or writing eBPF. The following will explain this picture in detail.

1) foo_kern.c hook implementation code, mainly responsible for:

Declare the Map node to be used
Declare the hook mount point and processing function

2) Compile into bytecode via LLVM/clang

Compile command: clang --target=bpf

The Android platform has an integrated eBPF compiler, which will be mentioned later.

3) foo_user.c user space processing function, mainly responsible for:

Load the bytecode compiled from foo_kern.c into kernel
Read the information in the Map and process it and output it to the user

4) When the kernel receives a loading request from eBPF, it will first verify the bytecode and compile it into machine code through JIT. When the hook event comes, the hook function is called

The kernel will verify the loaded bytecode to ensure the security of the system. The main verification rules are as follows:

a. Check whether the GNU GPL is declared and whether the kernel version supports it

b. Function calling rules:

Allows bpf functions to call each other
Only BPF helper functions allowed by the kernel are allowed to be called . For details, please refer to the linux/bpf.h file
Functions other than those mentioned above and dynamic linking are not allowed.

c. Process processing rules:

Do not use loops to prevent the kernel from getting stuck in an infinite loop.
No unreachable branch code is allowed

d. The stack size is limited to MAX_BPF_STACK.

e. The compiled bytecode size is limited to BPF_COMPLEXITY_LIMIT_INSNS.

5) Hook mounting points, mainly including:

In addition, there are a lot of examples in the samples/bpf directory in the kernel source code. If you are interested, you can read them.

IV. Use of eBPF on Android Platform

After the above boring explanation, everyone should have a basic understanding of eBPF . Now let's practice it through a small example of monitoring performance on the Android platform.

The requirement of this small example is to count the number of system calls made by each application in the system over a period of time.

1. Android system compiles support for eBPF

Currently, the Android compilation system has integrated eBPF, and android.bp can be used to easily compile eBPF bytecode in the Android source code.

android.bp example:

The relevant compilation code is in soong's bpf.go. Although Google has few documents about soong, at least the code is relatively clear.

Here $ccCmd is usually clang, so its compilation command is mainly clang --target=bpf. It is no different from the normal bpf compilation.

2. eBPF hook code implementation

After solving the compilation problem, the next step is to start implementing the hook code. We are going to use the tracepoint hook, and first we need to find the tracepoint functions sys_enter and sys_exit that we need .

The function is defined in the include/trace/events/syscalls.h file

1) The trace parameters of sys_enter are id and an array of length 6.

2) The trace parameters of sys_exit are two long integers id and ret.

After finding the hook, the next step is to write the hook processing code:

1) Define a map to save system call statistics. When DEFINE_BPF_MAP declares a map, macro functions for deletion, modification, and query are also generated. For example, the following function will be generated in this example

bpf_pid_syscall_map_lookup_elem

bpf_pid_syscall_map_update_elem

bpf_pid_syscall_map_delete_elem

2) Define the callback function parameter type, referring to the previous tracepoint definition.

3) Specify the tracepoint event to monitor .

4) Use the bpf_trace_printk function to print debug information, which will print the information directly to ftrace.

5) Search for the specified key in the map.

6) Update the value of the specified key.

3. Loading hook code

We only need to push the *.o file we compiled to the system/etc/bpf directory of the phone, restart the phone, and the system will automatically load our hook file. After successful loading, the map and prog files we defined will be displayed in the /sys/fs/bpf directory.

The system loading code is in system/bpf/bpfloader, and the code is very simple.

The main operations are as follows:

1) Write 1 to the following two nodes in the early-init phase

– /proc/sys/net/core/bpf_jit_enable

Enable eBPF JIT. Defaults to 1 when the kernel sets BPF_JIT_ALWAYS_ON

– /proc/sys/net/core/bpf_jit_kallsyms

Allow privileged users to read kernel symbols through the kallsyms node

2) Start bpfloader service

–Read the *.o file in the system/etc/bpf directory and call the loadProg function in libbpf_android.so to load it into the kernel.

–Generate the corresponding /sys/fs/bpf/ node.

– Set the property bpf.progs_loaded to 1

The sys nodes are divided into two types: map nodes and prog nodes, which are map_<filename>_<mapname> and prog_<filename>_<mapname> respectively.

Below is the node information on the Android Q version.

You can use the following command to debug dynamic loading

4. User space program implementation

Next we need to write a user space display program, which essentially reads the BPF map through a system call in user mode .

1) eBPF statistics only work when bpf_attach_tracepoint is called. bpf_attach_tracepoint is a function in bcc. Android packages part of bcc into libbpf and puts it in the system library.

2) To obtain the map's fd, bpf_obj_get will directly call the bpf system call.

3) Wrap fd into BpfMap. Android defines many convenient functions in BpfMap.h.

4) Traverse the map callback function. The return value must be android::netdutils::status::ok (has been modified in the new version of Android).

5. View the operation results

Execute mm directly in the directory , push the compiled bpf.o to the /system/etc/bpf directory, push the statistics program to the /system/bin directory, restart, and see the results.

The first one is pid, and the second one is the number of system calls.

So far, we have introduced how to use eBPF on the Android platform to count the number of system calls made by each PID in the system within a period of time.

In addition, there are still many technical details that have not been studied in depth, but after all, this is just a preliminary exploration, so I will stop here for now, and I will study it further in depth later. The time for research is still relatively short, so if there are any mistakes, please correct me.

References

A brief history of eBPF (Part 2) :

https://cloud.tencent.com/developer/article/1006318

Two articles about Google's native use of ebpf :

https://source.android.com/devices/architecture/kernel/bpf

https://source.android.com/devices/tech/datausage/ebpf-traffic-monitor

BCC :

https://github.com/iovisor/bcc

5T technical resources are available for free! Including but not limited to: C/C++, Arm, Linux, Android, artificial intelligence, microcontrollers, Raspberry Pi, etc. Reply " peter " in the official account to get them for free! !

Remember to click Share , Like and Watching , give me some power

Latest articles about

■CPU cache consistency: from theory to practice

■Throw some cold water on the cunning Hongmeng

■The process of receiving network data packets

■Let's talk about the current AI and a bunch of other things in plain language

■Vomiting blood sorting | Liver over Linux interrupt all knowledge points

■Introduction to Linux V4L2 subsystem and video codec equipment

■Arm64 stack backtrace

■Unbeatable! I strongly recommend taking the software exam this year!

■Domestic real-time operating system: real-time comparison with RT-Linux and Zephyr

■MIPI-DSI Display Process D-PHY