Improved design of Linux system software shell protection technology

Shelling is a very effective way to protect the software kernel. Currently, most of the shelling methods under Linux systems directly inherit the shelling theory and methods of Windows programs, and have made limited extensions on traditional shelling tools. There are still very few shelling tools that are implemented solely under LINUX systems. How to minimize the exposure of kernel information under Linux systems, increase the difficulty of cracking by adding difficult anti-reversing means, and hide the shelling protection program well are all the main problems to be overcome at present.

According to the different ways of loading the shell program, the existing software shelling technology can be divided into: embedded, attached and included. The most classic embedded shell is UPX, which supports multiple file types and has an advanced compression algorithm. However, this method lacks anti-dynamic tracking function. Crackers can use dynamic tracking and debugging methods to crack UPX [1]. Reference [2] describes the attached shell (SELF shell): adding code segments for processing security operations to the original elf file format, but without built-in decompression, which easily exposes the location of the shell. The included shell combines the two types of shells, compression and protection, but the entry address will change during the shell loading process. Combining the advantages and disadvantages of the above three shelling methods, this article proposes an improved shell protection method in the Linux system - shelling and reconstructing executable files - SRELF [3]. The improved algorithm combines the shell program with the key parts such as code segments and data segments in the target executable file, introduces polymorphic deformation technology, makes the program polymorphic, improves the anti-cracking ability of the shell program, and also well hides the shell information. Experiments have shown that the improved method solves the problem that the entry address of the shell is easy to change, making the shell program appear in a polymorphic form, which greatly increases the difficulty of anti-cracking.

1 Shelling principle

Shelling is essentially to attach a special program to the application program and point the execution entry of the program to the attached special shell program. The shell loading process is shown in Figure 1. First, the shell program needs to obtain the application programming interface, that is, the API address. In the shell program code, the required API address is dynamically loaded using the explicit link method. After passing the shell program, the data of each block is decrypted according to the definition; if compression technology is used during shelling, it must be decompressed before decryption, and then the decompressed file must be mapped to the specified memory address. After modifying the input table of the original program file, the code address in the HOOK-API table is filled to indirectly obtain the control of the program. After verification and testing, jump to the original entry point (OEP) [3]. Most of the existing software packing methods change the original file structure, change the program entry address during the relocation process, and map part of the program into the address space during the loading process. If the cracker knows how to address in a packing program, then when the file is loaded into the memory, the information of the packing program can be found. In response to the above problems, this paper proposes an improved packing method SRELF.

2 SRELF Packing Method

2.1 Principle of SRELF method

SRELF——packs and reconstructs executable files. Its biggest feature is that the reconstruction deformation makes the reconstructed program polymorphic. First, the core part of the target elf file is extracted, and then combined with the prepared decryption or decompression program, anti-static analysis and anti-dynamic tracking program to make the packing program polymorphic. Finally, the integration program is traversed and a new elf executable file is reconstructed according to the standard format of the elf file.

2.2 SRELF method implementation

As shown in Figure 2: The basic flowchart of the SRELF packing method. First, understand the structure of the application binary interface file (i.e., the extension is elf file). Figure 3 shows the structure of a standard elf file. From the figure, you can see that an ELF header is at the beginning of the file, which stores a roadmap describing the file organization. This is followed by some segments or sections. The segment contains the information required for the file to run, and stores the information of the object file for linking and relocation. [page] Step 1: Extract the core part of the target file

Extract the executable instructions, dynamic link table, segment or section information of the core part. Since there are indirect jumps in the assembler, the control flow graph generated by the disassembly is interrupted, which affects the accuracy of extracting the core code. Here, an indirect jump program method is introduced [4].

1) Single-step downward tracking from the shell to the OEP of the original program, and modify the overflow flag when an exception is thrown;

2) Analyze the type of the operand. If it is a direct addressing type, enter 4); otherwise, enter 3);

3) Find the definition statement or function entry, set the CREAT_SUSPENDED counter, if the counter is increased by one, open the host process as a suspended child process, call GetThreadContext() to obtain the context of the child process initialization thread;

4) After the judgment, the program performs input locking, simulates the definition position and the target indirect jump operation on the statement sequence, and copies the content of the acquired register. After using the indirect jump statement to obtain the information in the elf header file, locate the header table of GetElfCore, then locate the core position of the required code segment, and copy the data in it.

Step 2: Encrypt the extracted core code and combine it with polymorphic deformation technology

The process injection encryption technology is used to reversely prevent the decompressor from attaching to the protected process. First, CONTEXT.EBX obtains the PEB address of the child process. After reading the image address of the PEB child process, the base address parameter is pointed to the retrieved image base address, and finally the monitoring of various external interrupts is completed. The encrypted core code is improved by polymorphic deformation technology, allowing it to change its own code, so that the search string recognition technology is ineffective. The code transformation encryption compression used in the article is used to hide itself. In order to prevent the search string method from detecting, the decryption code triggers the code module, the program inserts it into itself, calls the function contained in the code module, moves the instructions around the routine, randomly adds useless instructions, and uses different registers and opcodes, so that the binary code corresponding to the decryption routine is completely different in different infected bodies [5]. The polymorphic deformation engine process is shown in Figure 4. Step 3: Integrate the core part and the shell part of the target file

Integrating the core part and the shell part of the target file requires precise calculations and setting the correct data of the elf file header, program header table, section header table, etc. Only in this way can the newly generated elf file run normally. When integrating elf and other data, the traversal integration method is used to analyze the core code blocks extracted earlier one by one. The core code after shelling is integrated by looping. [page]

1) First, set up a state table to record the state type of each variable. Record the size of the physical file and the size of the memory occupied by the loaded memory. First, execute the initial state of the function and set all functions except the entry function to untainted. If a variable appears in the state table of multiple forward code segments, recalculate its size and update the state table.

2) Internal execution: Execute the internal instructions one by one in order, change the variable state table, write the core content of the target file to the back of the shell part, and form the output state table when the execution reaches the last step in the segment. At the same time, check the security of the program deformation state and pass the interface function parameters [6].

3) Loop rewinding integration part: When the core code is fully executed, check whether there is a loop header containing a deformation in the subsequent code segment. If so, the code block is the outermost loop header that meets the conditions, set the parameters of the attributes and size in the program header table to the new calculated values, and check whether the type status of the variables in the input and output state tables has changed. If so, restart the rewinding execution until the state table stops updating.

3. Comparison of performance of related algorithms

Table 1 gives a comparison of the new packing algorithm SRELF with the ASProtect algorithm, tElock algorithm, and Armadillo algorithm. Table 2 gives the definitions of related symbols. 1) Security Improvement

Since SRELF uses binary code for encryption or compression, and does not limit the size of the shell program, sufficient junk instructions can be inserted into the code [7]. Coupled with the highly complex encryption deformation compression algorithm, the high security of the shell program is guaranteed. In addition, the deformation reconstructs the elf file, and the content of the elf file is completely changed, which increases the difficulty of static analysis and shelling of the header file. In addition, sufficient anti-dynamic tracking instructions are added to SRELF to prevent the protected file from being dynamically tracked. Through the above security analysis, it can be proved that the security of the shelled elf file has been greatly improved.

2) Reduced computational effort

3) Enhanced camouflage

The structure of the elf file after packing has not changed, and the SRELF packing method does not need to change the program entry address, which hides the shell program very well.

4) Improved scalability

Since the SRELF method theoretically has no limit on the size of the shell program, it has good scalability. Therefore, the encryption method, anti-static analysis method, and anti-dynamic tracking method in SRELF can be updated synchronously in the future to improve and strengthen it.

In summary, SRELF overcomes the problems exposed in existing shelling methods, solves the major defects of changing the elf file structure and the program entry address, makes the shelled program polymorphic, and increases the difficulty in anti-shelling. Therefore, the SRELF method is a shelling method with both high security and good feasibility.

4 Conclusion

This paper studies the common problem of existing shelling software in anti-cracking - the emergence of new security defects that do not conform to all known patterns [8], summarizes the shortcomings of existing shelling methods, and proposes an improved shelling method to address the shortcomings - the reconstructed deformed SRELF shelling algorithm. Software shelling combines encryption, deformation, anti-static analysis and anti-dynamic tracking of key codes, while having a relatively streamlined amount of calculation, improving the anti-cracking ability of the encryption part. This paper only considers the introduction of program deformation in the shelling process, making it difficult for the shelling software to remove it. The next step will be to solve the problem of program compression in the shelling process. In addition, the improved method will be further applied to Linux system software.