Testing Tools and Test Automation

The history of human evolution and development is a history of continuous creation and use of tools. Tools are the physical manifestation of human imagination and a great boost to social progress. For testing, tools are also indispensable. Even if you want to judge whether a manufacturer's testing level is in the "barbaric era" or has evolved to the "modern society", you can get a rough idea by observing the test tools it uses. In fact, many test projects, especially performance and stability test projects, must be completed with the help of test tools; verifying the large-scale deployment capabilities of services is unimaginable without the support of tools. For a simple example, if there is no test tool to test a device that can access 4,000 PPPoE at the same time, you can only build an environment with 4,000 clients, which is almost impossible to implement in practice, not to mention that there will be many similar test projects, and each version needs to be tested repeatedly.

1. Test tools

With the explosive development of network technology, a wide variety of test tools have also been developed. According to their main functions, they can be roughly divided into the following categories* (*Note: The current test tools are relatively complex and may not be completely strictly classified. For example, Chariot and Avalanche can provide powerful traffic generation functions and are also good business simulation tools).

Traffic generation tools: mainly used to generate large-scale network traffic and test the forwarding plane function of the equipment. Some of these tools are software installed directly on the host, such as Chariot; some are dedicated hardware, such as test instruments provided by professional manufacturers such as Spirent and IXIA;

Protocol simulation tools: mainly simulate signaling protocols to test the control plane function of the equipment. For example, routing protocol simulation, MPLS-related protocol simulation, authentication access protocol simulation and other test tools;

Business simulation tools: mainly simulate application layer protocols and customer services to test the application and business carrying capacity of the equipment. General L4-L7 test instruments and tools provide powerful business simulation capabilities, such as Avalanche, BPS and other test instruments and Chariot software;

Attack tools: including hacker tools, Fuzzing and Vulnerability test tools, to test the security and attack prevention capabilities of the equipment. Typical DDOS tools include Mu Dynamics, Codenomicon, BIFFIT, SAINT, NESSUS, nMAP, and SYN flood.

Platform tools: Generally, they provide a secondary development platform with a complete integrated development environment, support a variety of high-level computer languages ​​suitable for testing (such as Perl, TCL, Python, etc.), can carry out complex secondary development, integrate Lib libraries that are encapsulated and abstracted to adapt to testing, and even provide some automated test suites that have been proven in practice, and can call other test instruments and tools through external interfaces. Similar to Microsoft's Visual Studio development environment, it is for development services, while the former is for testing services. Platform tools require huge investment, mainly to meet the needs of manufacturers to build their own unique testing capability system, and are generally developed and maintained by manufacturers themselves. H3C has built this type of platform, called the Versatile Test Platform (VTP).

Generally speaking, for mature protocol or application testing, there are excellent commercial test instruments and test tools that can meet more than 80% of testing needs. However, for the latest protocols and applications, or non-standard customized needs of specific customers, manufacturers are required to have certain independent development capabilities for test tools. Take H3C as an example. When the 802.1x protocol was first applied in China, the device would have a high probability of software crashing when a large number of users accessed the device at the same time. Therefore, the test team developed a tool to simulate the access of a large number of 802.1x users, and finally found and solved the problem quickly. The commercial 802.1x test tool with similar functions did not appear on the market until about two years later.

H3C has a deep understanding of the importance of test instruments and test tools in optimizing test efficiency, improving test level, and improving product quality. It has invested a lot in this regard. On the one hand, it has purchased a large number of advanced commercial test instruments and tools in the industry, such as test instruments and test software from companies such as Spirent, IXIA, BPS and Veriwave. On the other hand, through a dedicated test platform team, it has also independently developed a large number of test tools and software to provide support for test requirements that commercial test software cannot cover, ensuring that H3C can launch the latest features at the fastest speed. The test tools developed by the team have now formed a series and become an important helper for test engineers, such as multi-client simulation series tools, routing protocol series test tools, consistency series test tools, and comprehensive service simulation series tools. The general test platform developed by the team has built a company-level automated test framework, providing a complete GUI and CLI automated test solution to serve H3C's full range of product testing services.

2. Test Automation

Test tools and test automation are twin brothers. The purpose of test tools is to replace some tedious manual test operations or complete test activities that cannot be completed by manual testing, and achieve a certain degree of test automation. The development and evolution of test automation is inseparable from the progress of test tools. With the progress and improvement of test tools, a large part of the test work can be done unattended and fully automated. Looking back at the development history of automated testing technology, it can be roughly divided into three generations. [page] The

first generation, tool-centered automation

time: Before the mid-1990s,

this generation of test tools used in automation was most typically capture/replay tools, that is, capturing the user's mouse and keyboard operations and recording them. These operations can be replayed during the next test to repeat the last test. These tools generally also provide simple script functions, and testers can also modify the recorded scripts as needed, such as adding loop operations and some simple judgment conditions, to strengthen the test. However, because the script language is simple, the script function is often just an embellishment. For example, QARun and WinRunner are typical representatives of this type of tool. This generation of test automation technology has great limitations:

limited automation. Each tool has its own unique script language, but it is not a full-featured script language. The operations that can be automated are limited, and it cannot constitute a complete automation solution. Scripts of different tools cannot be shared;

it has poor adaptability to changes in SUT (System Under Test). If the GUI of the SUT changes, the recorded script can hardly be used again, which is almost a fatal flaw in an era when software is constantly improving and changing.

The second generation, script-centered automation

time: the late 1990s to the early 21st century

This is the era of personal heroism in automation. At this stage, some test teams have realized the importance of adopting a unified script language, and have found a fully functional script language suitable for testing work and vigorously promoted it in the team. However, due to limited experience and lack of good top-level design, test automation mainly relies on the subjective initiative of test engineers. Eight immortals cross the sea, each showing their magical powers, everyone is a script engineer, and a large number of test scripts are generated.

Although this generation of automation has a unified script language, test engineers can also share a small amount of scripts. But in general, each team worked independently, with different styles and uneven quality. Personal automation achievements that were closely related to the personal test environment were difficult to fully transform into effective team platform accumulation. However, this stage trained a large number of skilled test automation engineers, laying a solid foundation for the next stage in terms of personnel and technology.

The third generation, platform-centric automation

Time: the beginning of the 21st century to the present

After several years of exploring the second generation of automation, insightful test managers and outstanding test engineers realized that the scripts generated by this wild growth had great problems in maintainability, reusability, and topology adaptability, and could not truly form a sustainable and effective team accumulation. Therefore, the top-level design of automated testing was put on the agenda: building an excellent automated testing platform; developing scripts based on logical topology and mapping them to physical topology when executing; abstracting common test operations into action words and implementing them as a general class library for all test engineers to use; formulating script development, acceptance, and maintenance specifications to ensure the consistency, universality, and maintainability of scripts. Scripts developed based on this test automation platform can truly be transformed into effective team accumulation. Taking

the development of H3C's test automation as an example, before 1999, it only used simple capture and playback test tools and wrote simple scripts based on these tools, which belonged to the first generation of automation. During 1999-2002, the test platform team introduced the TCL language adapted to communication equipment testing and developed a general test platform, but the unified ATF (Auto Testing Framwork) was not yet mature and was in the second generation of automation. In 2003, the H3C test team released ATF and started the development of the Testbladev1/v2 script system, which marked that H3C's test automation entered the third generation and was continuously optimized in practice. Based on VTP and ATF, H3C has achieved automation of more than 80% of functional tests and provided multiple automated test suites for performance tests, stress tests and persistence tests.

III. Outlook: The fourth generation of automated testing technology

So will there be a fourth generation of automated testing technology? The answer is yes. The next generation of automation technology must be network-centric test automation, which can also be called cloud-centric test automation. All test equipment (real and virtual), test instruments and test hosts are uniformly managed through a test automation management system, and what will be presented to the test engineer is a test equipment cloud. Test engineers can remotely log in to the test automation management system and submit their own automated test tasks through the task management system. They only need to clearly describe the device type required for the test task, the link type of the device connection, and the test suite to be executed. The system will search and calculate in the test cloud according to the rules to find out what time can provide the test execution environment required for this test task. Test engineers can schedule to run the automated task at any time after this time and receive the automated test results on time.

Compared with the third generation, the fourth generation of automated testing technology will have a qualitative leap in manageability, ease of use, and equipment utilization, but it must still be based on a stable and reliable test platform and a complete test script system as the basis for test execution, which means that the third generation of test automation cannot be crossed. Otherwise, the so-called cloud test is a sourceless water.

IV. Conclusion

Test automation has greatly improved the test efficiency, freeing test engineers from simple and repetitive mechanical operations and devoting more energy to more creative test design and more complex test execution. But we must also recognize the limitations of test automation. First of all, automation is just a mechanical repetition of existing test designs and will not exceed the existing test cognition. Secondly, in complex testing scenarios, there are a wide range of factors that affect test results. It is difficult to rely on machines to make judgments, and it must be done by people.

These limitations determine that automated testing cannot completely replace manual testing. However, the role of testing tools and automation technology in complex environment simulation and business model construction is irreplaceable. Therefore, even manual testing cannot be separated from testing tools and test automation technology. It can be said that the progress of testing tools and test automation is driving the development of the entire testing industry.