Introduction to Software Test Automation Framework

The so-called automated testing framework is a collection of assumptions, concepts, and practices that provide support for automated testing. The following describes several commonly used automated testing frameworks:

1. The Record/Playback Myth

Every automated testing tool vendor will advertise that their tools are very easy to use and that testers without a technical background can easily automate all tests by simply recording the test operation process and then playing the recorded test script. This statement is very irresponsible.

Now let's analyze why automated testing cannot be done solely by recording/playback.

The scripts created by recording are basically written in a scripting language in a hard-coded manner. When the application changes, these hard codes also need to be changed. Therefore, the cost of maintaining these recorded scripts is very high, almost unacceptable.

All test scripts must be recorded when the application can be executed correctly. If a defect is found during the recording process, the tester must report it to the defect management mechanism. The entire recording script can only be continued after the defect is fixed. In this case, if you only rely on recording scripts for testing, the efficiency is very low.

At the same time, these recorded scripts are not very reliable. Even if the application is not changed at all, it may not be executed due to some unexpected conditions. If the tester uses the wrong scripting language when recording the script, the script must be re-recorded.

To sum up, although it seems easy to establish automated test scripts through recording, in fact, the following problems will be encountered: ① Most testers do not have a technical background and find it difficult to fully master the testing tools; ② The application must reach a certain stability before recording the test script; ③ The recorded test script is too tightly coupled with the test data; ④ The cost of maintaining automated test scripts is very high.

Therefore, it is far from enough to rely solely on recording/playback to complete automated testing. We should find a method that can solve the above problems and perform automated testing well.

2. Data-driven automated testing framework

Data-driven automated testing is a testing method proposed to solve the problem of tight coupling between development and testing. By establishing an association between the software metadata defined by testing and development, the metadata mapping table, a loose coupling relationship is established between testing and development. Whether the tester modifies the test script or the developer modifies the software, they only need to modify the metadata mapping table to ensure that testing and development are carried out simultaneously. In this way, the workload of debugging test scripts can be reduced, and automated testing can be better implemented.

●What is a data-driven automated testing framework?

A data-driven automated testing framework is a framework that reads input and output test data from a data file (such as an ODBC source file, Excel file, Csv file, ADO object file, etc.), and then passes it into a pre-recorded or hand-written test script through variables. Among them, these variables are used to pass (input/output) test data used to verify the application. In this process, the reading of the data file, the test status and all test information are written into the test script; the test data is only contained in the data file, not the script, and the test script is just a "driver", or a mechanism for transmitting data.

Data driven scripts

Data-driven scripts are scripts that are associated with applications. These scripts are recorded or written manually in the proprietary language of the automation tool, and then the variables in them are assigned appropriate values ​​as input for test data. These variables serve as a medium for some key application inputs, allowing the script to drive the application through external data.

1) Variable data, hard-coded component flags

These data-driven scripts often contain hard-coded data, sometimes very fragile identification strings in some widgets. When this happens, the script can easily become inoperable due to program changes.

2) Highly technical, repetitive test design

Another common feature of data-driven scripts is that all the efforts made in test design are ultimately reflected in the scripting language of the automation tool, or copied into manual and automated test scripts. This means that everyone involved in the development or execution of automated tests must be very proficient in the programming language of the test environment and automation tools.

●Advantages and disadvantages

1) Advantages: ① Test scripts can be created simultaneously with application development, and when application functions change, only the scripts of the business functions need to be modified; ② Model-based design is used to avoid duplicate scripts and reduce the cost of creating or maintaining scripts; ③ Test input data, verification data and expected test results are separated from the scripts and stored in another data file, which is convenient for testers to modify and maintain; ④ By judging whether the function return value is "True" or "False", error handling can be performed, which increases the robustness of the test script; ⑤ Automated test developers create data-driven test processes, and testers create test data; ⑥ Test results are collected during the test process and expressed in the context of input data, which can simplify manual result analysis.

2) Disadvantages: ① You must be very proficient in the script language in the automated testing tool; ② Each script will correspond to multiple data files, and these data files need to be stored in their respective directories according to the functional categories of the scripts, which increases the complexity of use; ③ In addition to maintaining the corresponding test plan based on the specific test data, testers also need to write these data into data files with different requirements; ④ When editing data files, you must pay attention to the transmission format required by the test script, otherwise errors will occur when processing the script. If maintained by specialized technicians, the automated testing framework that relies on data-driven scripts is simpler and faster to implement. However, maintenance work is difficult, and this data-driven model needs to be maintained, so even long-term maintenance will lead to failure. [page]

Keyword-driven automated testing (also known as table-driven test automation) is a variation of data-driven automated testing that supports tests consisting of different sequences or multiple different paths. It is an automation framework independent of the application, and is suitable for manual testing while handling automated testing. The keyword-driven automated testing framework is built on a data-driven approach, and the table contains instructions (keywords) rather than just data. These tests are developed as data tables using keywords, and they are independent of the automated tools that perform the tests. Keyword-driven automated testing is an effective improvement and supplement to data-driven automated testing.

The functions included in the entire process of keyword-driven automated testing are driven by keywords, and keywords control the entire testing process. The following takes "Post a Payment" as an example to illustrate how this automated testing method works (Table 1).

Pros and cons analysis

The keyword-driven automated testing framework is a completely different idea. It separates the things that change from the things that do not change in the traditional test script. This separation makes the division of labor clearer and avoids their mutual influence. The development and implementation of this model may be difficult and time-consuming compared to the traditional testing process, because we are trying to completely isolate our tests from the changes in the automation tools and the applications themselves. In order to achieve this goal, the most important thing is to enhance the component functions provided by the automation tools, such as error correction, avoidance, and data synchronization. However, such an investment is a one-time investment. Once the development is completed and put into use, the benefits it brings to us are huge. It is the easiest to maintain and use in the automated testing framework, and it can be repeatedly used in various applications and play a long-term role.

In addition, there are now some commercial products that meet the requirements, which reduces the difficulty of implementing such a framework. Using a keyword-driven automated testing framework, testers do not need to record test scripts, but design test scripts.

4. Hybrid automated testing framework

Combining the comparison of the above automated testing frameworks, the most successful automated testing framework at present should be the one that uses data-driven and keyword-driven automation testing frameworks in combination: using data-driven scripts as input, the test results are obtained through processing by the keyword-driven framework to complete the automated testing process. This allows data-driven scripts to utilize the libraries and tools that are usually provided by keyword-driven frameworks. These framework tools can make data-driven scripts more compact and less prone to failure.

Keyword-driven automated testing framework model

The following will introduce an automated testing implementation solution guided by the keyword-driven automated testing framework idea - the keyword-driven automated testing model, which was developed by Carl Nagle of SAS Institute. Figure 2 describes the structure of the test model.

This model mainly consists of a core data-driven engine, component functions, support libraries, and application mapping tables. Automated testing begins with an initial script that passes high-level test tables to high-level drivers. When high-level drivers process these tables, they call middle-level drivers when they encounter middle-level test tables. Middle-level drivers also perform similar processing when processing middle-level tables. When low-level drivers process low-level tables, they try to synchronize the application with the test. When a low-level driver encounters a low-level keyword component for a component, it determines the type of the component and calls the corresponding component function module to process the instruction operation. All of these elements rely on the information in the mapping table, which is a bridge between the automated test model and the application being tested.

●Application Mapping Table

The application mapping table is one of the most critical components in the automated testing model. Before designing the test, the tester first defines a set of naming conventions for each object in the application, and uses a mapping table to link these names with the object names recognized by the automation tool, so that the tool can accurately locate and manipulate the object. Our test scripts only require single-point maintenance. In the above example, if the name of the button or the displayed text changes, then all places in the script involving these names must be modified. If we establish such a mapping and use the logical object SavePushButton to represent the real button object that confirms the save, then this example can be written as "Click SavePushButton". When the name of the button or the displayed text changes, you only need to quickly modify the corresponding recognition method in the mapping table without modifying the script (Table 2)

Component functions

Component functions are functions that implement user operation instructions for interface objects. One component object type corresponds to one component function library. For example, for a text box object, testers may perform multiple operations on it: input text, verify the value of the text box, verify certain properties of the text box, etc. The functions that implement these operations are placed in the component function library of the text box. General testing tools provide such functions, and we can add additional code to them to detect errors, correct errors, and help synchronization. This type of code is necessary to implement unattended automated testing.

Component functions provide an isolation layer between applications and automation tools. Without this isolation layer, changes or improvements to the automation tools themselves will affect existing scripts. However, with component functions, we can add a pair of patch codes to adapt to these changes and transfer damage to the test. Component function keywords and their parameters constitute the lowest level of the automation model. After understanding the low-level vocabulary and mapping table, we can build test tables based on them.

●Test tables and core data driven engine

Test tables are divided into low-level, middle-level and high-level. The low-level test table specifies the details of each step of the test. These instructions act directly on the interface objects and cannot be further subdivided. The middle-level test table assembles the low-level test tables to perform more useful tasks. The same low-level table can be used for multiple middle-level tables, so we should develop as few low-level tables as possible, and then assemble them according to different purposes to achieve maximum reusability. Similarly, the high-level test table assembles the middle-level tables to form a test loop. Each loop is complete and can be customized with different types and quantities of tests.

For example, the three actions of opening a web page, logging in, and closing a web page can be represented by three low-level tables. Each table defines the specific steps to implement the corresponding action, so the low-level table is also called the step table. The low-level table uses the object names defined in the mapping table and the low-level keyword thesaurus defined by the component function. Table 3 is a low-level table that implements the login action. The low-level table keyword representing "login" is likely to appear in middle-level tables such as "verify wrong login", "verify correct login", and "verify blank login". These middle-level tables together constitute the "verify permissions" high-level table.

Corresponding to the above three test tables, the core data-driven engine is divided into high-level drivers, middle-level drivers and low-level drivers. The high-level driver reads each record of the high-level table. If it encounters a middle-level table keyword, it passes the table to the middle-level driver, and so on, until it reaches the low-level table. The low-level driver calls the component function corresponding to the low-level instruction in the keyword thesaurus to complete the final execution. Finally, it should be noted that such a hierarchical structure is not fixed and can be adjusted according to actual application conditions.

●Support library

Support libraries are programs and tools, such as file handling, string handling, buffer handling, database access, logging tools, etc., which provide the most basic support for the automation model.

Conclusion

Automated testing framework is undoubtedly an inevitable development direction for enterprises to implement automated testing. It is important to have a proper foundation for successful test automation. In order to choose a suitable automated testing framework, enterprises need to comprehensively consider many factors such as maintenance cost, test data, testability, tester skills, etc. Looking back on the development process of automated testing, past experience tells us that it is impossible to rely on simple recording/playback testing methods or traditional test script tools to complete testing, because the scripts generated by recording are difficult to maintain and have a short lifespan.