Building the next generation of software-centric automated testing systems

1. Introduction: Design Challenges of Automated Test Systems

测试管理人员和工程师们为了保证交付到客户手中的产品质量和可靠性，在各种应用领域 (从设计验证，经终端产品测试，到设备维修诊断) 都采用自动化测试系统。他们使用自动测试系统执行简单的“通过”或“失败”测试，或者通过它执行一整套的产品特性测试。由于设计周期后期产品瑕疵检测的成本呈上升趋势，自动化测试系统迅速地成为产品开发流程中一个重要的部分。这篇“设计下一代自动化测试”的文章描述了一些迫使工程团队减少测试成本和时间的挑战。这篇文章还深刻地洞察了测试管理人员和工程师们如何通过建立模块化软件定义型测试系统来克服这些挑战。这种测试系统在减少总体成本的同时，显著地增加了测试系统的吞吐量和灵活性。

Today's test engineers face a new set of pressures. The product development environment they face is as follows:

Product designs are more complex than previous generations

In order to remain competitive and meet customer requirements, development cycles are required to be shorter and shorter

Product testing costs are increasing, while budgets are shrinking

Increasing design complexity

Today, the most obvious trend in test and measurement is the increasing complexity of devices. For example, the consumer electronics, communications and semiconductor industries continue to require the integration of digital images/video, high-fidelity audio, wireless communications and Internet connectivity into a single product. Even in cars, complex automotive entertainment and information systems, safety and early warning systems, and control electronics on the body and engine are integrated. The design of the test system not only needs to be flexible enough to support a wide range of tests on different product models, but also needs to be able to be upgraded to provide more test points required for new test functions.

Shorter product development cycle

Due to the competitive nature of wanting to continuously improve new products and technologies and have the first market share, the design and test engineering team can only continuously shorten the product development cycle. To this end, the engineering team must design new test strategies to reduce test time and improve test efficiency from design to production.

Increasing test costs and decreasing test budgets

Adding device functionality usually results in a more expensive and time-consuming test process. However, the cost of building each function is decreasing, which forces engineering departments to reduce costs and budgets, as shown in Figure 1. Engineers must improve test strategies to reduce total costs by increasing test system throughput, reducing maintenance and upgrade costs, and reducing required capital investments.

Figure 1. Data from SIA shows that over time, the cost of silicon (or device functionality) decreases, but the cost of test continues to increase.

2. Increasing test costs and decreasing test budgets

To meet the challenges of increasing device complexity, shortened development cycles, and reduced budgets, test managers and engineers are forced to abandon traditional test design strategies based on traditional box instruments or "big iron" proprietary ATE systems. These stand-alone instruments lack the flexibility necessary for software processing, and the user interface is defined by the manufacturer and can only be updated by the manufacturer through firmware. This makes it difficult to execute tests that are not defined in the instrument firmware and tests for new standards; or to modify the system when requirements change. Because these devices were originally designed as stand-alone instruments, they lack necessary integration capabilities, such as data streaming and synchronization functions. Proprietary ATE systems (such as highly integrated product chip testers) can provide the required performance, but the cost is quite expensive, which may cause engineering teams to abandon and redesign the system prematurely.

In response to these situations, test managers and engineers are implementing a modular, software-defined test architecture based on widely adopted industry standards that provides:

Greater test system flexibility: scalable to multiple applications, business units, and product stages

高性能的结构：可以显著提高测试系统吞吐量，并提供与不同仪器厂商之间的密切联系和集成，包括精密直流信号、高速模拟和数字信号和射频信号的生成与

analyze.

Lower test system investment: Reduce initial capital investment and maintenance costs while increasing equipment utilization across multiple test requirements

Longer test system life: Based on widely adopted industry standards, allowing technology upgrades to improve performance and meet future test needs

As a leader in automated test, NI is committed to providing product engineers with the hardware and software they need to design the next generation of automated test systems. This in-depth developer's guide contains the information needed to design the next generation of automated test system architecture. The introduction describes a test system architecture as shown in Figure 2, providing engineers with strategies to cope with a series of challenges such as increasing device complexity, shortened development cycles, and reduced budgets.

Figure 2. NI provides a complete set of hardware and software solutions for designing automated test systems.

3. Hierarchy 5: Automated Test System Management Software

Automated test systems need to implement a variety of tasks and measurement functions: some of these tasks and functions are related to the device under test (DUT), while others are common to each DUT. In order to minimize maintenance costs and ensure the life of the test system, it is very important to implement a test strategy that separates DUT-level tasks from system-level tasks, so that engineers can quickly reuse, maintain and modify test programs (or modules) throughout the development cycle to meet specific test requirements.

In all test systems, there are different operations depending on the device under test, and there are also operations that are common to all devices under test, such as system-level tasks.

Different operations for each device

• Instrument configuration

• Measurement

• Data collection

• Result analysis

• Calibration

• Test the operation of the module common to each device

• User interface

• User Management

• DUT tracking

• Test process control

• Storing results

• Test report

Some companies have written their own test executives and allocated valuable engineering resources to develop test management software from scratch. This often results in lost productivity and time tied up in maintaining the software. To maximize productivity, engineering teams should leverage commercially available test management software, such as NI TestStand software, to reduce the development of common operations for each device. By leveraging this software, engineers can focus on the development of proprietary operations for each device. For more information, see the white paper Developing a Modular Software Architecture. [page]

4. Structural level 4: application development software

In the test system architecture, application development environments (ADEs), such as NI's LabVIEW and LabWindows/CVI, play a key role. With these tools, test system developers can communicate with a variety of instruments, integrate measurements, display information, connect to other applications, and so on... The ideal ADE for developing test and measurement applications needs to provide a series of application requirements such as ease of use, efficient compilation performance, integration with a variety of I/Os, and programming flexibility. Ease of use is not only about how quickly you can get started and use it. With easy-to-use ADEs, developers can easily integrate processing routines with a variety of measurement devices, create complex user interfaces, deploy and maintain applications, and modify applications when product designs are improved and system needs are expanded.

For more information, see the Choosing the Right Software Application Development Environment white paper.

5. Structural level 3: measurement and control services

Measurement and control services provide connectivity, system configuration, and diagnostic tools to various hardware resources in the system, which is critical. For example, NI Measurement and Automation Explorer (MAX) can automatically detect hardware resources, including data acquisition, signal conditioning hardware; GPIB, USB, and LAN-controlled instruments; PXI systems, VXI devices; modular instruments, etc., so developers can configure them in one place. Integrated diagnostic tests ensure that the device functions properly, and test panels provide developers with a quick way to check the functionality of the hardware before starting programming. Measurement and control services also provide integration with the application development software layer through application programming interfaces (APIs), so that developers can easily program their devices. In fact, the components of this service software - hardware drivers, application programming interfaces (APIs), and configuration managers must be seamlessly integrated into the ADE to maximize performance, improve development productivity, and reduce total maintenance costs.

6. Structural level 2: calculation and measurement bus

At the heart of every automated test system is a computer (in the form of a desktop PC, server workstation, laptop, or embedded computer that works with PXI and VXI, etc.). An important aspect of using a computing platform is the ability to connect (and communicate) with a wide variety of instruments in the test system. There are many different instrument buses available for standalone or modular instruments, including GPIB, USB, LAN, PCI, and PCI Express. These buses have different capabilities, and some are more suitable than others for specific applications. For example, the GPIB bus is widely used in instrument control and has wide availability for instruments; the USB bus offers wide availability, easy connectivity, and high throughput; the LAN bus is very suitable for distributed systems, and the PCI Express bus provides the most efficient performance.

The widespread use of personal computers has led to the continuous advancement of high-performance internal buses, including PCI and PCI Express buses, which have the lowest latency and highest data throughput or bandwidth. The PCI bus provides a bus bandwidth of up to 132MB/s, while the PCI Express bus, as an evolution of the PCI bus, provides a bandwidth of 4GB/s to meet the growing bandwidth requirements while being fully compatible with the PCI bus in software. Figure 3 explains the latency and bandwidth performance of the most popular instrument control buses.

Figure 3 Comparison of various instrument control buses. PCI and PCI Express buses provide better bandwidth and latency, that is, better overall throughput performance.

7. Architecture Level 1: Measurement and Device I/O

Fundamentally, there are two types of instrument architectures today—traditional instrumentation and virtual instrumentation. Figure 4 illustrates the similarities between the two architectures. Both have measurement hardware, a chassis, power supplies, buses, processors, operating systems, and user interfaces.

Figure 4. Traditional instrumentation and virtual instrumentation architectures have similar hardware components; the main difference between the two architectures is where the software resides and whether the user can access it.

From a hardware perspective, the most obvious difference is how the components are organized. Traditional or stand-alone instruments put all the components in the same box for each separate instrument. The measurement functions, analysis, display, and control of the instrument are defined by the vendor.

In contrast, modular software-defined virtual instruments integrate common measurement hardware to help users define their own measurements and user interfaces in software in addition to using standard functions. Using a modular approach, engineers can define the measurement functions of the test system and build a scalable system to meet future needs. With a modular, software-defined approach, users can make custom measurements, measure for emerging standards, or modify the system when requirements change (for example, adding instruments, channels, or new types of measurements). This combination of flexible, user-defined software and scalable hardware is the core of modular instrumentation.

8. Summary: Designing a new generation of automated test systems

设备复杂性增加、开发周期缩短和预算降低使工程团队有机会重新评估现有的自动测试策略，并且寻找出提高效率减少成本的方法。在设计新一代自动化测试系统时，加入可以增加系统灵活性、提供更高测量和吞吐量性能、降低测试系统成本并且延长寿命的策略是非常重要的。模块化的软件定义型自动测试系统克服了以往的基于独立式仪器或成本高昂的私有ATE系统解决方案的不足之处。模块化的硬件平台基于广泛采用的工业标准平台，诸如PXI等，允许工程师们开发可扩展的测试系统，将各个仪器供应商提供的功能紧密地集成到一起。另外，它还允许工程团队集成现有的设备投资来降低实现的初始成本。利用最新PC技术(诸如多核处理器和PCI Express总线)的软件定义型测量，新一代自动化测试系统可以显著提高吞吐量性能，并且可扩展以满足不同产品阶段和业务部门的需求。

Many companies have implemented a modular, software-defined test system strategy and demonstrated a return on their investment. For example, Microsoft designed a test system for the Xbox 360 controller based on NI LabVIEW and PXI modular instrumentation that is twice as fast as the previous generation of test systems. The US Air Force developed a test architecture to support their advanced fighter aircraft. Using a PC-based software and hardware architecture, they reduced costs and reduced the size of the test system by half. Sanmina-SCI built an FDA-approved pharmaceutical device test system using NI TestStand and PXI products, exceeding their requirement to test 83,000 devices per week and exceeding their production requirements by 95%.

[page]

9. NI related products and white papers

As a leader in the field of automated testing, NI is committed to providing product engineers with the hardware and software they need to design a new generation of automated test systems.

software:

•NI TestStand test management framework

• LabVIEW graphical programming language

•LabVIEW SignalExpress interactive measurement software

hardware:

• Modular instruments (oscilloscopes, multimeters, RF modules, switches, etc.)

• Multifunctional data acquisition

• PXI system components (chassis and controllers)

• Instrument control (GPIB bus, USB bus and LAN)

Technical White Paper

NI provides the Developer's Guide to Designing Next-Generation Automated Test Systems. This guide is a collection of white papers that are designed to help develop test systems that reduce costs, increase test throughput, and scale to meet future needs.