Hardware-centric design approach to building high-reliability medical electronic devices

Almost everyone who uses a cell phone has experienced a few dropped calls. While system failures or glitches in these and other consumer products are inconvenient, they are not catastrophic. However, a system failure in medical electronic equipment can be life-threatening, which is why medical devices, as well as the components integrated into these systems and the software running in these devices, must pass rigorous testing and meet the stringent requirements of the U.S. Food and Drug Administration (FDA).

To ensure that our new design would perform reliably and successfully pass the FDA approval process, we adopted a highly structured design approach called the Scrum/Sprint development process. In addition, by reducing the number of functions implemented in the software, we were able to reduce the chances of software errors. We implemented these functions in Xilinx FPGAs. To better understand this approach, let’s first analyze the design process for medical devices.

Three-stage life cycle

The FDA has established strict regulations, requirements, and guidelines for medical electronics to ensure the safety of the general public. Among these regulations, the FDA has established strict requirements for the life cycle of medical devices (Figure 1). In general, electronic product companies must comply with the requirements of the above regulations in the following aspects, including all components, parts, or accessories of the medical device, any software used in the production process of the medical device, and all software used by the device manufacturer's quality system, such as programs that can record and maintain the device's history.

Figure 1. Diagram of the entire life cycle of medical device design as defined by the FDA.

We can divide the entire life cycle of medical devices into three main stages. The first is the early stage of the product life cycle (Figure 2), which is the least systematic stage of all three stages, during which companies mainly focus on research and development in theory and conception. The duration of this stage ranges from a few weeks to several years, which is closely related to the complexity of the system that the company is preparing to develop.

Figure 2: Applying virtual instruments and HEI design methods in the early stages of a product's life cycle can provide a clear understanding of the problems that need to be solved. (print)

产品整个生命周期初期阶段的基本组成部分是数据采集与分析。通常情况下，研究人员与产品设计规范小组会使用多种工具来精简流程。在这个阶段，HEI通常会采用美国国家仪器(NI)公司的LabVIEW产品来调整FPGA的I/O。一旦充分理解问题，我们就能设计出解决方案。对于器件开发及原型设计，我们结合直观的图形编程，重复使用数学与信号处理功能来开发新的算法。然后，通过使用商用套装硬件，我们对现实世界的数据进行参考来对算法的性能进行验证。在许多情况下，我们都能使用NI提供的基于FPGA的原型设计平台来实现最终器件的实验原型。确切地说，我们能够将LabVIEWReal-Time模块和FPGA模块同NI CompactRIO结合使用，以便在算法设计和器件原型阶段之间迅速实现叠代。使用硬件套件进行原型设计，不仅能够显著缩短硬件开发与集成时间，而且还使我们能够将更多精力集中到交付功能强大、性能可靠的软件设计中。

Figure 3. Software design review and verification is usually completed in the middle of the product life cycle.

The second phase of the medical device life cycle can be called the middle of the product life cycle (see Figure 3), which can fully meet the needs of the designed device in terms of productization, verification, review and manufacturing. The focus of this phase is to develop accurately defined specification documents that clearly state quantifiable requirements. Once these specifications are defined, a clear mapping relationship can be established between the specification documents and the actual implementation code.

Figure 4. The traditional “waterfall” development process requires each stage of development to be completed before moving on to the next step.

In today’s complex medical device market, customers must accelerate time to market and get there first. Many companies are using the traditional “waterfall” development method to accomplish this. In the “waterfall” development method, the design team needs to complete each step of the design process before moving on to the next step (Figure 4). The waterfall method relies heavily on having complete and accurate specifications at the beginning of the project. However, in the medical device market, more often than not, requirements will continue to evolve as the product is developed. This requires a process that can take evolution into account. HEI’s Scrum/Sprint development process is an ideal solution that can fully address this problem.

Figure 5 – In the “Scrum/Sprint” process, only high-level system architecture and specifications are required to start a project.

Figure 6 - The project team determines the "Sprint Backlog" work tasks for each "Sprint" in the cycle, expands and distributes them. The project team then evaluates progress and removes relevant obstacles in daily "Scrum" meetings. The team delivers product features to the customer at the end of each "Sprint".

In the Scrum/Sprint process, we only require the high-level system architecture and specifications to start the project. We break down the project into "Sprints" that are 4 to 6 weeks long. In each "Sprint", we identify all the tasks we think the process will require and put them in a "burn-down" list. Figures 5 and 6 show the relevant process diagrams. HEI uses the Scrum/Sprint development process throughout the company, which not only speeds up our development process by 30%, but also enables us to complete the implementation of new products several months ahead of schedule. Table 1 compares waterfall development and Scrum/Sprint development solutions.

The third and final stage of medical device development is the end of the product life cycle (Figure 7). This stage requires very little engineering work, but customer feedback and market success will help drive the concept formation of the next generation of products, and the life cycle will enter a new cycle.

Figure 7 - The later stages of the product life cycle involve bringing the product to market, obtaining feedback, and helping to determine the features of the next generation of products. This completes the work of this cycle and moves it into the new ideation stage.

Using a Scrum/Sprint product development process, combined with FPGA-based hardware packages and advanced FPGA software design tools from R&D to manufacturing, HEI is able to rapidly develop derivative technologies for future products. We have found that the common core architecture we use in the development of multiple products can be used in many cases. For example, the pump controller architecture that can be scaled for IV and infusion pumps can also be used in other design projects that can control motors to enable blood transfusion management.

Why Hardware is Better Than Software

In order to use this approach effectively and further speed up the design process, it is necessary to change the way designs are conceived, from being software-centric to being more hardware-centric. As you may know, medical device recalls reached a new high in 2008, up 43% from 2007. FDA experts believe that there are two main reasons for recalls: defects in raw materials used in production and problems in software development. It is relatively easy for companies to manage the quality of raw materials, but it is much more difficult to solve the quality problems of software. As the amount of code in device software increases rapidly, the problem will only become more serious. This problem is particularly prominent after the FDA's Consumer Health and Safety Division stated that medical device designers have many safety responsibilities.

At HEI, we think there is a potential solution to this problem, but rather than doing more testing, code review, and introducing more processes, we can simply write less software and offload more logic to hardware components such as Xilinx FPGAs. Let's look at common reasons why software failures occur and how we can use FPGAs to solve these problems.

Eliminate deadlock

Most modern devices need to be able to handle multiple tasks at the same time, but most embedded processors still have only one processing core. This means that the processor can only execute one instruction at a time. At the same time, parallel processing is not much better because they still have to share the main system CPU. In addition, other shared resources such as communication drivers, hard disks and user interface components can also cause deadlocks - two or more processing processes wait for each other to release resources.

Because deadlock conditions often depend on multiple processes and usually require events to be synchronized in order, deadlocks are very difficult to reproduce and debug. Most deadlock problems are difficult to find with unit testing alone. They are often found by code reviewers, skilled system testers, or sometimes even by luck.

In contrast, with FPGAs, independent processes have their own physical circuitry, so there are no shared resources. At each clock signal, the combinational logic not only latches in each circuit, but also stores the associated values ​​in independent registers. Because the processes do not depend on other resources, deadlock issues cannot occur. This gives you more confidence in the results of simulation and unit testing because unknown factors such as resource contention are no longer an issue.

Incompatible middleware

When developing embedded software, you rarely have to write every line of code from scratch. There are many tools that can help firmware designers improve their work, such as simple drivers, network protocol stacks, operating systems, and even code generation tools. Although these systems are usually fully tested independently, software defects can still exist. Because of the various ways to combine tools and libraries, the possibility of combining components in new ways is very large.

For this reason, the FDA requires that for all software packages used in medical devices, companies must verify the software protocol stack based on the specific use case of their specific design. What does this mean? For example, if we are using a signal processing library that contains a fixed-point fast Fourier transform and need to detect the presence of a specific frequency component, we do not need to verify that the FFT returns the correct value for all possible inputs. However, we do need to verify that the FFT returns the value we expect for all valid inputs that meet the specifications. For example, if we are only detecting tones that can be heard by the human ear, there is no need to test whether the value returned is correct for inputs above 20KHz.

However, seemingly independent software components are not necessarily so. Therefore, if we use a software stack in conjunction with an SPI driver and a real-time operating system (RTOS), we need to validate all of these components to be confident in the FFT. If the FFT delivers valid output to the SPI driver, but the SPI driver fails systematically, then there is obviously a problem. Then, if we decide to adjust the SPI driver, we need to validate the entire software stack again. This is cumbersome, and one faulty component can slow down the development of the entire system. For this reason, at HEI, we reuse as many proven combinations of middleware and RTOS drivers as possible. For example, we can use middleware drivers on the NI Single-Board RIO platform.

In addition to validating the software for each specific use case, we also need to validate all third-party intellectual property (IP) we use in our FPGA-based designs. However, once we have completed the validation of all our use cases, we will have confidence that the IP will work as expected when integrated with other components. Let's look at our FFT example above again. If we use an FPGA, we can obtain or generate the FFT IP core just as we would with software, and verify its numerical correctness for each use case. However, the risk of intermittent failures is greatly reduced because we do not need all the processor middleware required for software-centric designs. In this way, the RTOS and SPI drivers are no longer their own IP cores because their operation does not directly affect the FFT. In addition, if we adjust the implementation of the SPI driver, we do not need to revalidate the unaffected parts of the system.

Buffer overflow management

Another example of how we can use FPGAs to mitigate errors that are common in software-centric systems is buffer overflow management. A buffer overflow occurs when a program attempts to store data past the end of the memory allocated for its storage, and the program overwrites some adjacent data that it should not have written. This is a difficult defect to detect because the overwritten memory can be accessed at any time in the future, and the situation may or may not cause an obvious error. One of the more common buffer overflow situations in embedded designs is caused by some high-speed communication, perhaps from the network, disk, or A/D converter. If the communication is interrupted for too long, the buffer will overflow, so we need to solve this problem to prevent various conflicts.

We can use FPGAs to manage buffer overflows in two ways. In the first example, we use FPGAs to manage circular transfers or double-buffered transfers, which can offload the real-time processor. In this case, the FPGA can be used as a coprocessor to reduce the interrupt load on the main processor. This is a common configuration, especially in high-speed A/D converters.

In the second example, we use the FPGA as a safety layer for protection functions, so that the I/O for the patient passes through the FPGA before reaching the processor. In this case, you can add additional safety logic to the FPGA so that in the event of a software crash on the processor, you can put all the outputs into a known safe state. The FPGA can act as a watchdog, and its logic can ensure that the risk to the patient is minimized in the event of a software failure. By introducing the FPGA into the main signal chain of the device in the architectural design, you can use one or both of these methods to deal with buffer overflows so that you can better handle the situation when it occurs.

事实上，越多地将软、硬件总体系统功能移至FPGA中，就能越快地完成设计流程，并最终越快地完成我们的设计在客户最终产品上的验证工作。如果我们能更快地验证我们设计方案在总体系统上运行的可靠性，那么我们的客户就能更快地验证整个最终产品，进而将其交付FDA审批。这一过程意味着我们的客户能够显著加速其产品的上市进程，改善患者的生活质量，甚至拯救生命。

If we implement the design using an ASIC process, we need to wait several months for the foundry to produce the hardware. The additional steps of verifying the physical design of the ASIC, creating masks, and sending the design to production introduce more opportunities for errors and defects. If the design is wrong in any of these steps, the result is a significant delay in the time to market. Since the FPGA is already produced and fully tested, we only need to worry about our design and software and ensure that the design meets the customer's specifications. The combination of "Scrum/Sprint" methodologies, hardware-centric ideation, the use of highly reliable tools, and the choice of FPGAs over ASICs allows us to differentiate our customers and make a difference for our customers' customers, the patients.