Software testing is important in embedded systems

In the world of embedded systems, it is not just the technology that is evolving, the tools and methodologies used to develop that technology are also maturing and improving.

In the early 1980s, I developed software for a small metrology company, applying engineering mathematics to coordinate measuring machines (CMMs). Our development lifecycle essentially treated production software as a sandbox. We would start with production code, add functionality, perform some fairly basic functional testing, and then deliver.

In such a small company, our engineering team naturally includes both software and hardware experts. In hindsight, it’s surprising that while the software we developed did require extensive customer support, there was almost no similar culture for the hardware we ran on.

Software development is an engineering discipline

Part of the difference between software and hardware support is a result of the original development process. However, the extensibility of the software and the resulting continuous expansion of functionality also plays a role. In short, there are many more wrong ways to do it than right ways, and this characteristic requires that it be treated as an engineering discipline.

None of this is new. Leading aerospace, automotive, and industrial functional safety standards such as DO-178, ISO 26262, and IEC 61508 have required this approach for years. However, having an engineering discipline mindset is critical to benefit from today’s state-of-the-art development and test tools.

The importance of software testing has recently been demonstrated by the development of ISO/IEC/IEEE 29119, a set of international software testing standards that can be used in any software development lifecycle or organization.

Requirements

Electrical system design often starts with a state machine and an understanding of the different modes of operation for a particular product. Engineers can often map the state machine functionality to logic very quickly and easily. If the state machine becomes more complex, it is often converted to software.

High-level requirements are essential to ensure that the system works properly. Such requirements characterize the business logic and expected functionality and can assess whether the system works as expected. Best practices follow a process from high-level requirements to analysis to coverage, and naturally, requirements traceability tools are designed for this purpose.

In a state machine model, the requirements that characterize each state are examples of high-level requirements. Tracing the execution path through the code to ensure that each requirement is interpreted correctly is one way to check for correct implementation.

Functional safety standards extend this to the concept of requirements traceability. They typically require users to exercise all code against high-level requirements and to account for and test all uncovered cases through low-level testing. More recently, the “shift left” in cybersecurity echoes this message, as shown in the V model in Figure 1.

Figure 1 As the name suggests, the V-model embodies the product development process, which shows the connection between the test specifications at each development stage. Source: LDRA

Test the components, then test the system

In any engineering discipline, it is important to ensure that components work properly on their own before being integrated into a system. To apply this thinking to software, engineers need to define lower-level requirements and ensure that each feature set is doing its job. Engineers also need to ensure that appropriate interfaces are provided to the rest of the system.

Unit testing involves parameterizing inputs and outputs at the functional and module level, checking to ensure that the connections between inputs and outputs are correct and that the coverage logic is followed. Unit testing tools can provide proven test harnesses and graphical representations that connect individual inputs and outputs to execution paths and can verify their correctness.

It is also important to understand the interfaces at the function and module levels. Static analysis tools can display these interfaces and connect the logic at different levels.

Find problems early

Engineers in any discipline will tell you that the earlier you find problems, the less expensive it is to fix them.

Static analysis performs source code analysis to model the execution of a system without actually running the system. Available as soon as the code is written, static analysis helps developers maximize the clarity, maintainability, and testability of their code. Key features of static analysis tools include:

Code-complexity analysis: Understand where your code is unnecessarily complex so engineers can perform appropriate mitigations.

Program Flow Analysis: Mapping the design of program execution - reviewing flow charts to ensure that the program executes as expected.

Predictive runtime error detection: Models code execution through as many executable paths as possible and looks for potential errors such as array bounds overflow and division by zero.

Adhere to Coding Standards: Coding standards are often chosen to ensure a focus on cybersecurity, functional safety, or in the case of MISRA standards, one or both. Coding standards help ensure that the code follows best programming practices, which is application agnostic and is undoubtedly a good idea.

Figure 2 Activities such as static analysis are an expense in the early stages of the development lifecycle, but they pay off in the long run. Source: LDRA

Code of sufficient quality

It’s no surprise that high-quality engineering products are more expensive; adhering to any development process has a cost, and developing the best possible product may not always be commercially viable.

Where safety is important, functional safety standards often require an analysis of cost and the likelihood of failure. Every system, subsystem, and component needs this risk assessment to ensure appropriate mitigation is performed. This makes sense whether the system is for functional safety or information security. If you test every part of the system with the same rigor, you will over-invest in the parts of the system where the risk is lower, and fail to adequately mitigate failures where the risk is higher.

Software safety practices start with understanding what will happen if a component or system fails, then tracing potential failures into appropriate activities to mitigate the risk of doing so. For example, consider a system that controls the guidance of an aircraft, where failure could be catastrophic. Rigorous mitigation activities must be performed with sub-condition coverage to ensure that code is generated correctly.

However, if the aircraft entertainment system fails, the aircraft will not crash, so the requirements for testing the in-flight entertainment system are lower than those for systems that may cause immediate loss of life.

The malleability of software is both a blessing and a curse. It's very easy to make a system do almost anything within reason. However, that same flexibility can also be an Achilles' heel when it comes to ensuring your software doesn't fail.

Even in the business world, while not all software failures are catastrophic, they are not desirable. Many developers working in safety-critical industries have no choice but to adhere to the most stringent standards. But the principles these standards promote exist because they have been shown to make the end product function better. So it makes perfect sense to adopt these principles in proportion, no matter how critical the application.

Although there is a confusing plethora of functional safety standards applicable to software development, the similarities between them far outweigh the differences. All are based on the fact that software development is an engineering discipline that requires us to establish requirements, design and develop to implement them, and test the requirements as early as possible.

Adopting this mindset will open the door to supporting tools across the industry, allowing for more efficient development of higher quality software.

Mark Pitchford, Technical Specialist at LDRA Software Technology, works with development teams looking to achieve compliant software development in environments where functional safety and information security are critical.