Avoiding Embedded Software Defects Through Testing

When a requirement is expressed, the expectation of business people is to achieve something that works. But there has been little agreement in business or software history on whether the final product actually works as expected. From an engineering perspective, any issues related to performance, reliability, security, etc. that do not meet the requirement should be considered a defect. From a business perspective, avoiding these issues is considered part of the expressed requirement.

The reasons for this disconnect between business expectations and the evolution of requirements during development and testing go much deeper than the literal meaning. It is a classic symptom associated with a traditional organizational culture where engineers believe that business people do not understand the complexities of the software development process. At the same time, the business side of the organization believes that engineers suffer from Smart Developer Syndrome (SDS) - a pathological disorder where engineers do what they think is best because they are smart.

Implementation strategy

So what is the solution? How can an organization bridge the gap between business goals and development processes? Strategy is key to ensuring engineers deliver software that meets expectations and embeds quality awareness into the software development process. By implementing a policy-driven development approach, organizations can reduce risk, increase productivity, and lower costs throughout the software development lifecycle, from creation to support. Policy-driven development is based on three core actions:

1. Define expectations in the form of policies to guide engineers on how to develop and test software

2. Train engineers on the business goals that drive these strategies

3. Automatically monitor policy adherence with the help of appropriate infrastructure

Clearly defined enforceable and measurable policies create consistency and precision, ensuring that a rigorous quality process defined in writing is followed through. In addition, policy enforcement automation is necessary to achieve clarity and measurability efficiently.

Guidelines or strategies?

If you ask an organization about their development strategy, many will quickly point to their best practices and guidelines. But guidelines and strategies are not interchangeable. Guidelines describe recommended behavior, while strategies describe desired behavior. All strategies have the following three components:

● The policy must be automatically enforceable: It is not feasible to manually check whether engineers follow the policy. Some automatic mechanism must be used to check for violations and alert.

● Policy violation alerts must be targeted: only engineers whose code violates the policy should be alerted. This way, engineers who comply with the policy can continue to work without interruption.

● An automated workflow must be developed to correct policy violations: The methods by which engineers discover and handle policy violations should be defined and automated.

The good news for embedded software organizations is that technology exists to automate policy enforcement, which, if done correctly, can improve quality, increase throughput, and reduce development costs and risks.

Matching strategy and process

"The Hoover Dam must be leak-proof" is a reasonable strategy that establishes an organization's goals in plain, clear language. Now consider the strategy of "check for leaks once a day." On the surface, this strategy appears to move the organization toward its goals, but closer inspection reveals the type of shortcomings that are prevalent in many embedded development shops.

Where this strategy falls short is that it allows the problem to happen first. Rather than supporting the goals of the first strategy, it simply checks for symptoms associated with not meeting the organizational goals identified in “Hoover Dam Must Be Watertight.” Unfortunately, many development testing practices follow this model of treating symptoms rather than the cause.

A much better strategy is to “strengthen the dam’s weakest point every day.” By matching policy and process to desired business objectives, you can create a framework to ensure and maintain the strength, security, and reliability of embedded software.

Figure 1: Development and testing platform for static enhancement coding strategy.

Strategy and Development Testing

Development testing is the continuous integration of software testing activities throughout the development process. It can reduce technical errors and prevent defects, which is the key to improving efficiency and reducing risks during the application life cycle. The development test platform is the infrastructure for implementing consistent application automation of development test practices, including static analysis, unit testing, peer checking, coverage analysis, runtime error detection, and accurate and objective measurement of productivity and application quality.

Each organization may find different development testing activities more valuable in specific circumstances. Policy-driven development testing can ensure that these development testing activities are correctly implemented in a value-focused, risk-aware, and verifiable manner. A development testing platform accepts inputs in the form of policies and translates them into a set of rules that define one or more processes. Such a platform automates the process, tracks adherence to the policy, and verifies that the results meet expectations. In other words, a development testing platform can give business leaders greater control over the development process and better visibility into the effects of process changes.

Conclusion

It is important to note that development testing is not a replacement for quality assurance (QA), but rather a process to confirm that software functionality correctly implements the original intent. Development testing is also not just a "shift left" of the testing process. The emergence of development testing represents consistency with the more iterative and flexible development processes currently adopted by industry organizations. Development testing is a model of software development behavior that, if implemented consistently, will create an environment where software defects cannot survive.