Verification plan improvement method to improve verification efficiency
[Copy link]
Project management is all about planning and execution. If everyone has planned their verification projects well, why are there quality issues and schedule delays? A good plan should include detailed goals described in measurable indicators, optimal resource utilization, and realistic schedule estimates. | | | | | Figure 1: The gap between expected and actual efficiency in a verification project. | | | | | | Figure 2: Validation team plan development flow chart. | | | | | | Figure 3: An actionable plan drives the validation process. | When managers and their teams develop plans, they often fail to address common issues that lead to schedule delays, low resource productivity, and poor product quality. Most verification plans focus only on task performance rather than how the verification problem is defined, regardless of the solution. This almost inevitably leads to loopholes in the verification process, which causes design defects (bugs) to be overlooked, and fixing these defects will cause delays in progress or lead to serious resource constraints and inefficiencies. However, we can avoid these problems and get twice the result with half the effort by simply improving the plan. Why the Validation Plan is Incomplete Due to time pressure, most verification teams do not conduct comprehensive verification planning. Instead, they skip this step and jump directly into the pre-design verification environment development process. This makes the plan immutable or inflexible, and such a plan is of little use because it is difficult to maintain the connection with the actual project. In other words, such a verification plan is not a plan at all, at best it is just a set of incomplete discussion notes. As the project progresses, the team starts working and realizes what they should achieve, such a plan will eventually become useless. The solution to the above problem is to make the verification plan an executable part of the verification process itself. When the verification process automation tool reads the verification plan, the verification plan becomes executable. This approach can be used to organize and generate project status reports and become the basis for analyzing data to decide the next action. In this way, the value of the plan is maximized, and it will serve as the beginning and checkpoint of the verification process throughout the life cycle of the project. By automatically testing the integrity of the verification process with the verification plan, the return on investment in developing and maintaining it can be directly increased. In this way, when changes must be made in the project, these changes will be recorded, tracked and tested by the updated executable verification plan, making the verification plan a valuable part of the verification project from specification definition to completion. Simply by planning better and using executable plans to test the integrity of the project, you can improve project quality, increase the predictability of the project schedule, and improve resource productivity. In other words, with better planning, the verification team is more likely to get the results they need. Using a plan throughout the project can help identify problems earlier, which is one of the keys to keeping the project on track. At the same time, tracking the progress of the project according to the planned standards can also enable all members of the team to better manage themselves, thereby improving productivity. What makes a good plan? Verification plans must shift from focusing on "how to verify" to "what to verify." By defining the important goal of what needs to be verified, the verification team can ensure the integrity and balance of the plan. At this point, the verification plan is more than just an engineering specification of how to complete verification. a. Validation Planning Basics From a high-level process perspective, a verification plan is quite simple. The basic steps include: 1. Analyze device specifications 2. Scoping the verification objectives 3. Determine the feature set of the design 4. Design a detailed coverage model 5. Use aggregate metrics to track verification progress 6. Estimate work and progress based on past metrics Often, team leaders tend to follow this process to plan their validation, but they miss at least one critical aspect at each step, cutting corners to save time and risking the project. As a result, they often lose. b. Analyze device specifications All projects always start with several specifications. The marketing process can help understand user needs, the management process defines resources and progress, and the constraints of self-build versus purchase; finally, the system engineers, software and hardware engineers, and the verification team will develop verification implementation specifications to guide the project. The biggest risk for a project is the inability to foresee changing goals. Some teams try to find out all the requirements at the beginning and then complete the entire project sequentially. This method is sometimes called the waterfall method. In fact, there are often repetitions in this process, and the risk of the project lies in the inability to fully predict the impact of repetitions and control the frequency of repetitions and the burden they add to the project. c. Define verification objectives The most common mistake during the definition phase of a project's verification objectives is the failure to engage in extensive verification discussions with all stakeholders. Defining and documenting verification objectives is the only way to determine whether the project specifications are well thought out and fully understood. Planning the details of a verification project after the project has begun will be inefficient and unpredictable. Management must invest sufficient time in the early stages to develop accurate and complete verification goals. The expansion and change of requirements is another issue, which will be discussed in the next section. All possible stakeholders must participate in setting verification goals, including management, marketing, system designers, hardware designers, software designers, and verification teams. Because only when the entire team participates in the conversation can the requirements of the entire project be clarified. Market requirements can never be complete because they only address market key factors. By definition, management resources and project schedule constraints are often in conflict, and as project goals are fully defined, the contradiction between the two must be mitigated. System engineers should put forward technical views that are closest to market needs, and software and hardware engineers should also provide valuable insights into the relevance of project implementation. The verification team is the core of the entire discussion group. It is they who constantly challenge other members of the project team to see and make up for the shortcomings of the project, and at the same time let them recognize which features cannot be verified, which is the key goal of verification-oriented design. If any of these points are missed, there will still be loopholes and risks in the project, which will inevitably lead to product quality problems, excessive costs or delayed schedules. d. Determine the feature set of the design Superb testing techniques have become a difficult part of verification, and metrics are a way to measure the verification process. Engineers have traditionally performed verification by designing test software, implementing checkers, and using code coverage tools. Test lists as a metric have fallen into disuse because they are too complex to specify and implement. Directed test software is sometimes easier to design, but they are not adequate for the scale of modern SoCs. Verifiers are not suitable for use as metrics because they only check for incorrect system behavior and do not record observed good behavior. Also, code coverage is only loosely related to behavior because the context of the device is not recorded for each line of code. Leading electronics development teams in the industry agree that functional coverage is the most accurate metric for verification. Functional coverage is part of coverage-driven verification (CDV), an approach that allows development teams to measure how much verification work they have actually completed, rather than how many (mostly redundant) simulation cycles they have performed. The language specification for functional coverage is designed to match the requirements specification obtained during the definition process. The assertion language in the coverage specification can also be used to capture implementation assumptions as part of assertion-based verification (ABV). Full coverage is the only way to fully understand the status of a project, while functional coverage is directly related to design characteristics. Assertion coverage is related to both functional coverage and implementation completeness, and hardware and software code coverage can tell us how well the design is completed. e. Design a detailed coverage model The result of the definition process is a set of features that need to be verified, and the specification must describe these features in order to be measured. Coverage is used to define the metrics for verification. The result of the definition is two types of metrics: explicit specification metrics and explicit implementation metrics. Explicit specification metrics are selected by engineers from the specification, and explicit implementation metrics are selected by engineers from the RTL implementation when the RTL is available. A typical oversight during the planning phase is a lack of careful study of the coverage model. The coverage model must be complete enough to represent all the features that need to be verified, yet concise enough so that the development team and verification tools can complete the task within the given time frame. This judgment is critical because it is much better to verify 100% of the majority of critical functionality than to have critical functionality missing from a very detailed coverage model. f. Use the collection method to track the verification process Management is nothing more than planning and execution, and tracking project progress is the only way to know whether the project team is executing. The question is how to select a small number of metrics that can represent project progress and reveal problems and risks in the project? Typically, the project team will define some important stages (milestones) to roughly outline the progress of the project. To maximize the efficiency of the project team, it should be noted that these milestones must be planned to move the most time-consuming or risky parts of the project forward. However, simply verifying a specific feature is no longer sufficient. More and more teams define milestones that expose possible system integration problems. They provide the right subset of functional modules to enable early system verification, which requires detailed definition of the features that must be present in each milestone. Use coverage to define these characteristics, and carefully define and track the important stages that have some characteristics of the system. In this way, the important stages of the project are no longer just a mark in the schedule as a sign of stage completion, but have a more important meaning, which is to effectively reduce the quality and schedule risks of the project. There are also some manually tracked key phases, including first working simulation, and generation, checking, and completion of the coverage model. These two key phases are also called implementation tracking. Anticipation and completion of these two key phases often overwhelm teams that are using coverage-driven verification for the first time. An executable verification plan can define the conditions for each of them, thereby improving the predictability of this critical early stage of the project. g. Estimate progress using past metrics Creating an engineering schedule is an art that reflects the experience of the team responsible for the project plan and reminds them of how long it took to complete similar tasks in the past. The more standards are used in functional verification, the more schedule estimation becomes an engineering art that makes the verification process visible. The problem is that few teams have access to these standards in the past, and few teams have invested in creating a model that can quantify and estimate project schedules. Some of our experienced customers are tracking more standards in addition to the coverage standards in an effort to further improve the accuracy of their schedule estimates. The metrics they track are used to measure the time it takes to implement, debug, and integrate each level of IP and its extraction in the design and verification environment, with different data points for different sizes of IP. They also need to estimate the different impacts of developing new IP versus reusing old IP, the impact of the project team's engineering experience and engineering skills on the project, and the trade-offs between automated verification processes and manual verification management. They not only look at hardware and software, but also at module, chip, and system-level issues. If there are no actual historical metrics, then the project team can come up with estimates. But today's schedules are often estimates of tasks, not metrics. By breaking down the estimates into more detailed levels, the project team can uncover the assumptions in their justifications. In addition, they will give detailed details about how each person will complete their work. This allows us to build a model or equation that uses historical metrics, supplemented with existing project estimates, to create a detailed project schedule. We then consider the accuracy of these estimates and the model itself, and make project schedule commitments based on the schedule estimates. With the right automated tools, project teams can compare these estimates to the overall project, task, and detailed standards, and learn from them to help plan the next project. Changing requirements Every engineer knows that requirements are always changing, and even delays in schedules do not change this fact. Many electronic products serve user markets that are more closely related to the seasons than to the end of the project. Managers are always looking for ways to control the change process and make projects more predictable. The key question is "How late can we introduce changes and still release a high-quality product on schedule?" The historical verification plan is a key part of the automated mechanism that drives and manages verification from specification development to project closeout. The automated verification mechanism includes: 1. Executable verification plan 2. Coverage and assertions for measuring verification progress 3. Verifiers and assertions for detecting and reporting violations 4. Stimulus generation Better validation planning and adopting an actionable validation plan can provide several benefits. 1. Improve product quality It is obvious that good planning leads to high-quality products. Every client we have helped improve their verification plan has found their problems after the process. Even for projects that are going well, the value of rigorous verification planning is obvious. Sometimes the greatest value of a verification plan is that it creates a common process and terminology for the plan. 2. Improve the predictability of project progress More accurate work estimates and earlier detection of deviations from the project plan enhance the predictability of the project schedule. At the same time, detailed setting of important stages can also help to identify key issues in the project earlier, giving the verification team the opportunity to successfully address these challenges in the normal course of the project. 3. Improve team efficiency An executable verification plan can help a team communicate more effectively and help them focus on the key factors of the project more easily. Standard reporting methods also increase the efficiency of the team and the management efficiency of the management, so that resources can be concentrated on the most critical tasks. By Steve Brown Marketing Director Cadence Design Systems Enterprise Verification Process Automation
| 
提升卡
变色卡
千斤顶
