This section presents an overview of the verification and validation activities performed throughout the life cycle for complex electronics. The activities and tasks are described in more detail in the corresponding phase sub-section. This page covers:
Validation is the confirmation, through objective evidence, that the system will perform its intended functions. The intended functions, and how well the system performs those functions, are determined by the customer. Did you create the system the customer really wanted? Will the system fulfill the customer's needs? Is this the right system for the customer?
Verification is the confirmation, though objective evidence, that the specified requirements have been fulfilled. Verification tasks all point back to the requirements. Does the design correctly and completely embody the requirements? Is the implementation a correct representation of the requirements? Is the system being built right?
While verification and validation are sometimes used interchangeably, they do represent different ways to ensure the system being built will meet the needs of the customer. If the requirements are validated (shown to be the right requirements), and the system is verified against the requirements, the resulting system will fulfill its purpose. However, getting the right requirements is hard, and requirements can change. Without validation activities throughout the life cycle, you can end up with a verified system that doesn't perform its intended functions.
Validation is mostly a concern at the system level. However, the requirements that are flowed-down to the complex electronics still have to be validated to be the right requirements. The actual system customer is not likely to be involved with validating the lower-level requirements, but the complex electronics requirements can be validated against the higher level requirements and functions.
Verification is best performed in a step-wise fashion, at the end of each life cycle phase. When the complex electronics requirements are completed, they can be verified to be correct, complete, and consistent with the higher level requirements. The design can be verified to embody the CE requirements, at various levels of detail. The physical programmed device can be tested to show that the requirements were correctly implemented. Testing is only one of the last verification activities. It should not be the only one.
Complex electronics adds some particular quirks to watch out for when verifying the devices:
Explain that various V&V related activities and analyses will be mentioned in each phase sub-section. Provide an overview list (taste) of the types of activities.
Verification is a parallel set of activities to engineering design and development. The assurance engineer performs various tasks each phase of the development. The sections below provide an overview of the types of verification activities that will be performed for complex electronics. Specific assurance tasks and analyses are covered in more detail in each life cycle phase.
At the requirements phase, the system or sub-system level requirements are flowed down to the complex electronics. This flow down is primarily the responsibility of the systems engineer, though hopefully the design engineer for the complex electronics will be involved , to prevent requirements being imposed on the hardware that it cannot meet!
|
Verification activity |
Performed by |
|
Evaluate requirements for the complex electronics |
Quality assurance, systems engineers |
|
Safety assessment |
System safety engineer |
|
Requirements review (e.g. PDR) |
All |
|
Identification of applicable standards |
Quality assurance, safety, design engineers |
|
Formal methods |
IV&V or knowledgeable practitioner |
Quality assurance engineers should review the requirements for correctness, completeness, and consistency. High quality requirements are worth their weight in gold. Good requirements make everyone's life easier, because bad requirements are difficult to verify, are often interpreted differently by various people, and may not implement the functions that are desired. Finding out during testing that the device is missing important functionality, or is too slow, is something you really want to avoid.
For safety- or mission-critical devices, formal methods might be used as a verification tool. The requirements can be defined using a special language that allows mathematical proofs to be generated showing that the device will not violate certain properties. Formal methods can be applied at only the requirements level (to make sure you get those correct), or also be used to verify the design when it is generated. Most projects will not use formal methods.
During the design entry phase, the complex electronics functionality is defined in a hardware description language (HDL). The HDL code can be simulated in a test bench and its behavior can be observed. This is an important verification activity that is usually performed solely by the design engineer. Quality assurance engineers may review the simulation plans (if they are produced) or results, and for critical devices they may witness some of the simulation runs.
|
Verification activity |
Performed by |
|
Evaluate design (HDL) against requirements |
Quality assurance engineer |
|
Functional Simulation |
Design engineer |
|
Safety assessment |
System safety engineer |
|
Design review (e.g. CDR, peer review) |
All |
|
Static analysis of HDL code |
IV&V, Quality assurance engineer |
Functional simulation involves emulating the functionality of a device to determine that it is working per the specification and that it will produce correct results. This type of simulation is good at finding errors or bugs in the design. Functional simulation is also used after the design synthesis step where the gate-level design is simulated.
The HDL code should be reviewed by one or more engineers who can assess the design. A good reviewer has to understand the system within which the device will operate, know the HDL language being used, and be able to compare what the device is designed to do against its requirements. This means that not just anyone can adequately review the design. Lack of knowledge or experience will hamper the review, and often cause the designer to think it is a waste of time.
For very complex or safety-critical devices, Independent Verification and Validation (IV&V) may be called in to review the design. One tool they can use is static analysis software for the HDL code. This can look for problems or possible errors in the code. This tool is very similar to some static analysis tools for software that look for potential logic or coding errors.
During design synthesis, the higher level designs are optimally translated to a gate-level design, which can then be mapped to the logic blocks in a complex electronic device. It is during synthesis that timing and area constraints can be specified by the user.
|
Verification activity |
Performed by |
|
Functional Simulation of gate-level circuit |
Design engineer |
|
Design review (peer review) |
All |
|
Design evaluation |
Quality assurance engineer |
Simulation is one of the primary ways that the design synthesis process is verified. In almost all projects, the design engineer is the one who generates the test bench, defines the simulation runs, and performs the simulations. Quality assurance is rarely involved, other than to perhaps verify that the simulations were performed. However, it is important to look at the design of the test bench and the simulation tests to make sure they are complete enough. This is the time to find errors or mistaken assumptions - not when you are integrating your complex electronics with other areas of the system.
Implementation is where the design meets the silicon - the higher level design is converted into a chip layout. The implementation process uses the tools supplied by the device (e.g. FPGA) vendor to match the functions that were defined in the design to the available blocks, gates, and other logic elements on the chip.
|
Verification activity |
Performed by |
|
Timing simulation |
Design engineer |
|
Static timing analysis |
Design engineer |
|
Device programming |
Witnessed by quality assurance engineer |
|
Fault injection testing |
Design or quality assurance engineer |
Timing simulations are simply functional simulation with timing information. The timing information allows the designer to confirm that signals change in the correct timing relationship to each other. The timing information is entered in the hardware description language model file and then simulated. However, since there is a possibility of not being able to simulate all combination of inputs, a timing analysis tool can be used to evaluate a fully synchronous design.
Static timing analysis is a process that examines a synchronous design and determines its highest operating frequency. The analysis considers the path from every flip-flop in the design to every other flip-flop to which it is connected through the combinatorial logic. The analysis is usually performed by a software tool, which calculates the best-case and the worst-case delays through these paths (critical-paths). Any paths that violate the set-up or hold-timing requirements of the flip-flop are flagged for later adjustment to meet the design requirements.
Understanding how the complex electronics will operate when given invalid input is very important in verifying the devices. The real world is messy, and noisy signals or broken interfaced hardware are unfortunately common. Simulation is a great place to perform fault injection testing by inputting signals that are out of range, whose timing is not correct, that have ringing or other signal problems, or that are noisy. Encouraging this type of testing, and helping to identify the likely types of faults, is one way that quality assurance personnel can actively participate in the verification of complex electronics.
While simulation is used extensively in complex electronic design, testing the actual chip can sometimes be an eye-opening experience. Simulation involves assumptions and compromises that may not match with the real world. Testing the programmed chip - either independently or integrated onto a circuit board - is a necessary step in verifying your design.
|
Verification activity |
Performed by |
|
In-circuit functional and timing tests |
Design engineer, may be witnessed by QA |
|
Sub-system and system tests |
All |
|
Safety verification |
All, but reviewed or witnessed by System Safety Engineer |
In-circuit verification tests the functionality and timing of the design on the actual chip. Ideally, special test software running on a host computer will interface with the device under test through available test ports, such as the JTAG port. This process is similar to in-circuit emulators that run embedded software on the target processor, and provide breakpoints and tracing into the actual software instructions.
The more common form of in-circuit tests is to manually run the complex electronics as part of a higher-level assembly to show that it meets all the specified requirements. This sub-system or system level test will show functionality at a black-box level, but will not provide a window into the internal functioning of the device.
If the complex electronic device is safety-critical, there will be a separate safety verification, usually at the system level.
The verifications mentioned above relate mostly to the product or system. Additional verifications are performed throughout the life cycle as part of the process assurance activities. These process verifications look at the management and development plans to ensure that a quality process is in place for developing the system (and the complex electronics). As the project moves from concept to requirements, design, implementation, and testing, process assurance tracks the project activities against the plans. If the project is deviating from the plans, then a course correction is required. Either the plans need to be changed to reflect reality (while still embodying a quality process), or the project needs to refocus on the defined processes. It's very easy to throw processes out the window when the schedule gets tight, but experience has shown that doing so only makes it more likely that additional problems will be created. Planning and implementing a quality process should help projects avoid massive debugging at the end of development.
Process verification is usually carried out via a series of audits at various times in the life cycle. The number of audits, and the depth to which they probe the project processes, will vary with the complexity of the project. For complex electronics, a low criticality component may only be subject to a single audit, which ensures that the development process is being followed. High criticality devices will have multiple audits that look at the development and supporting processes (such as configuration management) at several times in the life cycle.
More details on audits can be found in the CE Assurance Plan page.