Skip all navigation and jump to content

Jump to site navigation

NASA Logo

+ Visit NASA.gov

Assurance Process for Complex Electronics

Home

Complex Electronics Background

Complex Electronics Assurance Process

TECHNIQUES

CHECKLISTS

Site Map

Life Cycle

PLANNING

V&V

REQUIREMENTS

PRELIMINARY DESIGN

DETAILED DESIGN

IMPLEMENTATION

testing

operationsoperations

SUPPORTING PROCESSES

PRINT THIS SECTION

Verification and Validation of Complex Electronics

This section presents an overview of the verification and validation activities performed throughout the life cycle. The activities and tasks are described in more detail in the corresponding phase sub-section (e.g., requirements or testing). This page provides an overview of verification and validation (V&V) activities for each phase of the complex electronics life cycle.

V&V Overview

Validation is the confirmation, through objective evidence, that the system will perform its intended functions. The intended functions, and how well the system performs those functions, are determined by the customer. Did you create the system the customer really wanted? Will the system fulfill the customer's needs? Is this the right system for the customer?

Verification is the confirmation, though objective evidence, that the specified requirements have been fulfilled. Verification tasks all point back to the requirements. Does the design correctly and completely embody the requirements? Is the implementation a correct representation of the requirements? Is the system being built right?

While verification and validation are sometimes used interchangeably, they do represent different ways to ensure the system being built will meet the needs of the customer. If the requirements are validated (shown to be the right requirements), and the system is verified against the requirements, the resulting system will fulfill its purpose. However, getting the right requirements is hard, and requirements can change. Without validation activities throughout the life cycle, you can end up with a verified system that doesn't perform its intended functions.

Validation is mostly a concern at the system level. However, the requirements that are flowed-down to the complex electronics still have to be validated to be the right requirements. The actual system customer is not likely to be involved with validating the lower-level requirements, but the complex electronics requirements can be validated against the higher level requirements and functions.

Verification is best performed in a step-wise fashion, at the end of each life cycle phase. When the complex electronics requirements are completed, they can be verified to be correct, complete, and consistent with the higher level requirements. The design can be verified to embody the CE requirements, at various levels of detail. The physical programmed device can be tested to show that the requirements were correctly implemented. Testing is only one of the verification activities. It should not be the only one.

Complex electronics adds some particular quirks to watch out for when verifying the devices:

  • Tool-induced design errors do occur and can be difficult to detect. The output of a tool may be different, depending on the hardware description language constructs you use. Tools are a vital part of complex electronics design, and the designer often does not know what errors a tool could potentially produce.
  • Complex functionality cannot be completely simulated, nor the resulting chip completely tested.
  • It can be difficult to detect faulty operation of complex electronics due to design or tool-induced errors, unexpected interactions, or even defects in the silicon.

Life Cycle Verification Activities

Verification is a parallel set of activities to engineering design and development. The assurance engineer performs various tasks each phase of the development. The sections below provide an overview of the types of verification activities that will be performed for complex electronics. Specific assurance tasks and analyses are covered in more detail in each life cycle phase.

Requirements

At the Requirements Phase, the system or sub-system level requirements are flowed down to the complex electronics. This flow down is primarily the responsibility of the systems engineer, though hopefully the design engineer for the complex electronics will be involved , to prevent requirements being imposed on the hardware that it cannot meet!

Verification activity

Performed by

Evaluate requirements for the complex electronics

Quality assurance, systems engineers

Safety assessment

System safety engineer

Requirements review (e.g. PDR)

All

Identification of applicable standards

Quality assurance, safety, design engineers

Formal methods

IV&V or knowledgeable practitioner

Quality assurance engineers should review the requirements for correctness, completeness, and consistency. High quality requirements are worth their weight in gold. Good requirements make everyone's life easier, because bad requirements are difficult to verify, are often interpreted differently by various people, and may not implement the functions that are desired. Finding out during testing that the device is missing important functionality, or is too slow, is something you really want to avoid.

For safety or mission-critical devices, formal methods might be used as a verification tool. The requirements can be defined using a special language that allows mathematical proofs to be generated showing that the device will not violate certain properties. Formal methods can be applied at only the requirements level (to make sure you get those correct), or also be used to verify the design when it is generated. Most projects will not use formal methods.

Design Entry

During the Design Entry Phase, the complex electronics functionality is defined in a hardware description language (HDL). The HDL code can be simulated in a test bench and its behavior can be observed. This is an important verification activity that is usually performed solely by the design engineer. Quality assurance engineers may review the simulation plans (if they are produced) or results, and for critical devices they may witness some of the simulation runs.

Verification activity

Performed by

Evaluate design (HDL) against requirements

Quality assurance engineer

Functional Simulation

Design engineer

Safety assessment

System safety engineer

Design review (e.g. CDR, peer review)

All

Static analysis of HDL code

IV&V, Quality assurance engineer

Functional simulation involves emulating the functionality of a device to determine that it is working per the specification and that it will produce correct results. This type of simulation is good at finding errors or bugs in the design. Functional simulation is also used after the design synthesis step where the gate-level design is simulated.

The HDL code should be reviewed by one or more engineers who can assess the design. A good reviewer has to understand the system within which the device will operate, know the HDL language being used, and be able to compare what the device is designed to do against its requirements. This means that not just anyone can adequately review the design. Lack of knowledge or experience will hamper the review, and often cause the designer to think it is a waste of time.

For very complex or safety-critical devices, Independent Verification and Validation (IV&V) may be called in to review the design. One tool they can use is static analysis software for the HDL code, which can look for problems or possible errors in the code. This tool is very similar to some static analysis tools for software that look for potential logic or coding errors.

Design Synthesis

During design synthesis, the higher level designs are optimally translated to a gate-level design, which can then be mapped to the logic blocks in a complex electronic device. It is during synthesis that timing and area constraints can be specified by the user.

Verification activity

Performed by

Functional Simulation of gate-level circuit

Design engineer

Design review (peer review)

All

Design evaluation

Quality assurance engineer

Simulation is one of the primary ways that the design synthesis process is verified. In almost all projects, the design engineer is the one who generates the test bench, defines the simulation runs, and performs the simulations. Quality assurance is rarely involved, other than to perhaps verify that the simulations were performed. However, it is important to look at the design of the test bench and the simulation tests to make sure they are complete enough. This is the time to find errors or mistaken assumptions - not when you are integrating your complex electronics with other areas of the system.

Implementation

Implementation is where the design meets the silicon - the higher level design is converted into a chip layout. The implementation process uses the tools supplied by the device (e.g. FPGA) vendor to match the functions that were defined in the design to the available blocks, gates, and other logic elements on the chip.

Verification activity

Performed by

Timing simulation

Design engineer

Static timing analysis

Design engineer

Device programming

Witnessed by quality assurance engineer

Fault injection testing

Design or quality assurance engineer

Timing simulations are simply functional simulation with timing information. The timing information allows the designer to confirm that signals change in the correct timing relationship to each other. The timing information is entered in the hardware description language model file and then simulated. However, since there is a possibility of not being able to simulate all combination of inputs, a timing analysis tool can be used to evaluate a fully synchronous design.

Static timing analysis is a process that examines a synchronous design and determines its highest operating frequency. The analysis considers the path from every flip-flop in the design to every other flip-flop to which it is connected through the combinatorial logic. The analysis is usually performed by a software tool, which calculates the best-case and the worst-case delays through these paths (critical-paths). Any paths that violate the set-up or hold-timing requirements of the flip-flop are flagged for later adjustment to meet the design requirements.

Understanding how the complex electronics will operate when given invalid input is very important in verifying the devices. The real world is messy, and noisy signals or broken interfaced hardware are unfortunately common. Simulation is a great place to perform fault injection testing by inputting signals that are out of range, whose timing is not correct, that have ringing or other signal problems, or that are noisy. Encouraging this type of testing, and helping to identify the likely types of faults, is one way that quality assurance personnel can actively participate in the verification of complex electronics.

Testing

While simulation is used extensively in complex electronic design, testing the actual chip can sometimes be an eye-opening experience. Simulation involves assumptions and compromises that may not match with the real world. Testing the programmed chip - either independently or integrated onto a circuit board - is a necessary step in verifying your design.

Verification activity

Performed by

In-circuit functional and timing tests

Design engineer, may be witnessed by QA

Sub-system and system tests

All

Safety verification

All, but reviewed or witnessed by System Safety Engineer

In-circuit verification tests the functionality and timing of the design on the actual chip. Ideally, special test software running on a host computer will interface with the device under test through available test ports, such as the JTAG port. This process is similar to in-circuit emulators that run embedded software on the target processor, and provide breakpoints and tracing into the actual software instructions.

The more common form of in-circuit tests is to manually run the complex electronics as part of a higher-level assembly to show that it meets all the specified requirements. This sub-system or system level test will show functionality at a black-box level, but will not provide a window into the internal functioning of the device.

If the complex electronic device is safety-critical, there will be a separate safety verification, usually at the system level.

Process Verifications

The verifications mentioned above relate mostly to the product or system. Additional verifications are performed throughout the life cycle as part of the process assurance activities. These process verifications look at the management and development plans to ensure that a quality process is in place for developing the system (and the complex electronics). As the project moves from concept to requirements, design, implementation, and testing, process assurance tracks the project activities against the plans. If the project is deviating from the plans, then a course correction is required. Either the plans need to be changed to reflect reality (while still embodying a quality process), or the project needs to refocus on the defined processes. It's very easy to throw processes out the window when the schedule gets tight, but experience has shown that doing so only makes it more likely that additional problems will be created. Planning and implementing a quality process should help projects avoid massive debugging at the end of development.

Process verification is usually carried out via a series of audits at various times in the life cycle. The number of audits, and the depth to which they probe the project processes, will vary with the complexity of the project. For complex electronics, a low assurance level component may only be subject to a single audit, which ensures that the development process is being followed. High assurance devices will have multiple audits that look at the development and supporting processes (such as configuration management) at several times in the life cycle.

More details on audits can be found in the CE Assurance Plan page.

 

FirstGov logo

+ NASA Privacy, Security, Notices

NASA

Curator: Richard Plastow
NASA Official: Cynthia Calhoun
Last Updated: 12/14/2009