Verification and Validation
of Complex Electronics
This section
presents an overview of the verification and validation activities
performed throughout the life cycle. The activities and tasks are
described in more detail in the corresponding phase sub-section (e.g.,
requirements or testing). This page provides an overview of verification
and validation (V&V)
activities for each phase of the complex electronics life cycle.
V&V Overview
Validation
is the confirmation, through objective evidence, that the system will
perform its intended functions. The intended functions, and how well
the system performs those functions, are determined by the customer.
Did you create the system the customer really wanted? Will the system
fulfill the customer's needs? Is this the right system for the customer?
Verification
is the confirmation, though objective evidence, that the specified
requirements have been fulfilled. Verification tasks all point back to
the requirements. Does the design correctly and completely embody the
requirements? Is the implementation a correct representation of the
requirements? Is the system being built right?
While
verification and validation are sometimes used interchangeably, they do
represent different ways to ensure the system being built will meet the
needs of the customer. If the requirements are validated (shown to be
the right requirements), and the system is verified against the
requirements, the resulting system will fulfill its purpose. However,
getting the right requirements is hard, and requirements can change.
Without validation activities throughout the life cycle, you can end up
with a verified system that doesn't perform its intended functions.
Validation
is mostly a concern at the system level. However, the requirements that
are flowed-down to the complex electronics still have to be validated
to be the right requirements. The actual system customer is not likely
to be involved with validating the lower-level requirements, but the
complex electronics requirements can be validated against the higher
level requirements and functions.
Verification
is best performed in a step-wise fashion, at the end of each life cycle
phase. When the complex electronics requirements are completed, they
can be verified to be correct, complete, and consistent with the higher
level requirements. The design can be verified to embody the CE
requirements, at various levels of detail. The physical programmed
device can be tested to show that the requirements were correctly
implemented. Testing is only one of the verification activities.
It should not be the only one.
Complex
electronics adds some particular quirks to watch out for when verifying
the devices:
- Tool-induced design
errors do occur and can be difficult to detect. The output of a
tool may be different, depending on the hardware description
language constructs you use. Tools are a vital part of complex
electronics design, and the designer often does not know what
errors a tool could potentially produce.
- Complex functionality
cannot be completely simulated, nor the resulting chip completely
tested.
- It can be difficult to
detect faulty operation of complex electronics due to design or
tool-induced errors, unexpected interactions, or even defects in the
silicon.
Life
Cycle Verification Activities
Verification
is a parallel set of activities to engineering design and development.
The assurance engineer performs various tasks each phase of the
development. The sections below provide an overview of the types of
verification activities that will be performed for complex electronics.
Specific assurance tasks and analyses are covered in more detail in
each life cycle phase.
Requirements
At the
Requirements Phase, the system or sub-system level requirements are
flowed down to the complex electronics. This flow down is primarily the
responsibility of the systems engineer, though hopefully the design
engineer for the complex electronics will be involved , to prevent
requirements being imposed on the hardware that it cannot meet!
|
Verification
activity
|
Performed
by
|
|
Evaluate
requirements for the complex electronics
|
Quality
assurance, systems engineers
|
|
Safety
assessment
|
System
safety engineer
|
|
Requirements
review (e.g. PDR)
|
All
|
|
Identification
of applicable standards
|
Quality
assurance, safety, design engineers
|
|
Formal
methods
|
IV&V
or knowledgeable practitioner
|
Quality
assurance engineers should review the requirements for correctness, completeness,
and consistency. High quality requirements are worth their weight in
gold. Good requirements make everyone's life easier, because bad
requirements are difficult to verify, are often interpreted differently
by various people, and may not implement the functions that are
desired. Finding out during testing that the device is missing
important functionality, or is too slow, is something you really want
to avoid.
For
safety or mission-critical devices, formal methods might be used as a verification
tool. The requirements can be defined using a special language that
allows mathematical proofs to be generated showing that the device will
not violate certain properties. Formal methods can be applied at only
the requirements level (to make sure you get those correct), or also be
used to verify the design when it is generated. Most projects will not
use formal methods.
Design Entry
During
the Design Entry Phase, the complex electronics functionality is
defined in a hardware description language (HDL). The HDL code can be
simulated in a test bench and its behavior can be observed. This is an
important verification activity that is usually performed solely by the
design engineer. Quality assurance engineers may review the simulation
plans (if they are produced) or results, and for critical devices they
may witness some of the simulation runs.
|
Verification
activity
|
Performed by
|
|
Evaluate
design (HDL) against requirements
|
Quality
assurance engineer
|
|
Functional
Simulation
|
Design
engineer
|
|
Safety
assessment
|
System
safety engineer
|
|
Design
review (e.g. CDR, peer review)
|
All
|
|
Static
analysis of HDL code
|
IV&V,
Quality assurance engineer
|
Functional
simulation involves emulating the functionality of a device to determine
that it is working per the specification and that it will produce
correct results. This type of simulation is good at finding errors or
bugs in the design. Functional simulation is also used after the design synthesis step where the
gate-level design is simulated.
The HDL
code should be reviewed by one or more engineers who can assess the
design. A good reviewer has to understand the system within which the
device will operate, know the HDL language being used, and be able to
compare what the device is designed to do against its requirements.
This means that not just anyone can adequately review the design. Lack
of knowledge or experience will hamper the review, and often cause the
designer to think it is a waste of time.
For very
complex or safety-critical devices, Independent Verification and
Validation (IV&V) may be called in to review the design. One tool
they can use is static analysis software for the HDL code, which can
look for problems or possible errors in the code. This tool is very similar
to some static analysis tools for software that look for potential
logic or coding errors.
Design Synthesis
During
design synthesis, the higher level designs are optimally translated to
a gate-level design, which can then be mapped to the logic blocks in a
complex electronic device. It is during synthesis that timing and area
constraints can be specified by the user.
|
Verification
activity
|
Performed by
|
|
Functional
Simulation of gate-level circuit
|
Design
engineer
|
|
Design
review (peer review)
|
All
|
|
Design
evaluation
|
Quality
assurance engineer
|
Simulation
is one of the primary ways that the design synthesis process is
verified. In almost all projects, the design engineer is the one who generates
the test bench, defines the simulation runs, and performs the
simulations. Quality assurance is rarely involved, other than to
perhaps verify that the simulations were performed. However, it is
important to look at the design of the test bench and the simulation
tests to make sure they are complete enough. This is the time to find
errors or mistaken assumptions - not when you are integrating your
complex electronics with other areas of the system.
Implementation
Implementation
is where the design meets the silicon - the higher level design is
converted into a chip layout. The implementation process uses the tools
supplied by the device (e.g. FPGA) vendor to match the functions that
were defined in the design to the available blocks, gates, and other
logic elements on the chip.
|
Verification
activity
|
Performed by
|
|
Timing
simulation
|
Design
engineer
|
|
Static
timing analysis
|
Design
engineer
|
|
Device
programming
|
Witnessed
by quality assurance engineer
|
|
Fault injection
testing
|
Design
or quality assurance engineer
|
Timing
simulations are simply functional simulation with timing information.
The timing information allows the designer to confirm that signals change
in the correct timing relationship to each other. The timing
information is entered in the hardware description language model file
and then simulated. However, since there is a possibility of not being
able to simulate all combination of inputs, a timing analysis tool can
be used to evaluate a fully synchronous design.
Static
timing analysis is a process that examines a synchronous design and
determines its highest operating frequency. The analysis considers the
path from every flip-flop in the design to every other flip-flop to
which it is connected through the combinatorial logic. The analysis is
usually performed by a software tool, which calculates the best-case
and the worst-case delays through these paths (critical-paths). Any
paths that violate the set-up or hold-timing requirements of the
flip-flop are flagged for later adjustment to meet the design
requirements.
Understanding
how the complex electronics will operate when given invalid input is
very important in verifying the devices. The real world is messy, and
noisy signals or broken interfaced hardware are unfortunately common.
Simulation is a great place to perform fault injection testing by
inputting signals that are out of range, whose timing is not correct,
that have ringing or other signal problems, or that are noisy.
Encouraging this type of testing, and helping to identify the likely
types of faults, is one way that quality assurance personnel can
actively participate in the verification of complex electronics.
Testing
While
simulation is used extensively in complex electronic design, testing
the actual chip can sometimes be an eye-opening experience. Simulation
involves assumptions and compromises that may not match with the real
world. Testing the programmed chip - either independently or integrated
onto a circuit board - is a necessary step in verifying your design.
|
Verification
activity
|
Performed by
|
|
In-circuit
functional and timing tests
|
Design
engineer, may be witnessed by QA
|
|
Sub-system
and system tests
|
All
|
|
Safety
verification
|
All,
but reviewed or witnessed by System Safety Engineer
|
In-circuit
verification tests the functionality and timing of the design on the
actual chip. Ideally, special test software running on a host computer will
interface with the device under test through available test ports, such
as the JTAG port. This process is similar to in-circuit emulators that
run embedded software on the target processor, and provide breakpoints
and tracing into the actual software instructions.
The more
common form of in-circuit tests is to manually run the complex
electronics as part of a higher-level assembly to show that it meets
all the specified requirements. This sub-system or system level test
will show functionality at a black-box level, but will not provide a
window into the internal functioning of the device.
If the
complex electronic device is safety-critical, there will be a separate
safety verification, usually at the system level.
Process Verifications
The
verifications mentioned above relate mostly to the product or system. Additional
verifications are performed throughout the life cycle as part of the
process assurance activities. These process verifications look at the
management and development plans to ensure that a quality process is in
place for developing the system (and the complex electronics). As the
project moves from concept to requirements, design, implementation, and
testing, process assurance tracks the project activities against the
plans. If the project is deviating from the plans, then a course
correction is required. Either the plans need to be changed to reflect
reality (while still embodying a quality process), or the project needs
to refocus on the defined processes. It's very easy to throw processes
out the window when the schedule gets tight, but experience has shown
that doing so only makes it more likely that additional problems will
be created. Planning and implementing a quality process should help
projects avoid massive debugging at the end of development.
Process
verification is usually carried out via a series of audits at various
times in the life cycle. The number of audits, and the depth to which
they probe the project processes, will vary with the complexity of the
project. For complex electronics, a low
criticality component may only be subject to a single
audit, which ensures that the development process is being followed. High criticality devices will
have multiple audits that look at the development and supporting
processes (such as configuration management) at several times in the
life cycle.
More
details on audits can be found in the CE
Assurance Plan page.
|
