Requirements Evaluation
One of the key assurance
activities for any project is expert evaluation and assessment of the
requirements. The requirements need to be evaluated by multiple people
with varying expertise, in order to identify incorrect, ambiguous, and
incomplete requirements. A Requirements Review is one way to accomplish
this evaluation.
The assurance engineer needs to perform an in-depth evaluation of the
complex electronics requirements, regardless of any formal or informal
reviews that the project holds. This evaluation can be used as an input
to any requirements reviews that are held. However, the assurance
engineer will be looking at the requirements in a much greater depth
than most formal reviews.
A fundamental question
for requirements evaluation is: How well do the requirements accomplish
the system and sub-system (Complex Electronics) objectives? This
question provides a framework within which to consider the specific
requirements for the complex electronics device. While it is very easy
to take a focused approach to low-level requirements, such as for
complex electronics, it is in the best interest of the project to
always keep a systems-level viewpoint. When the whole system is
considered, in relation to the specific low-level requirements,
inconsistencies and missing requirements can be identified.
For Complex Electronics,
the main goals of requirements evaluation are:
- Ensure that all appropriate higher-level
requirements are included in the CE requirements specification.
- Identify any inconsistencies or
conflicts within the requirements, or with other requirements
documents.
- Ensure that the quality of the document
(e.g. readable, unambiguous) is adequate for its usage.
- Ensure decomposition of the higher-level
requirements into those suitable for the CE device.
Higher Level Requirements
The first step to
evaluating the complex electronics requirements is to identify all the
higher-level requirements and documents that influence the device. The
following list of documents is not exhaustive, but provides examples of
documents used by many projects. The assurance engineer performing this
evaluation should look at the list of project documents and select any
that may provide requirements that the complex electronic device has to
implement.
- System specification or requirements
document. While
the complex electronic device requirements are normally derived
from sub-system or other lower-level requirements documents, there
is the possibility of missed requirements from the system
specification. A review of the system specification can be useful in
identifying missing requirements for the complex electronics, and
possibly for the sub-system.
- Sub-system specifications or
requirements documents for all sub-systems that the complex
electronics interacts with. Besides the particular sub-system that the complex
electronics resides in, the assurance engineer needs to consider
the requirements on sub-systems that interface with the complex
electronics (or its sub-system). Sub-systems that can influence
the complex electronics (e.g. through noise or excessive power
draw) should also be considered in the evaluation.
- Interface specifications. The interface between the complex
electronics device and other elements of the system may be
documented in an interface specification or control document
(ICD). The interfaces between elements of the system that interact
with the complex electronics are important areas to check for
inconsistencies or missing elements.
- Operations concepts. For some systems, the concept of
operations for the system can be used to evaluate whether the
behavior of the complex electronics is consistent with the system
operations. For example, consider a time critical function for a
system, such as capturing rapid fluctuations in temperature. If
the complex electronics is one component used to fulfill that
function, then the complex electronics needs to have requirements
for rapid reading of ports, sufficient speed for data transfer,
and possibly a requirement for oversampling the data.
- Preliminary Hazard Analysis. The Preliminary Hazard Analysis (PHA)
should always be reviewed, even if the complex electronics is not
planned to be part of a hazard control, to ensure that no function
of the CE device can cause a hazard or interfere with other hazard
controls. If the complex electronics is being used as a hazard
control (or part of one), then the PHA will provide information
that should be reflected as requirements on the complex
electronics.
- Failure Modes and Effects Analysis. If a Failure Modes and Effects
Analysis (FMEA) is performed on the system, review it for any
information on possible failures that can impact the functioning
of the complex electronics. Those failures might need to be
mitigated by some element of the system or within the complex
electronic device itself. If the failure modes of the CE device
are included in FMEA, make sure that the CE requirements include
ways to prevent or mitigate those failure modes.
- Fault Tree Analysis. If a Fault Tree Analysis (FTA) is
performed, review it to see if the complex electronics is a
possible cause of or contributor to a system failure. If so,
ensure that the requirements include ways to prevent or mitigate
those problems. If not, consider whether any other identified
failures in the system could have a detrimental impact on the
complex electronics.
- Standards, Policies, and Procedures. NASA has few policies or standards
specific to complex electronics. However, the assurance engineer
should review the standards, policies, and procedures that relate
to systems, electronics, and software for NASA and the Center
where the system is being developed. These standards may impose
requirements that need to be reflected in the complex electronics
specification. At a minimum, consider these documents:
- NPR 7120.5C NASA
Program and Project Management Processes and Requirements
- NPD 8720.1B NASA
Reliability and Maintainability (R&M) Program Policy
- NPD 8730.2B NASA
Parts Policy
- Software-related
requirements:
- NPR 7150.2, NASA
Software Engineering Requirements
- NASA-STD-8719.13,
NASA Software Safety Technical Standard.
- NASA-STD-8739.8, NASA
Software Assurance Technical Standard
- Lessons learned, historical data,
guidelines. This
assurance process provides a first-cut at guidelines for
requirements for complex electronics. If time permits, the
assurance engineer should look for other sources of guidance,
including failures in similar devices, to identify possible
requirements for the device under consideration.
Once the appropriate
documents are identified and gathered, the assurance engineer will
construct a cross-reference of all related higher-level requirements
and their counterparts for the complex electronics. If a traceability
table of the complex electronics requirements is provided, it is a useful
tool for evaluating whether the higher-level requirements are included.
All higher-level requirements that the complex electronics implements
(all or part of the requirement) should trace to specific requirements
in the complex electronics specification. In addition, all complex
electronics requirements should trace back to a higher-level
requirement.
Inconsistencies and Conflicts
With any complex system,
it is difficult to ensure that there are no inconsistencies or
conflicts within the large set of requirements for the system. When
higher-level requirements are decomposed to lower-level, sub-system
requirements, it is possible for information to be lost or changed.
This is especially true if the sub-system requirements are written by
the engineers responsible for those sub-systems. Even in the best
managed requirements development, however, mistakes do happen.
Conflicting requirements
are those that state essentially the same requirement (or refer to the
same item), but have different values or constraints. For example, a
requirement for the electronic circuit board that will include a
complex electronic device may state that the device will have 24 pins,
while the requirements for that device may specify 28 pins. Another
example is the data rate at which a sensor must be read. The complex
electronics may have a requirement for 1 hertz, while the software that
processes the data states that the sensor will be read at 10 hertz.
Inconsistencies are
basically subtle conflicts. These are the cases where the requirements
are specifying a behavior, interface, or timing for the complex
electronic device that cannot meet all or part of another requirement.
Finding inconsistencies often requires looking at multiple parts of the
system, especially when they have to work together to implement a
higher-level requirement.
For example, a system
requirement might be to acquire images from a camera at a range of
rates (number of images per second). At some point in the data path,
the image data will be moved by the complex electronics from a FIFO
(first-in first-out temporary storage) into memory. The speed at which
the complex electronics moves the data is specified in its requirements
document. When assessing the requirements, the assurance engineer
refers to the system requirement and calculates the maximum and minimum
rates at which data is acquired, which correspond to the rates at which
the data must move through the system to prevent a bottleneck. In this
case, the speed specified for the complex electronics can meet most of
the system image acquisition rates, but is not adequate for the highest
acquisition rate.
Requirements Quality
Requirements are
evaluated against “goodness” criteria that define
properties the requirements should have. The requirements for complex
electronics should meet these criteria to a reasonable extent. That
means that the requirements do not have to be perfect, but have to be
“good enough”. Complex systems or those with critical or
safety requirements will require more formalism and detail in the
specific requirements. Smaller, less critical, projects may be tolerate
some “slack”, or lack of detail, in the requirements,
because all the stakeholders are involved and understand what the
requirements mean. The assurance engineer performing the requirements
evaluation should exercise judgment in determining whether the
requirements meet, or do not meet, a specific criterion.
These are the qualities
each requirement, and the set of requirements as a whole, should meet:
- Complete. This quality defines what the device
has to do, in terms of structure (inputs, outputs) and behavior.
Areas that are uncertain should be so noted in the requirements
document. Requirements should be included for the device responses
to all possible inputs in all achievable circumstances.
- Correct. The set of requirements contain all
the information needed to design the device and no more. The
requirements define a device that will meet higher-level
requirements for the system.
- Concise. The requirements are as short as
possible (least number of words) without adversely affecting any
other criteria
- Unambiguous. The requirements are understood to
mean the same thing by all parties. Can the designer, quality
assurance, systems engineering, the project manager, and other
parties all agree on what each requirements means?
- Understandable or Readable. The requirements are not confusing to
one or more of the parties. They are written in clear English
without the use of special terminology, except when necessary. If
a requirement is not understandable it is, by default, also
ambiguous.
- Consistent. The requirements do not conflict with
each other in this specification (internally), or with higher (or
lower) level requirements (externally)
- Precise. The requirements exactly define the
behavior. When appropriate, they define the range of acceptable
values, tolerances, timing, etc.
- Verifiable. Each requirement can be verified by
test, analysis, simulation (demonstration), or inspection in a
cost-effective way.
- Traceable. Each requirement can be traced to one
or more higher-level requirements. Tracing means that the
requirement implements all or part of the higher-level
requirement. Derived requirements are those that are needed in
order to implement the higher-level requirement. As such, they
still trace to that higher-level requirement.
- Implementation independent. The requirements do not specify details
such as the layout on the chip. However, there will be constraints
based on the chosen hardware that need to be documented as
requirements.
- At the right level of abstraction/detail. Are the requirements detailed enough
and not simply restating higher-level requirements? Are the
requirements so detailed that they essentially provide the design?
The requirements should be somewhere between these two extremes.
- Modifiable. If the system requirements that feed
the complex electronics requirements are fluid, the device
requirements may need to be changed. This is not really a
criterion for individual requirements, but something to consider
when defining the other requirements.
Note that some of the
criteria used to judge a requirements document cannot be
“maximized” except at the detriment of other criteria. For
example, a very concise set of requirements may not be understandable
by some project members or other stakeholders (e.g. manager, systems
engineer, customer). The key to evaluating a set of requirements is to
focus on the most important elements for the specific project. For
example, if the requirements need to be reviewed and approved by some
higher-level or external authority, then understandability is an important
quality.
Requirements Decomposition
The decomposition, or
derivation, of requirements from system-level down to component or
device level is the major step in defining requirements for complex
electronics. When evaluating the requirements, it is important to
consider whether the requirements are too general, too specific, or if
they have the correct balance between these extremes.
Requirements are too
general if they simply restate the higher-level requirement, sometimes
verbatim. Consider the high-level requirement to provide communications
from the system in a particular protocol. The protocol is a combination
of hardware and software elements, and the complex electronics will
play a part in that protocol. However, simply restating that the
complex electronic device follows the protocol tells the design
engineer nothing about what piece of the protocol the complex
electronics needs to implement.
Requirements are too
specific if they constrain the design solution to only one possibility.
For complex electronics, requirements that specify placement on the
chip, specific function blocks, or internal wire (signal) length are
probably too specific. However, sometimes constraints have to be
imposed on the design, and therefore are included in the set of requirements.
A maximum internal signal path length may be required because longer
paths are too susceptible to noise. Such a requirement is a constraint
driven by the system reliability requirements, the environment the
system will operate in, and the physics of the device.
When a requirement is
too general (which is the more common problem), it cannot be verified
at the device level. When evaluating the requirements, ask yourself if
you can devise a test for the requirement that only involves the device
and supporting equipment. If not, the requirement may be too general.
Also, if you find yourself asking “but what does this device do
to fulfill the requirement”, it is probably too general.
What to Look for During the Evaluation
Use the Requirements
Review Checklist, which contains specific items to look for in
requirements for complex electronics, as an evaluation guide. Modify
the checklist to make it specific to your project.
A good specification for
complex electronics will contain:
- A description of how the device fits
into the larger system. A block diagram is very helpful.
- A description and list of all the major
functions the device will perform. A block and/or flow diagram can
be used to show this information.
- A description of the device and
interfaces, such as:
- Chip physical
information (size, type, number of pins, etc.)
- I/O pin mapping and
description (output drive capability, input threshold level)
- Timing estimates for:
- Setup and hold times
for input pins
- Propagation times for
output pins
- Clock cycle time
- High-level estimates and goals
- Gate count estimate
- Power consumption
target
- Constraints on the device
- Other requirements or criteria the
device must implement
- Design-related choices (may be in a
management plan)
- Tools that will be
used at all stages of development
- Hardware Description
Language chosen
Error handling is an
issue that may be overlooked until the last minute, and then pasted
into the design. A good approach to error handling needs to take a
systems point-of-view to decide what errors, faults, and failures that
the system can ignore and which require a response. The response to a
particular fault or failure may involve many sub-systems. For complex
electronics, it is important to know what part it will play in the
system-level response to faults and failures. In addition, the
assurance engineer evaluating the complex electronics needs to ensure
that the systems engineer has considered the possible faults and
failures of the device, and the effects of those errors on the rest of
the system, and determined the best strategy for responding to the
problems.
When evaluating the
requirements for complex electronics, consider:
- Is there an exception handling mechanism
(possibly a hardware-software cooperative arrangement) for error conditions
that may be detected?
- Can the system actually detect errors
that the complex electronics may produce?
- Are errors that the complex electronics
is expected to respond to actually detectable by some element of
the system?
- For response actions to errors, who is
responsible for the action? What action is be to taken? What are
the affects of the action on the CE device and on the system? Are
there any constraints and conditions on the action?
Commercial Off-the-Shelf (COTS) and Re-use
“IP”
(Intellectual Property) cores may be purchased from the chip provider
or third party vendors, to reduce the design time for standard
components. Additionally, previously created components may be used in
new designs, for the same reasons. While re-use is a good thing, in
general, the COTS or re-used components need to be evaluated for this
system. Differences between systems can lead to subtle (and sometimes
not so subtle) problems with the component and its interactions with
the rest of the design.
It may not be known at
the time the requirements are defined whether any COTS or re-used
components will be included in the complex electronics. If not, defer
this analysis step until the design phase. However, the earlier these
questions are considered, and answered, the easier it is to add or
modify requirements to maintain a safe and functional system.
To help ensure
compatibility of the COTS or reused component, consider these
questions:
- Can the COTS or re-used component meet the
complex electronics requirements? Does it provide all the
necessary functionality? Can it operate with the required timing?
Is it compatible with other components, including other COTS or
re-used components? Can it operate within the constraints defined
for the complex electronics? Does it impose any constraints on
other elements of the complex electronics?
- Does the COTS or re-used component have
any functionality that is not required for this device? Will that
functionality need to be “blocked out”, and if so, how
will it be done? What are the consequences if this functionality
is accidentally invoked (through design error or effects during
operation)?
- Are the consequences of any functions
unknown? Do you know how the function can fail? What will be the component
output if invalid input is provided? Does the component have any
internal error checking or mitigation?
- Is there any information on operational
history or usage in other systems? Consider the types of systems
that the component was used in. Similarity in part (e.g. the same
FPGA) and in system type increases the likelihood that the
operational history will be comparable in this system.
- Are the interfaces for the component
consistent with the rest of complex electronics and the system?
Check signal levels and timing, in particular. If any element is
edge-triggered, make sure that the component is looking at the
same edge (rising or falling) that the interfacing component
assumes.
- How much documentation is provided? Is
there any information on the development process, tools used, or
amount and type of testing performed?
|
