Assurance Activities for Complex Electronics Requirements

As the engineers develop and derive the requirements for complex electronics, the assurance engineer(s) works alongside to assess the processes and the product (requirements). These two activity streams are opposite sides of the same coin, and result in a validated set of requirements.

Use the Tailoring chart to determine which activities or analyses are required for a particular assurance classification. Activities that are not required may still be performed, if desired. The assurance activities for complex electronic requirements include:

• Risk analysis
• Requirements Evaluation
• Requirements Review (formal or informal)
• Interface Analysis
• Traceability Analysis
• Modeling
• Simulation
• Formal Methods
• UML
• Criticality Mapping
• Decision Tables/Trees
• Reverse Engineering of Requirements
• Fault Tree Analysis
• Failure Modes and Effects Analysis

Every method listed above does not have to be applied to every project. The table below uses the Complex Electronics Classification to map the activities, and depth of each activity, against the classification. This table allows for easy tailoring of the assurance activities to the device complexity and assurance level.

Risk Analysis

A risk analysis is performed at the Requirements phase to identify the most critical risks to a successful completion of the complex electronics. Identifying potential problems early on allows the project to apply resources in an intelligent manner. Risks can be prioritized, so that the critical risks receive the most effort in mitigating. Some risks may only be monitored unless there is a significant change in the factors that determine the risk. For unlikely, but critical, risks, a contingency plan may be developed.

From an assurance point-of-view, the risk analysis points to the critical areas of project development for monitoring. For example, if requirement volatility is a major risk, then the assurance engineer should be closely watching how changes to requirements are input, agreed to, and analyzed for impacts. The assurance engineer can proactively suggest requirements management tools that may be of help and ensure that project plans and processes address how to add, delete, or modify requirements.

Details of the risk analysis technique can be found in the Techniques section.

Requirements Evaluation

One of the key assurance activities for any project is expert evaluation and assessment of the requirements. Incorrect, ambiguous, and incomplete requirements have been shown to be the major cause of system failures and accidents. Getting the requirements right is not easy, but it is vital to the success of the project.

For Complex Electronics, the main goals of requirements evaluation are:

• Ensure that all appropriate higher-level requirements are included in the CE requirements specification.
• Identify any inconsistencies or conflicts within the requirements, or with other requirements documents.
• Ensure that the quality of the document (e.g. readable) is adequate for its usage.
• Ensure decomposition of the higher-level requirements into those suitable for the CE device.
  • For example, a higher-level requirement to monitor a temperature would be decomposed into requirements for the complex electronics (read a thermistor 10 times a second, average every 10 readings, output the average reading once a second) and for software (receive the thermistor reading once a second, convert it to a temperature, check it against a range of acceptable values, and perform specified operations if it is out of range).

Details of requirements evaluation can be found in the Techniques section. Use the requirements checklist when evaluating the requirements.

Requirements Review

A Requirements Review is a more formalized process that brings together individuals with various understandings of the system and the complex electronics. The goal is to have each person review the requirements against their area of knowledge. By performing a wide review, problems related to interfaces, system operations, etc. are more likely to be found.

Details of a requirements review can be found in the Techniques section. Use the requirements checklist when evaluating the requirements.

Interface Analysis

Interface analysis is a static analysis technique used to demonstrate that the interfaces (inputs, outputs) between the complex electronic device and other parts of the system do not contain any errors. This analysis will be performed (updated) at multiple times throughout the life cycle, as more details are available, or when changes are made to the system.

At the requirements phase, the main goal of this analysis is to identify input or output signals that are incorrect (wrong range, wrong type), missing, or present but not connected (e.g. an output that doesn't go anywhere). Timing relationships will be looked at as the device is designed.

Details of interface analysis can be found in the Techniques section.

Traceability Analysis

Traceability analysis is the process of linking requirements for a system element to other system elements (higher or lower in the hierarchy), and to the various implementations of the requirements (design, physical components, test procedures). The purpose of this analysis is to ensure that the requirements for the system element (e.g., a complex electronic device) are correct and complete, and that the final implementation of the system element meets those requirements.

Tracing comes in two basic types:

• Forward tracing, from higher level requirements into lower level/derived requirements, and eventually into design (including HDL modules) and test
• Backward tracing from lower-level elements to higher-level requirements

The purpose of forward tracing is to make sure all requirements are completely and correctly implemented in the complex electronics. Backward tracing ensures that additional functionality is not added to the system (i.e., that all system aspects are driven by the requirements). Forward tracing of requirements is the minimal implementation of Traceability Analysis. For critical systems, both types should be used.

Details of traceability analysis can be found in the Techniques section.

Modeling

Modeling is an investigative technique that uses a mathematical or physical representation of a system or sub-system that accounts for all or some of its known properties. Models are often used to test the effects of changes of system components on the overall performance of the system.

Model-based development focuses on creating a complete (and possibly formal) model of the system or sub-system. Models are an abstract and high level description of the system (or system component), expressed as statements in some modeling language or as elements in a modeling tool. Unlike standard requirements documents, models can be executable (able to simulate the process flow within the system).

The standard "requirements, design, implement, unit+ integration+system test" development cycle becomes "requirements, model,verify (test) and debug, physically implement, system test". Testing is pushed earlier in the life cycle to the modeling phase. In theory, the model-driven approach allows developers to construct, test, and analyze their designs before they start to implement them.

One advantage of model-based development is moving some of the testing activities earlier in the life cycle. If major problems are found, they can be resolved with less impact on the budget or schedule of the project. Disadvantages include a reliance on the automatically generated design (which may not be generated correctly) and the difficulty of knowing how well the model conforms to reality. Interactions between parts of the system may not be evident until the system is operational. Testing on the model should not replace thorough system testing.

Simulation

Simulation is the process of executing the requirements (system) model. In a simulation, the user can provide inputs to the model and see how it reacts. Specific, planned tests can be run to show that the model meets the required behavior, or the simulation can be used in an interactive mode to explore the model reactions.

Simulations can be used to validate the requirements, by showing the customer how things will work. The customer may even be provided a computer program (the simulation) to play with and try out various scenarios. Simulations can also be used to understand system or component behavior under various conditions that are not always easy to produce in the real world, such as signal noise, missing inputs, and invalid combinations of inputs.

The strength of simulations are:

• Early validation of the requirements.
• The ability to test the requirements (system) early and identify any errors or ambiguous situations, when they are easier to correct.
• The ability to explore how the system will react under unexpected or invalid inputs. The model (requirements) can be corrected if the actions are not what are desired.

The main problems with modeling and simulation are:

• The time and expense to create the model. Executable models required for simulations often have to be written in specialized languages.
• The difficulty in creating a model when the system is still abstract.
• Results of simulations may not correspond to the real system. The simpler the model, the more likely it is to vary from the real system. The limitations of modeling and simulation need to be understood, and the results of simulation should be considered in the light of the fidelity of the model.

Formal Methods

When the model is formally defined, it becomes “formal methods”. Formal models can be mathematically proven to hold (or not hold) a particular property. The NASA Formal Methods Guidebook states: “Formal Methods (FM) consists of a set of techniques and tools based on mathematical modeling and formal logic that are used to specify and verify requirements and designs for computer systems and software.” Formal Methods therefore has two parts – formal specification and formal verification.

System and complex electronics requirements are usually written in “human-readable” language. This can lead to ambiguity, when a statement that is clear to one person is interpreted differently by another. To avoid this ambiguity, requirements can be written in a formal, mathematical language. This formal specification is the first step in applying FM.

Formal Specification is useful in shaking out the requirements and ensuring that they are consistent and complete. By creating a Formal Specification, you may find errors such as undocumented assumptions, inadequate off-nominal or boundary case behavior, traceability and inconsistency, imprecise terminology and logic errors. Formal Specifications are also the first step in formally verifying that the CE implementation meets the requirements.

UML

A growing trend in systems engineering is to use the Unified Modeling Language (UML) or SysML (the systems engineering extension) to describe the system. In many cases, tools can take the developed model and automatically generate the hardware description language code or other design elements.

The UML is the industry-standard language for the specification, visualization, construction, and documentation of the components of systems. UML helps to simplify the process of system design by making a model for construction with a number of different views. The UML represents a compilation of best engineering practices which have proven to be successful in modeling large, complex systems, especially at the architectural level.

UML defines twelve types of diagrams, divided into three categories:

• Structural Diagrams which include the Class Diagram, Object Diagram, Component Diagram, and Deployment Diagram
• Behavior Diagrams which include the Use Case Diagram (used by some methodologies during requirements gathering); Sequence Diagram, Activity Diagram, Collaboration Diagram, and Statechart Diagram
• Model Management Diagrams which include Packages, Subsystems, and Models.

Criticality Mapping

Criticality mapping identifies complex electronics requirements and design elements that have safety implications. The method starts with a list of system hazards. It then evaluates each system requirement or component in terms of the safety objectives derived for that component. If a system component includes complex electronics, the criticality mapping process is applied to the functions the complex electronics will perform. The end result should be either a) the complex electronics has no safety related objectives or b) a mapping of the safety objectives to particular parts (or all) of the complex electronic device.

The results and findings of the Criticality Mapping should be fed back to the System Requirements and System Safety Analyses. For all discrepancies identified, either the system requirements should be changed because they are incomplete or incorrect, or else the complex electronics requirements must be altered to match the system requirements. The Criticality Mapping identifies additional hazards that the system analysis did not include, and identifies areas where system or interface requirements were not correctly assigned to the complex electronics.

Decision Tables/Trees

Decision Tables (or Trees) are used to describe the required external behavior of some aspect of a system. These tables can be used to make sure that all combinations of events are accounted for in the behavior of the system. For example, if the complex electronics performs operations based on the state of several input signals, you want to make sure that all combinations of those inputs are addressed in the design.

Decision Tables are easy to produce. They are essentially X-Y tables, with the conditions that can influence the decisions on the left, followed by the set of possible decisions (actions). The possible combinations of conditions are in the columns to the right. Decision Tables make no assumptions about the order in which the conditions are evaluated. In the Techniques section, the Decision Table page provides more details and shows an example.

A Decision Tree is basically a graphical Decision Table. Decision Trees can be drawn as a modified flow chart, where each node (decision) has a yes and no branch leading to either another node or an action (decision). One advantage of Decision Trees is that they provide a sequential way to represent how the various conditions are processed. However, since complex electronics are parallel rather than serial, Decision Tables may be a better choice.

Decision Tables and Trees are a good analysis for ensuring that all combinations of input conditions have a corresponding action. Unexpected inputs (or combinations thereof) are quite common in complex systems. The designer (and assuror) of complex electronics cannot assume that the world outside the device will always behave as specified.

Reverse Engineering of Requirements

The level of detail of requirements for complex electronics varies. The project may simply use higher-level requirements to define what the complex electronics must do, and document the more detailed specification within the design. In situations where the complex electronics requirements are unavailable or inadequate, the assurance engineer may need to create a requirements specification for the device, in order to support future analyses. The assurance engineer may also wish to follow this process to independently derive the requirements as a cross-check to the project activities.

Requirements reverse engineering assumes that the project is already past the requirements phase and into design (or a later phase). The assurance engineer has available the following resources:

• System requirements
• Sub-system, component, etc. requirements
• Design of the complex electronics (architecture, behavior, detailed, etc.)
• Interface information from the complex electronics to the rest of the system.

The basic process for reverse engineering is to apply an iterative approach, using the “next level up” requirements and the complex electronics design information. Follow the derivation process in Developing Requirements and then compare the result (complex electronics requirement) against the design. Do they match up? If not, take the design function and create a requirement the design would meet. Consider if that requirement could be a “derived” requirement from the higher-level system element. The end result of this highly manual process is a detailed set of requirements that the complex electronics design meets.

The chances are good that in performing this task you will find areas where higher-level requirements are wrong, ambiguous, or incomplete. You will also likely find some design elements that are incorrect or incomplete with respect to the system requirements. Use the project problem reporting system and/or the requirements change system to iron out the issues and ensure that the appropriate document (requirements or design) is changed.

Fault Tree Analysis

A Fault Tree Analysis (FTA) is a top-down approach to failure analysis that starts with an undesirable event, such as a failure or mishap/accident or malfunction, and then determining all the ways that event can happen. It is a system for identifying causes of hazards, but not the hazards themselves. Fault Trees are often used to look at the hardware aspects of the system for possible failure modes. Traditionally, Fault Trees did not consider software, programmable elements, or operations as possible causes of events.

Fault Tree Analysis should include complex electronics, and consider faults and failures that are design related, as well as actual problems with the silicon chip. The task is to identify all possible events that can lead up to a failure mode. FTAs use a deductive approach to identify critical paths and provide minimal sets of states or critical paths which will lead to the top event.

Including complex electronics in an FTA is a little trickier than strictly addressing hardware failures. The complex electronics can perform multiple functions and operate in many modes or states. It performs different functions at different times.

Details of Fault Tree Analysis can be found in the Techniques section. Specific design issues and faults to consider are being investigated. Guidance for including complex electronics in Fault Trees will be added when the investigation is complete.

Failure Modes and Effects Analysis

Failure Modes and Effects Analysis (FMEA) is a bottom-up method used to find potential system problems while the project is still in the design phase. Each component in the system is examined, and all the ways it can fail are listed. Each possible failure is traced through the system to see what effects it will have, and whether it can result in a hazardous state. The likelihood of the failure is considered, as well as the severity of the system failure.

When complex electronics are included in an FMEA, the key fault modes (ways the design and device can fail) are identified. The effects of abnormalities in the complex electronics on other components in the system, and on the system as a whole are the result of the analysis. This technique is used to uncover system failures from the perspective of the lowest level components. It is a “bottom-up” (or “forward”) analysis, propagating problems from the lowest levels, up to a failure within the broader system.

The FMEA asks “What is the effect if this component operates incorrectly?” Failures for the component are postulated, and then traced through the system to see what the final result will be. Not all component failures will lead to system problems. In a good defensive design, many errors will already be managed by the error-handling part of the design.

Details of Failure Modes and Analysis can be found in the Techniques section. Specific design issues and faults to consider are being investigated. Guidance for including complex electronics in FMEAs will be added when the investigation is complete.