Assurance Plan for Complex Electronics:

Assurance Process: Testing

This printer-friendly page contains the following sections:

• Testing Phase
• Testing from the Development Point of View
• Assurance Process for Testing Phase

Testing Phase

This Overview page for the Testing Phase contains the following sections:

• Overview
• Testing Process
• Entrance Criteria
• Exit Criteria
• Roles and Responsibilities
• Testing Site Map

Overview

While simulation is used extensively in complex electronic design, testing the actual chip can sometimes be an eye-opening experience. Simulation involves assumptions and compromises that may not match with the real world. Testing the programmed chip - either independently or integrated onto a circuit board - is a necessary step in verifying the design.

In-circuit verification tests the functionality and timing of the design on the actual chip. Ideally, special test software running on a host computer will interface with the device under test through available test ports, such as the JTAG port. This process is similar to in-circuit emulators that run embedded software on the target processor, and provide breakpoints and tracing into the actual software instructions.

The more common form of in-circuit tests is to manually run the complex electronics as part of a higher-level assembly to show that it meets all the specified requirements. This sub-system or system level test will show functionality at a black-box level, but will not provide a window into the internal functioning of the device.

If the complex electronic device is safety-critical, there will be a separate safety verification, usually at the system level.

Testing Process

The diagram below shows the implementation process for complex electronics. The Testing for Developers page describes the engineering process to create the design. The Assurance Process page describes the activities to assess and verify the design.

Entrance Criteria:

The following criteria should be met prior to beginning the testing process.

• The complex electronic device is programmed or manufactured.
• Test procedures are written, approved, and configuration controlled.
• The design (at all levels) is configuration controlled.
• Design documentation is of adequate quality and quantity to allow the test procedures to be reviewed.

Exit Criteria:

At the end of the testing phase, the following criteria should be met:

• All requirements for the complex electronics are verified.

Roles and Responsibilities

The table below describes typical activities during the Testing Phase for both engineers and assurance personnel.

Testing Phase
Role	Typical Activities
Systems Engineer	Define system level tests. Ensure that all system requirements are verified.
Electronics Designer	Define board level tests. Ensure board functions properly in all anticipated nominal and fault conditions.
CE specialist (optional)	Define chip level tests. Ensure device functions properly in all anticipated nominal and fault conditions.
System Safety	Review and approve safety verifications. Witness safety tests. Assess changes for safety impacts.
CE/Quality Assurance	Review and approve device (and higher) test procedures. Witness some or all of the tests. Ensure problems are identified, reported, and properly fixed. Assess design changes for impacts.

Testing Site Map

The table below describes the information contained on the other pages in the Testing section.

Testing Site Map
Overview	This page
Test Process for Developers	The process of testing complex electronics.
Assurance Process	Describes the assurance process for complex electronics testing.

Testing from the Development Point of View

Complex electronics are tested at multiple levels to ensure correct functionality and adherence to other properties. Some tests can only be done at a low level (e.g., at the chip or board level), where faults and failures of external components can be simulated. Some tests require other system elements in order to accurately reflect how the complex electronics will function in the full system. It is the responsibility of the complex electronics designer, with guidance from the systems and assurance engineers, to determine how to partition the tests at the various levels.

This section covers:

• Post-programming Testing
• Integration Testing
• Environmental Testing
• System Testing
• Acceptance Testing
• Guidance for Test Procedures

Post-Programming Testing

Once the complex electronics is programmed (or manufactured, for ASICs), the device is tested to verify correct programming and functionality. These tests are usually performed in some sort of test fixture where various inputs can be provided to the chip. The main focus of these tests is to verify that the logic in the chip is correct and that no design or programming errors were introduced. The inputs to the device are tested within the correct range, outside the range, and at the edges of the range. Usually, only one input at a time is tested. This type of testing is the equivalent of white box unit testing for software.

If the complex electronics contains embedded software, that software has to be tested along with the hardware device. The software should be of equivalent fidelity as the complex electronics. The interfaces between the complex electronics and the software should be exercised, and any joint operations (where the two work in concert) should be verified to work as expected.

Requirements may be verified at this low level if they cannot be tested at a higher level of assembly. In some cases, this is the only level where certain erroneous inputs can be provided, because the board or other hardware that the chip will be installed in will prevent those inputs from reaching the chip. Fault injection testing is best performed at this level of testing, where simulations of various faults (and combinations of faults) are provided to the chip, and the result (output) is verified to be correct and reasonable.

Coverage of the design elements by the testing effort may be measured as the tests are performed. The ideal is coverage of all possible combinations of inputs and states, which is usually not practical. Coverage of the paths through the device (e.g., processing from an input to an output), decisions, and states (especially transitions between states) is usually adequate. The main advantage of measuring coverage is that you know when to stop testing (i.e., when you've achieved adequate coverage). It also points out areas of the design that are not well exercised.

Burn-in is the process of running an electronic device long enough to pinpoint early failures. This process doesn't help with the design, but it will help weed out a bad chip. Burn-in may happen at the chip level, or later at the circuit board or assembly level.

Integration testing

Integration testing is performed whenever components or assemblies are brought together to form a more complex system element. Testing at this level focuses on the interfaces between the complex electronics and the hardware (e.g., circuit board) it is part of. Software interfaces are also tested, if the complex electronics communicates with external software. (Internal embedded software was tested at the device level.) Timing is an important interface characteristic that needs to be verified.

Requirements that cannot be tested at a higher level are verified at this level of testing. Fault injection testing can be performed to show that the board or assembly functions properly with anticipated faults and errors. The test scenarios should be as realistic as possible, and exercise all aspects of the complex electronics and the circuit board or assembly. Design Coverage may be measured during integration testing, especially if requirements are being verified. Burn-in of the circuit board or assembly may be performed as part of the integration testing.

Environmental testing

Depending on the environment that the system will have to function (or survive) in, the system or sub-systems may undergo various types of environmental tests. These tests include:

• Vibration. Verify that the system element can withstand the vibration environment it may be exposed to without breaking. Systems operating in spacecraft, airplanes, and land vehicles all have vibration requirements.
• Radiation. Verify that the system can survive and function in the radiation environment that it is expected to operate in. Radiation is of most concern to systems in spacecraft and airplanes.
• Acoustic. If the system will be operated where a human can hear it, the level of noise the system can generate may be limited.
• Thermal. Thermal cycling can be used to show that the system can tolerate temperature changes, turn on after a long cold soak, and operate at expected environmental temperatures.
• Vacuum. If the system will be in space or otherwise exposed to vacuum or low pressures, the system is tested at the expected environmental pressure.
• Electromagnetic Interference (EMI). The system is measured for the amount of electromagnetic radiation it puts out (both through radiation and via the power lines). The system is also tested to identify its susceptibility to radiated or induced EMI from the environment (e.g., other systems).

During these environmental tests, the system as a whole is tested. However, the tests should make sure that they exercise the complex electronics primary and critical functions to make sure the device is not affected. This is especially true for EMI, thermal, and radiation tests.

System Testing

System testing is the primary method of system requirements verifications. It may also verify complex electronics requirements, if they were not previously verified (for example, if they depended on other parts of the system). System testing makes sure all the parts work and play well with each other. While some level of off-nominal or fault testing will be performed, the focus is on system events, external failures, and operational problems, rather than on faults at the complex electronics level.

As with environmental tests, system tests will not explicitly test the complex electronics. However, critical or important functions of the complex electronics should be exercised within the system tests. For example, if a complex electronics device is used to determine when a door can be opened, then testing opening the door under several scenarios should ensure that the primary functions of the device are verified.

Acceptance Testing

The Acceptance Test is one way to validate the system. The customer writes the test to make the system perform the desired actions. If the system passes the test, and the customer is happy, then the system is validated.

Guidance for Test Procedures

When tests are used to verify requirements, the rigor of the test procedure and the fidelity of the product under test become important. The following list provides guidance for tests that are meant to be verification activities:

• Test procedures are developed, reviewed, approved, and under configuration management control.
• Each requirement that will be verified should be explicitly listed in the test procedure. Ideally, the steps used to verify that requirement are identified, possibly by listing the requirements in the appropriate section.
• For each test, the testing stimulus, sequence, and test conditions (e.g., applied voltage) are defined.
• Each test or test section has pass/fail criteria used to determine if the requirement is verified or not.
• The configuration identity for the complex electronics and other items under test are recorded in the test information. The design version for the complex electronics is one piece of identifying information. The items under test have to be baselined and configuration managed prior to any requirement verification.
• Test equipment, including simulators of other system elements, should be identified. Version information and calibration data for the test support equipment needs to be recorded.
• The results of the test should be recorded. For manual tests, a filled-out test procedure may be all that is needed. For automatic tests, or ones that rely on analyzing data post-test, the inputs and outputs of the test should be saved. Ideally, all of this information will be gathered and controlled through the configuration management system.
• Problems and anomalies need to be fed into the project problem reporting/corrective action system for resolution.

Assurance Process for Testing Phase

The assurance role during the testing phase is to ensure that the tests are adequate to verify the requirements. Beyond that, good assurance engineers help the project define tests that exercise the system, or system element, in various situations, including fault and failure conditions. The real world is never perfect, and systems need to be able to respond appropriately to the environment they are operating in.

Use the Tailoring chart to determine which activities or analyses are required for a particular criticality classification. Activities that are not required may still be performed, if desired. Assurance activities for complex electronics in the testing phase include:

• Test verification (procedure review/approval/test witnessing)
• Problem Trend analysis
• Process Verification
• Functional and Physical Configuration Audits
• Update Risk analysis
• Update Traceability analysis
• Update Criticality Mapping

The table below uses the Complex Electronics Classification to map the activities, and depth of each activity, against the classification. This table allows for easy tailoring of the assurance activities to the device complexity and criticality.

Tailoring Guidance for Assurance Activities - Testing Phase
	Low	Moderate	High
Test Verification	Review/approve procedure. Witness test or review test results.	Review/approve procedure. Witness test and review test results.	Review/approve procedure. Witness test and review test results. Ensure rigorous testing.
Problem Trend Analysis	Not performed	Review problem reports occasionally	Formal trend analysis
Process Verification	Informal	Moderately formal	Formal Audits
FCA/PCA	Performed	Performed	Performed
Risk analysis	Informal	Informal	Formal
Traceability Analysis	Ensure all requirements trace to test, analysis, or inspection.	Ensure all requirements and design elements trace to test, analysis, or inspection.	Ensure all requirements and design elements trace to test, analysis, or inspection. Perform backward trace.
Criticality Mapping	Not performed	Update with test procedure and steps.	Update with test procedure and steps.

Test Verification

During testing, the assurance engineer is responsible for reviewing the test procedures (and approving them, in most cases) to ensure that the tests adequately verify the requirements. In addition, the assurance engineer often witnesses the tests, verifying that the success criteria are met for the requirements. Alternately, the assurance engineer may review the test results to ensure that testing occurred as planned.

Test related activities for the assurance engineer include:

• Verify that the testing strategy has been documented in a plan and/or procedure, and that testing occurs according to the plan.
• Verify that the planned tests will completely verify the requirements in all reasonably expected situations. This includes verifying the functionality and performance in nominal situations and when other parts of the system have errors. How gracefully does the device handle errors it may encounter? How gracefully can it handle any internal faults?
• Verify that the planned tests will exercise all modules or other division in the device. Not every level of testing has to exercise all modules, but each module should be tested at some level (device, circuit board, sub-system, or system).
• Review the test procedures for feasibility, especially if the complex electronics is integrated with other system elements. Can the tests be performed without risk to the system components? Are test points available to allow access to signals required by the procedure? Is the test operating the device in an operational mode, as close as possible to how it will be operated within the system?
• Review the test plans and procedures to identify any areas where testing is weak. You are looking for modules that are only minimally tested, requirements that are only verified under some circumstances, and other areas where additional testing may be helpful.
• Witness tests (as agreed to in the project plans) and document any anomalies and problems.
• Review the test results to verify that no unnoticed anomalies occurred. Sometimes during testing many events are occurring and an anomaly unrelated to the aspect of the particular test may be missed.
• Ensure that any embedded software in the complex electronics is verified, and that the interfaces between the software and the complex electronics are correct.

One area to pay particular attention to is Commercial Off-the-Shelf (COTS) or re-used IP (Intellectual Property) modules or cores. Make sure the tests show that the IP module is accessed correctly and that it performs as specified. Also ensure that testing verifies that the unused functions, if activated within the device, do not cause a safety hazard or other critical problem.

Problem Trend Analysis

Problem Trend Analysis identifies repetitive problems and assesses how often given problems occur. It also provides a mechanism to track progress of problem resolution. The main objective of this analysis is locating where key problems are occurring and the frequency of occurrence.

Problem Trend Analysis is more of a system-wide activity, rather than focused solely on complex electronics. As such, it should be performed by the quality assurance or systems engineer, to understand where problems are occurring. Regardless of who performs the analysis, a knowledgeable assurance engineer needs to review the problem reports that relate to the complex electronics (and the board, etc. that the chip is part of). Pay particular attention to problems that could indicate design errors in the complex electronics. Also note the number of unexplained anomalies that might relate to the device.

More detail on Problem Trend Analysis can be found in Section 8.2 of NASA Reference Publication 1358, System Engineering "Toolbox" for Design-Oriented Engineers

Process Verification

Process assurance activities for this phase include:

• Verify the defined processes are in place and are being followed correctly.
• Verify configuration management is functioning properly to control revisions to the design that may occur during testing activities.
• Verify that problems and anomalies are being recorded in the project problem reporting/corrective action system, and that the problem resolutions are correct, approved, and properly implemented.
• Perform a level of impact analysis for any changes to the design, considering the testing that has occurred before and the possibility of affecting other parts of the system.
• Ensure that all safety verifications are performed. Keep the system safety engineer apprised of any design changes that may affect safety.

Functional and Physical Configuration Audits (FCA/PCA)

The FCA is the formal examination of the "as-tested" functional characteristics of a configuration item (CI). The audit verifies that the item meets the requirements specified in its functional baseline documentation. Any discrepancies are identified and recorded. Functional configuration audits also assure that the technical documentation accurately reflects the functional characteristics of the device. To perform an FCA, the test procedures and test results used to perform testing are examined against the specifications.

The PCA is the formal examination of the "as-built" configuration of a configuration item (hardware and software) against its technical documentation. The PCA normally includes a detailed audit of engineering drawings, specifications, and technical data (including COTS documentation). For complex electronics, the design documentation, such as the HDL code, should be audited as well. The PCA for a CI is not performed until the FCA has been completed.

Update Analyses

Analyses previously performed should be updated at this time.

Risk Analysis

Evaluate previous risks to identify those that no longer apply or that have changed their priority based on changes in probability or impact. Identify any new risks relevant to this phase of development and determine which require mitigation plans. Check that preventive measures and/or contingency plans exist for all identified risk items and that the risk, with mitigations in place, is acceptable for completing the Testing phase and going operational.

Traceability Analysis

Complete the requirement tracing into the test procedures where the requirements are verified. For high criticality devices, trace backward from the test procedures to ensure that testing is focused on the requirements (what the system has to do) and the design (how it will do those activities). Coverage analysis may be performed to ensure that all elements of the design are exercised in at least one test. Any requirements that are not verified in a test (or by analysis or inspection) should be reported to the project, to ensure that the requirements are included in future tests.

Criticality Mapping

Criticality mapping identifies complex electronics requirements and design elements that have safety implications. At the testing phase, the previously developed matrix is extended to show the test procedures and steps that verify the critical requirements (and therefore are safety related). This mapping should be provided to the system safety engineer, who may choose to witness the tests or review the results.

Other Analyses

The other analyses, FMEA, FTA, and Interface, do not require updates during this phase, unless there is a design change.