Testing from the Development Point of View
Complex electronics are tested at multiple levels to ensure correct functionality and adherence to other properties. Some tests can only be done at a low level (e.g., at the chip or board level), where faults and failures of external components can be simulated. Some tests require other system elements in order to accurately reflect how the complex electronics will function in the full system. It is the responsibility of the complex electronics designer, with guidance from the systems and assurance engineers, to determine how to partition the tests at the various levels.
This section covers:
Post-Programming Testing
Once the complex electronics is programmed (or manufactured, for ASICs), the device is tested to verify correct programming and functionality. These tests are usually performed in some sort of test fixture where various inputs can be provided to the chip. The main focus of these tests is to verify that the logic in the chip is correct and that no design or programming errors were introduced. The inputs to the device are tested within the correct range, outside the range, and at the edges of the range. Usually, only one input at a time is tested. This type of testing is the equivalent of white box unit testing for software.
If the complex electronics contains embedded software, that software has to be tested along with the hardware device. The software should be of equivalent fidelity as the complex electronics. The interfaces between the complex electronics and the software should be exercised, and any joint operations (where the two work in concert) should be verified to work as expected.
Requirements may be verified at this low level if they cannot be tested at a higher level of assembly. In some cases, this is the only level where certain erroneous inputs can be provided, because the board or other hardware that the chip will be installed in will prevent those inputs from reaching the chip. Fault injection testing is best performed at this level of testing, where simulations of various faults (and combinations of faults) is provided to the chip, and the result (output) is verified to be correct and reasonable.
Coverage of the design elements by the testing effort may be measured as the tests are performed. The ideal is coverage of all possible combinations of inputs and states, which is usually not practical. Coverage of the paths through the device (e.g., processing from an input to an output), decisions, and states (especially transitions between states) is usually adequate. The main advantage of measuring coverage is that you know when to stop testing (i.e., when you've achieved adequate coverage). It also points out areas of the design that are not well exercised.
Burn-in is the process of running an electronic device long enough to pinpoint early failures. This process doesn't help with the design, but it will help weed out a bad chip. Burn-in may happen at the chip level, or later at the circuit board or assembly level.
Integration testing
Integration testing is performed whenever components or assemblies are brought together to form a more complex system element. Testing at this level focuses on the interfaces between the complex electronics and the hardware (e.g., circuit board) it is part of. Software interfaces are also tested, if the complex electronics communicates with external software. (Internal embedded software was tested at the device level.) Timing is an important interface characteristic that needs to be verified.
Requirements that cannot be tested at a higher level are verified at this level of testing. Fault injection testing can be performed to show that the board or assembly functions properly with anticipated faults and errors. The test scenarios should be as realistic as possible, and exercise all aspects of the complex electronics and the circuit board or assembly. Design Coverage may be measured during integration testing, especially if requirements are being verified. Burn-in of the circuit board or assembly may be performed as part of the integration testing.
Environmental testing
Depending on the environment that the system will have to function (or survive) in, the system or sub-systems may undergo various types of environmental tests. These tests include:
- Vibration. Verify that the system element can withstand the vibration environment it may be exposed to without breaking. Systems operating in spacecraft, airplanes, and land vehicles all have vibration requirements.
- Radiation. Verify that the system can survive and function in the radiation environment that it is expected to operate in. Radiation is of most concern to systems in spacecraft and airplanes.
- Acoustic. If the system will be operated where a human can hear it, the level of noise the system can generate may be limited.
- Thermal. Thermal cycling can be used to show that the system can tolerate temperature changes, turn on after a long cold soak, and operate at expected environmental temperatures.
- Vacuum. If the system will be in space or otherwise exposed to vacuum or low pressures, the system is tested at the expected environmental pressure.
- Electromagnetic Interference (EMI). The system is measured for the amount of electromagnetic radiation it puts out (both through radiation and via the power lines). The system is also tested to identify its susceptibility to radiated or induced EMI from the environment (e.g., other systems).
During these environmental tests, the system as a whole is tested. However, the tests should make sure that they exercise the complex electronics primary and critical functions to make sure the device is not effected. This is especially true for EMI, thermal, and radiation tests.
System Testing
System testing is the primary method of system requirements verifications. It may also verify complex electronics requirements, if they were not previously verified (for example, if they depended on other parts of the system). System testing makes sure all the parts work and play well with each other. While some level of off-nominal or fault testing will be performed, the focus is on system events, external failures, and operational problems, rather than on faults at the complex electronics level.
As with environmental tests, system tests will not explicitly test the complex electronics. However, critical or important functions of the complex electronics should be exercised within the system tests. For example, if a complex electronics device is used to determine when a door can be opened, then testing opening the door under several scenarios should ensure that the primary functions of the device are verified.
Acceptance Testing
The Acceptance Test is one way to validate the system. The customer writes the test to make the system perform the desired actions. If the system passes the test, and the customer is happy, then the system is validated.
Guidance for Test Procedures
When tests are used to verify requirements, the rigor of the test procedure and the fidelity of the product under test become important. The following list provides guidance for tests that are meant to be verification activities:
- Test procedures are developed, reviewed, approved, and under configuration management control.
- Each requirement that will be verified should be explicitly listed in the test procedure. Ideally, the steps used to verify that requirement are identified, possibly by listing the requirements in the appropriate section.
- For each test, the testing stimulus, sequence, and test conditions (e.g., applied voltage) is defined.
- Each test or test section has pass/fail criteria used to determine if the requirement is verified or not.
- The configuration identity for the complex electronics and other items under test are recorded in the test information. The design version for the complex electronics is one piece of identifying information. The items under test have to be baselined and configuration managed prior to any requirement verification.
- Test equipment, including simulators of other system elements, should be identified. Version information and calibration data for the test support equipment needs to be recorded.
- The results of the test should be recorded. For manual tests, a filled-out test procedure may be all that is needed. For automatic tests, or ones that rely on analyzing data post-test, the inputs and outputs of the test should be saved. Ideally, all of this information will be gathered and controlled through the configuration management system.
- Problems and anomalies need to be fed into the project problem reporting/corrective action system for resolution.
|
