This paper presents the evolution of test philosophies and procedures used in preflight checkout of Mercury spacecraft at Cape Canaveral, Fla. The impact on preflight operations of tight schedules, mission changes, discrepant performance of ground and spacecraft equipment, and new information gained from ground testing and flight are discussed. Included in this discussion are numerous examples to illustrate the kinds of problems that were encountered and their effects on preflight operations. In addition, this paper presents the lessons learned in preflight preparation and checkout over the 4-year span of the program.
Test operations personnel learned that only formalized testing with all inter-dependent systems operating simultaneously would provide a flight- ready spacecraft. Tests emphasized astronaut safety and included participation of the astronaut as often as possible. Few substitutes for actual flight equipment were permitted during spacecraft assembly, rigging, and testing. Such matters of quality control as cleanliness, component limited-shelf and limited-operational life, and equipment failure, influenced the test philosophy. Validation and troubleshooting of spacecraft systems revealed the need for many more test points to be provided for in-place testing. Repair and bench testing of failed equipment reemphasized that the equipment needed to be made more accessible for removal and reinstallation. Rapid feedback of test results and failure analyses to design and manufacturing personnel was necessary and led to the increase of inspection and on-the-spot failure analysis. Digital checkout equipment was developed and proved that digital computer systems were superior to analog methods in providing information and control to test engineers.
Preflight preparation and checkout experience began at Cape Canaveral in 1959 with the Big Joe boilerplate spacecraft. This spacecraft was the first to be launched in Project Mercury. The Big Joe spacecraft was designed and built by the National Aeronautics and Space Administration (NASA) to determine the aerodynamic and heating characteristics of the Mercury shape.
In the following year, a variety of test and checkout equipment and the first production spacecraft arrived at Cape Canaveral. During the next 2 years, the techniques and procedures for preparation and checkout of spacecraft for manned flight were developed and refined.
By the time of the early maimed flights, these preparations and procedures had been proved through operational experience. A formal but flexible operations routine had evolved, incorporating close coordination with design, mission management, manufacturing, and quality control groups. For example, components were inspected and tested before installation; and work to be done on the spacecraft was described in detailed work sheets. This procedure controlled the disturbance of spacecraft components and assured that the status of the spacecraft configuration was known at all times. Detailed test procedures had been written, and step-by-step test results recorded. Checklists had been established to guide spacecraft assembly and configuration before each test.
 In preparation for spacecraft testing, component mockups and simulators were constructed and used as substitutes for components that were fragile, dangerous to handle, or in short supply. However, it was found that these mockups and simulators could not be constructed accurately within a reasonable cost and time schedule, and therefore they proved to be of marginal value.
For example, wooden pyrotechnics mockups did not properly establish cable fits, and substitute escape towers did not establish clearances. This resulted in delays and difficult working conditions when modifications had to be made while at the launch pad. Other simulators did not work because of the high packaging density and multiple interfaces inside the Mercury spacecraft.
Ultimately, it was deemed necessary to fitcheck all flight items simultaneously, and, where substitutes had to be used, exact flight types were required. Because better facilities for mechanical modifications were available at the point of manufacture, experience indicated that complete assembly of the spacecraft should be accomplished at the factory; this was true even for those components which had to be removed for shipment.
In the early planning stages of Project Mercury, it was thought possible to deliver flight-ready spacecraft to Cape Canaveral, conduct a single, total spacecraft test in the hangar, and launch very soon thereafter. However, it was demonstrated that more preflight preparation was required at the launch site and formal procedures evolved from experience.
Before spacecraft testing was begun, very careful preparations were made. Each step had to be formalized through configuration documents, checklists, and test procedures. The ground-support. equipment was tested to prove its readiness. Test complexes were checked for compatibility with the particular spacecraft. The spacecraft was put into test position and its configuration conformance to test plans was established; of particular concern were proper cabling and plumbing of all systems. Then the spacecraft was connected to the complex and testing was begun.
Various efforts were made to accelerate preparation of the spacecraft; for example, when the spacecraft was idle, as during periods when data were being analyzed, efforts were made to continue work on the apparently-unaffected systems. However, it was found that this work would adversely affect the test setup and there by the spacecraft preparation schedule. Mercury components were so closely packed that there was little room for a man to work inside the spacecraft without accidentally damaging such things as cables, tubing, connectors, or cameras. Generally, it was ruled that only test-associated work would be done on a spacecraft while it was being tested.
Early in the program before systems interrelationships had been completely analyzed, some equipment was damaged when tests of one system influenced another. For example, reaction control system (RCS) valves in a dry state overheated when activated by the automatic stabilization and control system (ASCS).
As test crews and planners gained experience in attending to these many details, test plans became more reliable. Offsetting this experience were the number of modifications made to the spacecraft to accommodate mission flexibility and safety and to improve systems performance. As a result, plans and procedures were constantly changing.
Gradually, a set of guidelines evolved which were used as the basis for all testing. Two principles served as foundations for checkout procedures throughout this evolution. The safety of the astronaut was considered foremost, and secondly, all philosophy was directed toward a test plan which would guarantee a flightworthy spacecraft at lift-off. These were expanded to six principles which were applied to all spacecraft tests.
Building block approach to testing. The operational status of each system and each component in the system was functionally verified before that system was operated concurrently or in conjunction with another system with which it might have an interface.
End-to-end testing. During testing, the initiating function and end function took place  sequentially as would actually occur in flight. The use of artificial stimuli was minimized. Implementation of this guideline was most evident in the hangar-simulated flight test.
Isolation and functional verification of all redundancies. All redundant signal paths were isolated and functionally proven by end-to-end tests. These included redundancies between the spacecraft and launch vehicle and redundancies within the launch complex.
Interface testing and verification. There were two basic interfaces in Mercury: The spacecraft to launch vehicle and the space vehicle to ground complex. These interfaces included RF, hardwire, and mechanical features. Tests involving these interfaces were consistent with the test philosophy previously discussed, namely, end-to-end testing and testing of all redundancies.
Mission profile simulation. Simulated mission tests, which included the spacecraft, launch vehicle, and ground complex, were designed to approach functionally actual mission conditions as nearly as possible. This procedure included simulating real-time functions through orbit insertion. The astronaut was aboard for these
simulations and functioned as he would during the actual flight. These simulated flights were made both in Hangar S and at the launch pad. Figure 14-1 shows preparations being made for a simulated flight test of a spacecraft in the altitude chamber.
The astronaut as an integral part of the system during tests. The astronaut was considered part of the total system and functioned during systems test and mission simulations as he would during the actual mission. This resulted in a dual advantage. The system tested was closer to flight configuration when the astronaut was included, and the astronaut became intimately familiar with the spacecraft and spacecraft system. Figure 14-2 is a photograph of a spacecraft on the RF tower for communication tests. This test, with Astronaut L. Gordon Cooper, was made to determine voice clarity under simulated flight conditions.
Components of proven design were planned for use in Project Mercury. In order to insure that only properly-operating items of these proven designs were used for flight, not only spare parts but also many components installed in the spacecraft were subjected to testing at  Cape Canaveral. Numerous component failures were experienced during these spacecraft tests; in addition, during 1962 and 1963 approximately 50 percent of component spares were rejected after testing. These failures were discovered by rigorous preinstallation acceptance (PIA) tests which in most cases exceeded the test conditions that could be achieved after component installation. These tests increased confidence that replacement components in a state of incipient failure would not be installed in Mercury spacecraft.
Control of the stock of spares also was tightened to assure that qualified replacements were available when troubles occurred. Spares which were significantly affected by shelf life were periodically tested and returned to stock. In special cases, the inspection was extended to the source. Vendor and component manufacturing plants were visited by engineers and inspectors to convey the nature of problems and to encourage higher quality of work.
Component acceptability and rejection rate was governed by such factors as performance criteria modification, the delivery of partially qualified items, and inadequate shelf life of some components.
During systems tests of Mercury spacecraft, the effects of electrical- current surges demonstrated the need for performance criteria modification. These current surges, resulting in momentary variation (transients) of the battery voltage, occurred during testing and were attributed to the normal starting of mechanical-electrical systems, such as the orbital timing device or the spacecraft cameras. As a result of these voltage transients, energized timers would occasionally exhibit early time-out, interference would appear in instrumentation amplifier outputs causing faulty indications, and noise would appear in the astronaut voice channels. The solution to these problems required inclusion of a special battery for the maximum altitude sensor, the addition of capacitors to circuits with time relays as a protective measure to prevent early time-out, and the replacement of components with dike items that demonstrated low susceptibility to voltage transients.
Extensive voltage-transient tests were added to spacecraft checkout procedures to prove equipment immunity to these effects. These procedures were frequently quite involved.
High voltage problems were encountered in, ground power supplies. Electrically-regulated power supplies operating over long lines tended to react and surge to high voltages and cause loss of remote sensing, and they were abandoned in favor of lead-acid batteries.
Equipment not fully qualified for flight when delivered included tape recorders, ECG amplifiers, water traps for the environmental control system (ECS), and impedance pneumographs.
Tape recorders were brought to acceptable status by careful shop assembly and testing. The electrocardiogram (ECG) amplifiers were redesigned and rebuilt. Internal voltage regulators and feedback loops were modified. Rebuilt units were requalified in detail. Later, new models were specified for vendor development based on experience with the originals.
In some cases, flight systems had to he modified to accept new components, and special systems tests had to be devised to be sure that the effects of the modification were not serious. For example, the environmental control system performance depended upon closely balanced pressures and flow, and testing of the system was conducted to insure that the addition of the water trap did not degrade system performance in the MA-9 spacecraft.
Many components procured for the Mercury Project proved to have inadequate shelf life because of the time period required to complete the program. Such items as rockets had to be refurbished because of an 18- month shelf limit established by the vendors. A system was established whereby equipment that utilized rubber O-rings was periodically exercised or returned to the vendor for replacement of time-critical components. For example the Environmental Control System (FCS) negative- pressure relief valves were operated at prescribed time intervals, and the Reaction Control System (RCS) relief valves were reconditioned by the manufacturer.
Another problem involved deterioration of the solder connections of nichrome bridge wires  to the electrical connectors of many pyrotechnic initiators. This deterioration resulted in a gradual increase in the resistance of squib circuits. This was precluded by establishing a time limitation of 6 months between date of soldering and actual use.
The need to prove the acceptability of spacecraft equipment required many changes in procedures and test equipment at the Cape. Component and systems simulators were used less and less in spacecraft testing even to the point of requiring participation of the astronaut, fully suited.
Bench simulators were made more realistic. Voltage-transient generators and ECG simulators were added to the Cape's test equipment. Battery source impedance and load impedances were more carefully simulated as was line noise character. There was, also, an increased tendency to operate equipment on the bench in exact connection with production models of their companion components and systems. Camera solenoids and transmitters were properly connected to and operated with instrument systems during final bench tests before spacecraft installation.
Some equipment and components such as rockets and pyrotechnics could not be fully validated before use. Reliability of these items was almost entirely dependent upon good design, workmanship, and qualification. This workmanship requires great patience and attention to detail even under the tedium of production lines. Considerable progress has been made in promoting this extreme attention to detail which directly contributes to the success or failure of each spaceflight mission and the safe return of the astronaut.
In the MA-9 mission, the three retropackage umbilicals and one of the two spacecraft-to-adapter umbilicals failed to separate from the spacecraft. Each of these contained two squibs, and initiation of either squib, should have resulted in explosive disconnection. Postflight analysis revealed that the umbilicals failed to separate from the spacecraft because the squibs were not loaded with the appropriate charge.
As the Mercury Project matured, it became evident that the use of proven and qualified components did not result in the reliabilities desired to satisfy man-rated system requirements. A more-rigid quality control procedure was required in all aspects of component and system assembly. The need for this requirement was typified by the number of discrepancies (performance or configuration deficiencies) to be corrected for the MA-9 backup spacecraft. A total of 720 system or component discrepancies were recorded, of which 526 were directly attributed to n lack of satisfactory quality of workmanship. Of this number, 444 required specially-scheduled time to correct.
Additionally, flight-safety considerations required that inspection be made of all parts and components scheduled for the space-flight program. These inspection requirements extended from the parts vendor to the Cape in order to locate and reject every defective or marginal part.
As a result of inspection, fourteen 1,500 watt/hour storage batteries were rejected for case leakage during preparation for the MA-8 mission. Several incidents of leakage were due to tooling holes not being plugged during manufacture. Others were from case cracks or undetermined sources. Also, an inspection of battery vent-pressure relief valves revealed dimensional deviations in valves after assembly, and improperly applied potting adhesive. Three of these valves failed to operate at proper pressures. These defective batteries were rejected following inspection at the Cape.
Five failures were experienced on gas-pressure regulator assemblies in the MA-S reaction control system (RCS). An investigation of the failed assemblies revealed that internal scratches, inadequate cleanliness, and improper torquing of end caps were contributing causes of these failures.
Also in the RCS, an examination of failed gas-pressure vent valves revealed damaged O-Rings. In one case an O-ring was found to be scuffed, while in another case the O-ring had a metal fragment driven into the material.
The MA-8 spacecraft was demated from its launch vehicle and returned to Hangar "S" to replace the manual selector valve in the RCS clue to a leakage encountered during a preflight pressure check of the system. Upon removing the valve, it was noted that the valve had been  installed out of alinement so that an excessive side-load was induced into the valve internal parts.
During an inspection of the escape-tower wiring for the MA-6 primary and backup spacecrafts, it was found that the electrical connectors had improperly-soldered joints. Additionally, it was discovered that improper insertion of the conductor wire into the solder had been made as a result of the use of an 18-gauge wire with a 20-gauge solder pot.
In Project Mercury, thousands of man-hours were expended in testing, calibration, assembly, and installation of a variety of hardware that later failed to meet performance specifications or that malfunctioned during systems tests in a simulated space environment. When malfunctions occurred or when these components failed to meet specifications, it was necessary to remove, repair, or replace them, a procedure which could have been avoided in a large percentage of cases if adequate attention to detail during manufacture or thorough inspection before delivery had been exercised.
Mercury spacecraft were literally packed with equipment and components were installed three deep in some instances. Limited interior working space, which posed a severe handicap to preflight preparation, resulted in a certain amount of wiring and equipment damage during normal work and test operations. Repair was a continuing work item during all phases of spacecraft modification and checkout. Any system affected by these repairs or modifications had to be reverified by test.
Extensive changes and modification were caused primarily by component or subsystem malfunction, as well as extension of mission requirements.
As an example, it became necessary to replace MA-6's life-limited carbon dioxide absorber in the ECS, since more time than had been planned was required to check out the system. This replacement required no less than eight major equipment removals and four revalidations of unrelated subsystems. It caused an overall delay of nearly 12 hours. By way of comparison, it took only an hour and a half to replace the carbon dioxide absorber itself. Ten and one half hours were used to gain access to the absorber and then to restore the spacecraft to its original condition.
The number of removals of equipment is an index of the amount of modification, repair, and servicing required as the program progressed. Figure 1 l-3 exemplifies the amount of work required at Cape Canaveral.
Early in the Mercury program motion pictures of the inside of a spacecraft in orbit showed washers, wire cuttings, bolts, and alligator clips floating in the cabin. The cabin fall became plugged on an early unmanned flight with similar free-floating debris.
Such evidence led to mole care in the habits of technicians working inside the spacecraft. A periodic tumbling of the spacecraft to dislodge and expose dirt and loose objects became standard practice at the Cape. Figure 14-4 shows a spacecraft in the tumbling fixture during a cleaning operation, and figure 14-5 shows the debris removed.
Technicians generally were not aware of the strict cleanliness required in handling components of the FCS and RCIS systems. It became necessary to specify handling procedures for these highly dirt-sensitive components. Many parts were kept in sealed plastic hags until installation was to begin, at which time ultra-clean handling methods were used. Vendors have delivered many items of spacecraft equipment which contained wire ends, solder balls, and stray hardware. Such items as gums, powders, lubricants, chips, and hydrocarbons have appeared on components where they could not be tolerated for proper operation. Hydrogen peroxide systems have yielded some decomposables which could have caused extreme reaction. Breathing oxygen and drinking water also have been contaminated. As a result, all consumables were chemically analyzed before being put into their spacecraft containers, and a variety of equipment which was found to contain contaminating deposits was carefully inspected and cleaned before being used. This equipment included astronaut suits, valves, hoses, and tubing.
It can be seen from the preceding discussion that there was a great need for troubleshooting spacecraft components both in spacecraft and on the bench. As a consequence, facilities at the Cape were expanded to include a malfunction investigation laboratory, staffed with experienced specialists in such areas as X-ray, spectroscopy, microscopy, and chemistry. Also, because of the need to qualify many spacecraft components, a laboratory of test equipment was developed and fully equipped with environmental chambers, shaker table, accelerator, impact tester, and pressure testing equipment.
In Project Mercury, it was necessary to add ground support equipment to that provided with the spacecraft and to devise means for testing individual components in the spacecraft. Test points were not available and interconnecting cables and tubing had to be broken into for tests. This invalidated the very circuit or pressure system that was being tested.
In Project Mercury it became necessary to plan exact steps and equipment configuration before troubleshooting was begun. Expected values in response to stimuli that had been carefully defined were listed in documents prepared in advance. It was found that preplanned troubleshooting procedures significantly reduced testing time.
III addition to drawings and standard spacecraft test procedure, the contractor provided logic diagrams and detailed drawings and specifications of systems and components. Systems consisting of many separate circuit elements were detailed by separate subsystem diagrams showing all wire routing throughout the spacecraft. As an example, instrument systems were broken down to show each sensor and  its signal conditioning and cable connections. This was in addition to overall system cabling diagrams and detailed drawings of repairable items including mounting details that were provided. These drawings were invaluable for troubleshooting.
To accomplish spacecraft preflight preparations and checkout within schedule objectives, it was necessary to increase the number of spacecraft undergoing preflight preparations at Cape Canaveral from one to two, or three, at any given time. This approach provided additional time for preflight preparations of each spacecraft without a corresponding increase in the time interval between successive missions.
Preflight preparations and checkout operations included: (1) modifications to update the untested spacecraft configuration based upon knowledge gained from previous flight experience; (2) modifications as extensive as reworking a spacecraft from a suborbital configuration to an orbital configuration to in crease spacecraft systems capabilities for extended mission requirements; and (3) changes and modifications resulting from component or system malfunctions during preflight testing.
Changes of considerable magnitude were made at the Cape only because it was more efficient and less time consuming than returning the spacecraft to the factory. In any event the final flight configuration of any particular spacecraft could not be entirely determined until successful completion of the preceding mission. This required that final configuration changes be accomplished at the Cape if the schedule was to be maintained.
On the average, spacecraft modifications accounted for more than half the time that the spacecraft remained at the Cape prior to flight. Examination of the average time that the spacecraft spent at Cape Canaveral shows that 60
 percent of the total time involved modification, repair, assembly, service, and inspection. Only 25 percent of the time was spent in hangar tests and 15 percent for all work and testing at the pad.
When converted to months, these percentages show approximately 3 months for hangar work, 1 1/3 months for actual hangar testing, and 1 month on the launch pad, for a total average time at the Cape of approximately 5 1/3 months.
The Mercury-Atlas spacecraft averaged 33 more hangar work days, (; additional hangar test days, and 4 additional days on the pad than the Mercury-Redstone spacecraft. This small increase in test and pad time indicated that the increased complexity of Mercury-Atlas missions was offset by increased experience, and therefore there was little effect on time required for hangar testing or total pad time. Figure 14-6 shows how the time was divided between testing and other work in the preparation of spacecraft 16 (MA-8).
Equipment having a short operational life caused many problems in meeting spacecraft checkout schedules. Dissipating chemicals, such as lithium hydroxide which was used to adsorb carbon dioxide in the astronaut's breathing-oxygen circuit, exemplify this problem. The carbon dioxide and oxygen partial-pressure sensors are another example. The latter had to be installed late in the countdown and required some spacecraft disassembly for installation. If holds occurred during the countdown, a new removal-activation-calibration- installation cycle had to be completed.
Spacecraft cameras also posed short-life problems in the Mercury Project. Most of the camera shutter and film advance mechanisms failed frequently. Complicating matters further was the fact that the camera solenoids caused momentary creep reductions in battery voltage because of periodic pulsing. In addition, excessive back-voltages in the coils caused failures of programer switches. Because cameras placed transients on other systems, these systems had to be tested often to establish their ability to accept these transients. This extended use, III turn, burdened the cameras and their failures accelerated. Spare cameras soon became unavailable.
Rapid failure detection and corrective action were basic requirements in maintaining program schedules. Facilities were provided to permit extensive failure analyses at Cape Canaveral where failure conditions were intimately known to engineers and technicians. These analyses provided a basis for accurate feedback through quality control channels. However,, the lack of spares often made it impossible to take the necessary action required in order to meet preflight preparation and test schedules.
One of the more time consuming problems during Project Mercury was that of spacecraft preparation for testing. This required connecting a large number of cables to the installed equipment, largely by breaking cables and tubing. As shown in figure 14-7, many of
these cables were draped around the spacecraft and through the hatch. Not only did this tend to invalidate systems being tested, but it required much detailed planning and preparation of the spacecraft and test equipment. Each time the spacecraft was moved, the entire test complex had to be torn down, putting another strain on equipment in the spacecraft.
A study of digital equipment proved that data conversion provisions could be built into  the spacecraft and permanently wired to test points. Pulse code modulation-a form of digital data-allows compression of data so that hundreds of measurements can be sent over one cable. By this means, test configuration problems can be greatly reduced. Spacecraft can be moved from place to place with much less breakdown and buildup of test configurations.
As the Mercury Project progressed, many different methods of data presentation and distribution were investigated Through these efforts it was demonstrated that data of the pulse form could be consistently transmitted by radio frequency or by cable. The receiving and conversion equipment which was used for about 4 years proved to be a reliable and accurate means for presenting data to test engineers.
In an effort to make data more immediately useful to these engineers, printers and digital displays were added. It was gradually realized that this immediacy was of prime importance to men who had to make constant decisions as to the state of their systems. These studies of improvements needed in spacecraft checkout led to the design of a digital computer-controlled system capable of automatic checkout of manned spacecraft. It also had the capability of use as a completely manual system controlled by test engineers miles from the spacecraft, thereby allowing a natural evolution to automatic checkout. An experimental model was assembled and proved during hangar and pad tests of Astronaut Cooper's spacecraft.
Close coordination between preflight operations personnel and those of other organizations, including contractors, subcontractors, vendor s the Department of Defense, the ground complex the network, and other NASA centers was maintained continuously to insure that interface compatibility of operations planning as well as equipment left no significant problems unresolved.
Throughout Project Mercury, a continuing series of technical, configuration, and mission review meetings were conducted to resolve problems to initiate action where necessary, to coordinate activities on a wide variety of matters affecting each Mercury mission and to
provide technical direction where required. In addition, engineering specialists from these organizations met frequently at Development Engineering Inspections (DEI) at the spacecraft manufacturing plant to review in detail the hardware being produced so that results of current experience were reflected in this hardware. Each of these efforts provided management a valuable tool for directing effort along the desired channels. A significant management device employed during the project to assure mission success was the spacecraft Flight Safety Review meeting held prior to each launch. At this meeting, every activity connected with the spacecraft was discussed in detail between management and engineering specialists to determine the flight readiness of the spacecraft. The criteria established for this review was that a Mercury launch would not take place in the face of unresolved difficulties that might affect mission success or flight safety.
The foregoing discussion has presented in detail the lessons learned in preflight preparation and testing during the Mercury Project. The conclusions that have been drawn from this experience follow:
(1) Test procedures incorporating the techniques of end-to-end testing, interface verification, and astronaut participation should be documented and continually updated.
(2) Spacecraft should be completely assembled at the factory, using a minimum of simulated hardware.
(3) The number of component malfunctions during testing proved the need for better quality control and inspection procedures.
(4) The lack of test points, spares, and formalized troubleshooting. procedures often hindered rapid malfunction analysis and corrective action.
(5) Limited shelf life and operational life of components create spares problems and possible delays in launch schedules.
(6) Pursuance of exacting cleanliness procedures the possibility of component and system malfunctions.
(7) An automatic checkout system can provide complete real-time test documentation and better control of test operations by test engineers than an analog system.