By George M. Low
Four aspects of spacecraft development stand out.
- Control of changes
- Interpretation of discrepancies.
Build it simple and then double up on many components or systems so that if one fails the other will take over.
Minimize functional interfaces between complex pieces of hardware. The main point is that a single man can fully understand this interface and can cope with all the effects of a change on either side of the interface.
Generally, tedious, repetitive tasks are best performed automatically; and selection of the best data source to use, selection of control modes, and switching between redundant systems are tasks best performed by the pilot.
Apollo Test Activities
The single most important factor leading to the high degree of reliability of the Apollo spacecraft was the tremendous depth and breadth of the test activity.
There are two general categories of tests.
- Those made on a single prototype device (or on a few devices) to demonstrate that the design is proper and will perform properly in all environments and
- those made on each flight item to assure that there are no manufacturing errors and that the item will function as intended. Both categories apply to individual parts, components, subsystems, systems, and entire spacecraft.
The first category includes development testing early in the design cycle and the very formal certification or qualification tests performed on test articles identical to the flight system. The second category covers acceptance testing.
Most important of all, the tests gave us a tremendous amount of time and experience on the spacecraft and their systems. Such experience-together with a detailed analysis of all previous failures, discrepancies, and anomalies-led us to the conclusion that we were ready to fly a lunar orbit with Apollo 8 and that we were ready to make a lunar landing with Apollo 11.
Acceptance testing played an equally important role. This testing starts with piece parts. […] Next, each component, or black box, is tested before it is delivered, and again before it is installed in the spacecraft. Then, factory testing of the complete spacecraft begins. First, the wiring is wrung out, and individual subsystems are tested as installed. Then, groups of systems are jointly tested. Finally, the complete spacecraft, with all of its systems functioning, is run in an integrated test. All normal, emergency, and redundant modes are verified.
After delivery to the launch site, similar (when practical, identical) tests are performed. A major test at Cape Kennedy is a manned altitude-chamber run of each spacecraft. The final acceptance test, of course, is the countdown itself.
A most important facet of acceptance testing is environmental acceptance testing. The primary purpose of acceptance vibration testing and acceptance thermal testing is to find workmanship errors. To do this, the environment has to be severe enough to find the fault (e. g., a cold-solder joint), yet not so severe as to weaken or fatigue the component.
a single qualification test may have missed a marginal condition, which the large number of acceptance tests could catch.
Note that 5 percent of all components failed under vibration, and 10.3 percent of all components did not pass the thermal testing. Remember that these components were otherwise ready for installation in the spacecraft. […] If these tests had not been performed, and if these failures had occurred in flight, we very likely would still be waiting for the first manned lunar landing.
Control of Changes
If the design has been verified and if a thorough test program has been completed, it should not be necessary to make any changes. Of course, this idealized situation does not exist in any program like Apollo where design, test, and flight often overlap and must be carried out at the same time. Changes may be required as a result of test failures, or another look at the design may identify a situation that could lead to a failure or to the inability to react to failure. Sometimes a more detailed definition of flight missions or operational use of the hardware itself leads to a requirement for change.
Since it is not possible to eliminate all changes, we have to start with the premise that any change will be undesirable. That is, a change will void all previous test and flight experience and, no matter how simple, may have ramifications far beyond those identified by the initial engineering analysis.
Because changes must be made nevertheless, it becomes important to understand and to control them, no matter how small. In Apollo, we handled all changes through a series of Configuration Control Panels and a Configuration Control Board. The panels considered minor hardware changes early in the development cycle, as well as crew procedures and all computer programs. The Board considered more significant hardware changes, all hardware changes after spacecraft delivery, and procedures or software changes that could affect schedules or missions.
The Apollo Spacecraft Configuration Control Board met 90 times between June 1967 and July 1969, considered 1697 changes, approved 1341, and rejected 356. We had a low rejection rate because proposed changes were reviewed before they came to the Board, and only those deemed mandatory for flight safety were brought before it. The Board is chaired by the Program Manager, who also makes the final decision on all changes. The Board includes the directors of all major technical elements of the NASA Manned Spacecraft Center and the contractors’ program managers.
We considered changes large and small. An example of a large change is the new spacecraft hatch that was incorporated after the fire. However, we reviewed in equal technical detail a relatively small change, such as a small piece of plastic to go inside the astronaut’s ballpoint pen.
The Board was established to discipline the control of changes; but it was found to serve a much larger purpose: It constituted a decision-making forum for spacecraft developer and user. In reaching our decisions, we had the combined inputs of key people representing hardware development, flight operations, flight crews, safety, medicine, and science.
It is difficult to describe, to those not directly involved in the Apollo Program, just how much work went into operational activities. First, we had to decide the kinds of mission to be flown: What would be the best series of missions to achieve a successful manned lunar landing at the earliest time? Then these missions had to be planned in detail: How should each mission be designed to meet the largest number of operational and hardware objectives, even in the event of unplanned events? (Operational objectives are concerned with guidance, navigation, trajectory control, rendezvous, etc.; hardware objectives are concerned with the verification of each system or subsystem in the flight environment. ) Finally, plans had to be made for the execution of the mission: Detailed rules were evolved for every imaginable contingency; the proper flight-control displays were defined to permit instantaneous reaction to emergencies, and countless hours were spent in simulations of every conceivable situation.
The basic principle in planning these flights was to gain the maximum new experience (toward the goal of a lunar landing) on each flight without stretching either the equipment or the people beyond their ability to absorb the next step.
Too small a step would have involved the risk that is always inherent in manned flight, without any significant gain-without any real progress toward the lunar landing. Too large a step, on the other hand, might have stretched the system beyond the capability and to the point where risks would have become excessive because the new requirements in flight operations were more than people could learn and practice and perfect in available time.
After Apollo 9 another decision had to be made: Were we then ready for a lunar landing, or was the step too big ? We decided that we faced too many remaining unknowns: performance of the lunar module in the deep-space environment, communications with the lunar module at lunar distances, combined operations with two spacecraft around the moon, rendezvous around the moon, and, of course, the lunar descent landing, surface operations, and ascent. In lieu of a landing, we planned to do as many of these tasks as possible on Apollo 10 without actually touching down on the surface of the moon.
The entire series of flights represented a step-by-step buildup, with each step leading closer to a lunar-landing ability. Our intent was to use the procedures developed on one flight on each subsequent mission. Changes were allowed only if they were essential for fright safety or mission success. By means of this buildup, we minimized the remaining tasks (descent, landing, surface operations, and ascent) that could be worked out only on the actual landing mission. The Apollo 11 crew was able to concentrate on these remaining tasks, to work them out in detail, and to carry them out with perfection.
Mission Planning and Execution
Once basic missions had been defined, each flight had to be planned in detail. The mission planner tries to fit into each flight the maximum number of tests of the hardware and the widest variety of operations.
After mission plans come the mission techniques (by another name, data priority). Given two or three data sources (for trajectory control), which of the sources should be believed and which discarded? Limits for each system had to be determined, and logic flows for every conceivable situation had to be developed.
Finally, the flight controllers take over. They had participated, of course, in the mission-planning and mission-technique activities; but now they had to work out each step of the flight and anticipate every emergency situation that might arise. What is the proper action when one fuel cell fails? What if two fail? The answers to thousands of questions like these had to be derived in terms of the specific mission phase. […] Each of these events was documented as a mission rule long before the flight, and mission rules were placed under “configuration control,” as was every other aspect of the Apollo system.
Many of the techniques used during the flight were developed during countless hours of simulations. Simulation is a game of “what-if’s.” What if the computer fails? What if the engine does not ignite? What if . . . ? The game is played over and over again. The flight controllers do not know what situation they will face on the next simulation. By the time of flight, they will have done simulations so often and they will have worked together as a team so long, that they can cope with any situation that arises.
Because the Apollo equipment has worked so well and because there have been so few contingency situations, one could conclude that much of the planning, many of the mission techniques, and much of the training were done in vain. But this is an incorrect conclusion. As a minimum, the state of readiness that evolved from these efforts gave us the courage and the confidence to press on from one mission to the next. Also, there were situations-the computer alarms during the descent of Apollo 11 and the lightning discharge during the launch of Apollo 12-that might have led to an abort if the team had been less well prepared and less ready to cope with the unexpected.
Flight Crew Training
Training for Apollo is not easy. Two highly sophisticated machines are involved, each far more complex than those in Gemini. The astronauts had to become expert in the workings of both spacecraft. They became computer programers and computer operators, space navigators, guidance experts, propulsion engineers, fuel-cell-power managers, environmental-control-system experts-to mention but a few areas of expertise. Of course, they had to learn how to control and fly two spacecraft with vastly different handling qualities under conditions of launch, translunar flight, lunar-orbit flight, lunar landing, lunar launch, rendezvous, docking, transearth flight, and reentry.
The astronauts also needed plans and procedures. Flight plans spelled out each step of the mission. Detailed “time lines” were developed for every function that had to be performed, minute by minute. Crew procedures and checklists were an adjunct to the flight plan. The step-by-step sequence for each spacecraft activity, each maneuver, each propulsive burn was worked out well in advance and was used again and again during practice and simulation.
Configuration control was as important in the astronaut training as in every other category. Simulators had to look just like the spacecraft to be useful, and last-minute spacecraft changes had to be incorporated in the simulators as well. Crew procedures that had worked well on one flight could not be changed, through “crew preference, “ for the following flight.
Attention to detail. Painstaking attention to detail, coupled with a dedication to get the job done well, by all people, at all levels, on every element of Apollo led to the success of what must be one of the greatest engineering achievements of all time - man’s first landing on the moon. The reports which follow amplify this observation.
DESIGN PRINCIPLES STRESSING SIMPLICITY
By Kenneth S. Kleinknecht
- Use established technology.
- Stress hardware reliability.
- Comply with safety standards.
- Minimize inflight maintenance and testing for failure isolation, and instead on assistance from the ground.
- Simplify operations.
- Minimize interfaces.
- Make maximum use of experience gained from previous manned-space programs.