The results of the 101 RCS checkout, he said, "illustrate what we believe to be a lack of adequate workmanship and quality control during the manufacture and checkout of the RCS system. A total of 352 squawks have been written against the S/C 101 SM RCS and quad A has only been partially tested. This high number of discrepancies, most of which cannot be directly related to design deficiencies, is mute testimony to our contention. Test units of the RCS have been built at MSC from scratch with no significant problems either during manufacturing, checkout, or test firing. Thus we have demonstrated that the system can be built successfully even without the specialized equipment and facilities at NAA. Furthermore, NAA has fabricated a number of units with a minimum of discrepancies. . . ."
CSM Manager Kenneth S. Kleinknecht enclosed Taeuber's memorandum and a summary engine failure report written by McDonnell Douglas Corp. after completion of the Gemini program in an October 26 letter to North American Rockwell's Apollo CSM Program Manager Dale D. Myers. Kleinknecht pointed out: "Their conclusion that system contamination was the most likely source of failure in flight, coupled with the fact that the Mercury Program was also plagued with a similar problem, and added to the facts presented in the report by Mr. Ralph Taeuber leads me to believe that positive action must be taken to tighten up the quality control, both at North American Rockwell Corporation and at all subcontractors and vendors that supply the parts for the Apollo RCS. . . . Something must be done to consistently bring the contamination of this system down to an acceptable level. The numerous problems with corrosion and foreign matter are occurring so frequently that it is possible we have other quality or procedural failure modes that are hidden by the constant and over-riding failure modes associated with contamination."
Kleinknecht added that he expected to receive within two weeks a written notice from North American that it was implementing a plan for corrective action and that the plan must include corrective action at the subcontractor and vendor levels.
Myers advised Kleinknecht December 4 that, to determine the cause of the recent valve failures from internal contamination, North American Quality & Reliability Assurance had begun an accelerated investigation October 22. All RCS valve suppliers were investigated, and one supplier was found to have introduced an improper cleaning sequence on an assembled helium-isolation valve, resulting in trapped deionized water in the valve. Valves suspected of moisture contamination were removed from the RCS and, after the supplier corrected the irregularities in his cleaning operation, the valves were returned for rework under North American source inspection surveillance. At the plant of the sub-tier supplier responsible for cleaning the valves that failed on spacecraft 101, a North American source inspector was now required to review the supplier's shop planning and indicate product acceptance by witnessing and verifying newly inserted inspection points on the supplier's in-process paper work.
Myers said that, as pointed out in Kleinknecht's letter, "systems and component contamination were a serious quality and technical problem faced by all major space programs. To rationalize these problems as workmanship and inspection errors introduced the risk of creating misdirected effort that attacks the result instead of the cause.
"The investigation and remedial action taken on the helium valves was a logical and aggressive response to apparent quality problems and is directed toward correcting both the unsatisfactory condition and eliminating the factors that cause the condition to develop. Suspected hardware was immediately removed from the production cycle, inspection surveillance was increased at critical points in the process to insure against continuation of the problem, and a longer range program was implemented to provide extra assurance that similar problems do not exist or develop at other suppliers.
"The process control investigation that revealed the cause of trouble with the helium valve was being expanded to include a re-evaluation of all suppliers involved with cleaning valves, regulators, etc., used in the Apollo CSM. In addition to a fresh look at the suppliers fabrication and cleaning activities, the process evaluation is a comprehensive review of North American and supplier specifications for compatibility between the requirements for one assembly and the next, and a re-survey of the suppliers facilities to assure he has the technical capability and equipment to meet the stringent Apollo CSM quality requirements. The plan of action for this process study is being developed, and action to the plan will commence within a week."