ES, IG-99-052, X-33 Cost Estimating Processes

X-33 COST ESTIMATING PROCESSES
IG-99-052

Executive Summary

Background
In July 1996, NASA selected Lockheed Martin Skunk Works (Lockheed) for Phase II of the X-33 Program, to design, build, and fly the X-33 unmanned test vehicle.⁽¹⁾ NASA is using a cooperative agreement⁽²⁾ for the X-33 Program, a first for a major technology program. Under the terms of the cooperative agreement for Phase II of the X-33 Program, NASA will provide $912.4 million (about 80 percent) and Lockheed will invest at least $212 million (about 20 percent). Lockheed is responsible for any costs that exceed or overrun its estimate.
Cost analyses and estimating activities are critical elements of any procurement process including a cooperative agreement. Program managers develop life-cycle cost estimates at the start of a program as guidance for needed resources and throughout the life of the program especially when a program undergoes major restructuring, since changes may significantly alter the cost of the program. If early life-cycle estimates are realistic, greater program stability results. The contracting officer and the source evaluation committee (SEC)⁽³⁾ use Independent Government Cost Estimates (IGCE's)⁽⁴⁾ to evaluate the reasonableness of contractors' proposed costs. Additionally, an independent cost estimate⁽⁵⁾ is required during the Non-Advocate Review,⁽⁶⁾ where the decision is made whether to proceed to the next phase of the program.

Objective
The audit objective was to determine whether the Phase II X-33 cost estimating process relied on sound methodologies and procedures that produced realistic cost estimates. Appendix A contains details on the objectives, scope, and methodology used for this audit.

Results of Audit
NASA did not adequately address cost reasonableness and cost risk for Phase II of the X-33 Program. Specifically, NASA did not prepare an IGCE for the source evaluations of proposed costs. Further, the Non-Advocate Review's cost estimate did not include a risk analysis to quantify technical and schedule uncertainties. Therefore, NASA management approved the program without the benefit of realistic estimates of the probable cost of the X-33 Program. A risk analysis would have alerted NASA decisionmakers to the probability of cost overruns in the program. Cost overruns put NASA's investment in the X-33 Program at risk. Since the program is under a cooperative agreement, the recipient may end its part of the partnership should cost overruns become too burdensome or request that NASA invest more money. In addition, Lockheed's current estimate at completion is considered overly optimistic. Due to unforeseen challenges in the development of technology and resulting schedule delays, Lockheed may need to contribute from $120 to $300 million more than planned to complete the work contemplated in the cooperative agreement. NASA places a high priority on the success of the program, and the cost growth being experienced places program success at risk.

Recommendations
NASA should improve its evaluation processes for cost reasonableness and cost risk to ensure that decisionmakers are provided complete and accurate information, that sufficient resources are available, and that the final "agreed to" price is fair and reasonable. Further, the X-33 Program's estimate to complete should be updated to reflect cost uncertainties and determinations made of how remaining work will be funded.

Management's Response
Management concurred with the recommendations and stated it was taking action to correct the reported weaknesses. Management plans to issue a regulatory change that will require a well-supported cost analysis for commercial cooperative agreements and will require the recipient to provide an estimate at completion that includes a breakdown of the industry contributions to complete the X-33 program. Management will provide the recommended regulatory change that well-supported independent cost estimates include a quantification of cost risk to the NASA Program/Project Management Working Group for consideration. The group is responsible for review of Agency policy guidelines.

Evaluation of Management's Response
Management's planned actions are responsive to all the recommendations.

FOOTNOTES

1. The X-33 Program consists of three phases. Phase I, which has been completed, consisted of concept definition and prelimary design of the X-33 vehicle. Phase II, the focus of the audit, consists of final design and demonstration of the technologies that would reduce risks. After evaluating the results of Phase II, Lockheed will make a decision on whether to proceed to Phase III, which is construction of a full-scale operational vehicle.

2. The cooperative agreement for Phase II of the X-33 Program is a legal instrument reflecting a voluntary partnership between NASA and a commercial firm, Lockheed, where substantial involvement is expected between both parties in carrying out the activity contemplated in the cooperative agreement. If at any time one of the partners decides it is not in its best interest to continue, it can terminate the partnership.

3. The NASA SEC is an evaluation team that assists the source selection official by providing expert analyses of offerors' proposals.

4. An IGCE is a Government-prepared estimate of the probable cost of a proposed procurement.

5. An independent cost estimate is any cost estimate developed by specialists outside and independent of the program office. The independent cost estimate serves as an analytical tool to validate or cross-check program office or contractor-developed estimates.

6. A team composed of NASA management, technical, and budget personnel who will not participate in the implementation of the proposed program or project performs this review.

Report in PDF

Background	In July 1996, NASA selected Lockheed Martin Skunk Works (Lockheed) for Phase II of the X-33 Program, to design, build, and fly the X-33 unmanned test vehicle.⁽¹⁾ NASA is using a cooperative agreement⁽²⁾ for the X-33 Program, a first for a major technology program. Under the terms of the cooperative agreement for Phase II of the X-33 Program, NASA will provide $912.4 million (about 80 percent) and Lockheed will invest at least $212 million (about 20 percent). Lockheed is responsible for any costs that exceed or overrun its estimate. Cost analyses and estimating activities are critical elements of any procurement process including a cooperative agreement. Program managers develop life-cycle cost estimates at the start of a program as guidance for needed resources and throughout the life of the program especially when a program undergoes major restructuring, since changes may significantly alter the cost of the program. If early life-cycle estimates are realistic, greater program stability results. The contracting officer and the source evaluation committee (SEC)⁽³⁾ use Independent Government Cost Estimates (IGCE's)⁽⁴⁾ to evaluate the reasonableness of contractors' proposed costs. Additionally, an independent cost estimate⁽⁵⁾ is required during the Non-Advocate Review,⁽⁶⁾ where the decision is made whether to proceed to the next phase of the program.
Objective	The audit objective was to determine whether the Phase II X-33 cost estimating process relied on sound methodologies and procedures that produced realistic cost estimates. Appendix A contains details on the objectives, scope, and methodology used for this audit.
Results of Audit	NASA did not adequately address cost reasonableness and cost risk for Phase II of the X-33 Program. Specifically, NASA did not prepare an IGCE for the source evaluations of proposed costs. Further, the Non-Advocate Review's cost estimate did not include a risk analysis to quantify technical and schedule uncertainties. Therefore, NASA management approved the program without the benefit of realistic estimates of the probable cost of the X-33 Program. A risk analysis would have alerted NASA decisionmakers to the probability of cost overruns in the program. Cost overruns put NASA's investment in the X-33 Program at risk. Since the program is under a cooperative agreement, the recipient may end its part of the partnership should cost overruns become too burdensome or request that NASA invest more money. In addition, Lockheed's current estimate at completion is considered overly optimistic. Due to unforeseen challenges in the development of technology and resulting schedule delays, Lockheed may need to contribute from $120 to $300 million more than planned to complete the work contemplated in the cooperative agreement. NASA places a high priority on the success of the program, and the cost growth being experienced places program success at risk.
Recommendations	NASA should improve its evaluation processes for cost reasonableness and cost risk to ensure that decisionmakers are provided complete and accurate information, that sufficient resources are available, and that the final "agreed to" price is fair and reasonable. Further, the X-33 Program's estimate to complete should be updated to reflect cost uncertainties and determinations made of how remaining work will be funded.
Management's Response	Management concurred with the recommendations and stated it was taking action to correct the reported weaknesses. Management plans to issue a regulatory change that will require a well-supported cost analysis for commercial cooperative agreements and will require the recipient to provide an estimate at completion that includes a breakdown of the industry contributions to complete the X-33 program. Management will provide the recommended regulatory change that well-supported independent cost estimates include a quantification of cost risk to the NASA Program/Project Management Working Group for consideration. The group is responsible for review of Agency policy guidelines.
Evaluation of Management's Response	Management's planned actions are responsive to all the recommendations.