[For a copy of this article in PDF format, which displays figures and equations, click here. Requires Adobe Acrobat Reader, free download.]

Moving to a structured estimating process can substantially improve the accuracy and quality of software cost estimates. The cost-estimating process is crucial to major systems providers because it has a significant impact on the profitability of large programs. In the past, the process of estimating software development costs has been less-structured and has delivered variable results.

In recent years, however, many systems providers have implemented a more formal process in which standardized metric elements are used to provide a quantitative basis for generating program estimates. The accuracy of estimates has increased, and the structured approach helps build increased confidence in the accuracy of the estimating process that simplifies downstream negotiations. The method has been successful on two major projects.

Large software projects present a complex estimating challenge. The methodologies, languages and standards used in the software development process are constantly changing. Human ingenuity has increased the difficulty of the measurement process, sometimes with nearly the entire value being added to the end results. Traditional bottoms-up measurement methods vary widely in accuracy because they depend, to a degree, on the skill and experience of the estimator.

Another problem with unstructured approaches is the lack of documentation created during the estimating process. This makes it difficult to quickly and reliably evaluate the inevitable changes and concerns that occur during the negotiation process.

The spreadsheets historically used to document bottoms-up estimates have built-in limitations. A major concern with spreadsheet models is that they are vulnerable to errors whose likelihood increases as the model becomes more complex. Something as simple as entering the wrong formula in a cell could change the final estimate by millions of dollars. This type of error is difficult to catch because of the difficulties in auditing a one-of-a-kind spreadsheet model. Another problem with spreadsheet models is that they provide a single-point answer that does not normally take range-estimating into account.

To overcome these challenges, most systems providers work with a variety of structured estimating tools. The basic advantage of the structured approach is that it provides a standardized methodology that uses historical data to refine the estimating process. Over time, as metrics used for generating initial estimates are calibrated to actual project results, accuracy increases. The structured approach also generates, as a byproduct of the estimating process, phase distribution of effort and scheduling information that can dramatically improve the project oversight function. The importance of this effort increases as the size and complexity of systems continues to grow and pressure increases to minimize project costs.

Using structured estimating tools, the process begins when the engineering members of the proposal team develop a solution set for meeting the requirements of the request for proposal. At this point, the estimating members of the team begin building the model by entering parameters that organize the costs in an understandable manner. An example of a typical parameter is one that measures the program's level of formality. A manned space flight, for example, requires a formal program that adheres to military standards so that high-level designs, detailed designs and test plans can be presented at standup presentations that require a considerable amount of preparation. The majority of military programs, on the other hand, now use high-grade commercial standards that simply involve technical interchange meetings demanding less preparation.

The program generates an estimate for each phase of the project, organizing the costs in any manner desired by the project team. The transparency of the estimate makes it possible for the project team to easily consider its reasonableness and, if necessary, make adjustments to the program parameters. An advantage of this approach is that it allows the user to enter probability levels for each parameter. This makes it possible to determine the impact of changes in various program parameters. The result is that the estimate includes a most-likely value, as well as a range of probable values based on the uncertainty factors entered by the estimators.

Estimators typically use the model for a rough order of magnitude estimates, firm proposal estimates and management evaluation of software development activities. In most cases, they consistently achieve a high level of accuracy for proposal estimates. The key to achieving these high levels of accuracy is a comprehensive calibration effort. The model is calibrated by evaluating completed programs in which the costs are known. The person who performs the calibration interviews the engineering lead on the program in an effort to understand variables such as the skill level of the developers involved and the characteristics of the project. A model is created to estimate the program after the fact. This is an iterative convergence process consisting of comparisons between the model's parameter selections and resultant cost estimate, and the real-world program characterization and cost.

After calibrating a considerable number of programs, general trends start to emerge. An awareness emerges about the capabilities of an organization and the requirements of its customers as they fit into the continuum measured by the model. The skill level of different groups can be compared within the organization and trends such as an increase in a particular group's performance can be identified as its members gain more experience. This process is critical to a successful structured estimating program. By fine-tuning the process it, is possible to dramatically improve both the accuracy and consistency of estimates over time.

Much of the advantage of using structured estimating tools comes well after the estimating process has concluded. A characteristic of structured estimating is that it delivers not simply a number, but also a methodology that makes it possible to trace exactly how the number was generated. The fact that estimators can easily provide backup information to explain the estimating process increases the customers' confidence. In many cases, even after the bid is accepted, the negotiating process may continue. Furthermore, the model provides an audit trail and makes it possible to evaluate the impact of changing program assumptions.

The model can also provide a framework for evaluating the performance of the project. Comparing actual results with detailed model estimates can provide early visibility of problems and can determine the resultant impact on the overall cost and schedule of the program. In certain cases, early results from a program may cause estimators to question parameters used in creating the model. In such a case, the model can be quickly updated to factor in the effect of changing early assumptions and determining the impact on the overall program budget and schedule. The model can also assist management decision-making by enabling management to compare the expected results of different program management alternatives.

For example, the use of parametric costing on a recent government proposal saved 1,000 hours and helped promote the use of less-expensive design alternatives. In the past, one manufacturer has used the bottom-up and similar-to basis of estimate (BOE) approaches to determine program development costs. On a recent project, in response to the government's cost as independent variable (CAIV) initiative, the firm tried the parametric costing approach in conjunction with the BOE approach. A commercial knowledge base consisting of extensive industry data was used to estimate the development and production cost of a new system. The use of the knowledge base saved a considerable amount of research time and guesswork on the part of the cost engineer and made it possible to evaluate alternative approaches on a much more timely basis. The conclusion of the development team was that the parametric costing approach provided a useful alternative to the bottoms-up and similar-to BOE approach that could offer significant benefits in future programs.

The example OEM recently received an invitation to bid on a government project. A critical part of the proposal was developing an accurate development and production cost estimate. In the traditional process, estimates of this type are produced by first breaking the system down into its individual components, both electrical and mechanical. The electrical hardware components, which typically constitute the bulk of the cost on Harris' projects, are analyzed down to the individual printed circuit board (PCB) level. At this point, cost engineers search through previous programs to find similar boards that were produced in the past. Once something similar is identified, the engineer makes an estimate such as: “The PCB used in the earlier project took 500 hours to design. The one in the current project will actually be a little easier. Let's estimate the cost at 425 hours.” Because this process is used so frequently, many major government contractors have people dedicated to collecting and organizing costing data from previous projects.

On the study under discussion, the customer requested a different approach. The motivation was not actually in saving time during the proposal process but rather in reducing life-cycle costs through the CAIV initiative. This initiative involves a twofold process. The first is a planning activity establishing and adjusting program cost objectives through the use of cost-performance analysis and tradeoffs. The second component involves execution of the program in a way that meets or reduces stated cost objectives.

Early in the acquisition process, the high leverage of CAIV-inspired cost/performance/schedule tradeoffs should shape the requirements and proposed design objectives. Later, the overall cost objectives can be allocated to specific cost and system elements. The bottoms-up and similar-to BOE approach is not conducive to CAIV methods because it takes so long to compare alternate design approaches from a cost standpoint. That's why the customer suggested that a parametric costing approach be used on this project in support of the standard approach.

Engineers examined several types of software packages that use the parametric costing approach. All of these packages provided estimating software and knowledge bases built on extensive real-world data and expertise. The engineers selected programs called SEER-H and SEER-SEM, primarily because they felt that its interface is more intuitive and easier to use. This software package reduces the learning curve by avoiding the use of less-than-obvious adjustments to present the estimating objective more clearly. It also bases the price of individual PCBs on their number of components rather than on their weight, an approach that usually provides more realistic results. The program is sensitive to the difference between mechanical and electrical elements as well as costs between labor and materials. It provides databases that are compiled from thousands of real-world projects to help make accurate estimates. Finally, it provides many tools that allow users to assess risks, make trade-offs, perform sensitivity analysis and create technology forecasts.

The proposal team started by developing the systems architecture, including defining the input, processing steps and output. Throughout the process, the cost engineer worked closely with the project team to develop a good understanding of the system at a high level. This is an essential part of the costing process. It helps the cost engineer comprehend the requirements of the system fully, so they are satisfied in making tradeoffs that may be required from a cost standpoint as the project moves further along. During this period, the cost engineer also began constructing the parametric model by entering high-level systems data into the software package.

As the proposal matured, the electrical systems engineers and software engineers diagrammed the system and the cost engineer began adding more detail. The cost engineer used the software tools to describe each component, identify its quantity, material composition, design and manufacturing processes involved. At this level, the SEER-H model began predicting recurring and nonrecurring development and manufacturing costs by identifying similar components from its database. While SEER-H is primarily designed to model the development process, it does include manufacturing cost estimation capabilities that are usually sufficient for low production programs. If higher quantities will be produced, then it would probably make sense to also develop a design for manufacturing-type models such as SEER-DFM.

As usual, the proposal development process involved a continuing series of changes and tradeoffs. The cost engineer stayed involved with the project and continually updated the model to reflect the latest thinking of the design team. The use of the parametric approach made it possible to provide near-real-time feedback about the team on the effect of choosing various options for the development and manufacturing costs. Using the conventional costing methodology approach, the lengthy period of time required to develop new cost estimates means that a considerable amount of effort may be put into design approaches that will later be ruled out from a cost standpoint.

The cost engineer included higher level costs (costs for the program management office) as a common factor that was automatically allocated to various elements of the project. This saved the need for what would have otherwise been a series of time-consuming calculations. The model also sped the development of productivity curves that estimated the rate at which manufacturing costs could be reduced during the life cycle of the project. Whenever the design was changed, these curves were automatically updated based on assumptions that had been entered previously, eliminating additional manual calculations.

The parametric cost approach saved a considerable amount of time on this project. The database provided by the software is more comprehensive than the existing project library. The software automatically selects appropriate historical projects, eliminating the time previously spent searching through the library. The software also removed the need for the cost engineer to create and maintain a complicated spreadsheet. However, there is still a need to capture, analyze and organize cost history data to calibrate, and if necessary, validate the model. Finally, the program can automatically generate a range of reports, including all of those that were needed on this project. The team also used the parametric results to perform engineering and management reviews of the cost data The result was a substantial time-savings — about 1000 hours of cost engineering support were saved using SEER tools.

At the same time, engineers involved in the project believe that the quality of information provided by the parametric approach is superior to the conventional method. The capability of the program to quickly evaluate the cost of alternate approaches saved considerable time during the proposal process by providing information that guided engineers toward a cost-effective approach sooner than without the process. The ability of the parametric costing approach to allow consideration of cost from the earliest possible stage of the design process is the primary reason why the government is encouraging its use and, based on the experience of this project, it works. Because this was the first use of this parametric model approach, a comparison was made using the conventional estimating approach at the end of the project and the results matched closely (within 7%).

The parametric approach is clearly an effective alternative that should be considered on larger projects with a high magnitude of nonrecurring development and manufacturing expenses.

Gary Constantine parametric estimating manager at Raytheon Imagery & Geospatial Systems Division. William J. Vitaliano is a systems engineer with Harris Corporation in Melbourne, FL. For more information contact Galorath Incorporated, 100 N Sepulveda Blvd, Suite 1801, El Segundo, California 90245. Phone: 310-414-3222 Fax: 310-414-3220 Internet: www.galorath.com.

RSS    Save to Del.icio.us  Digg This