AMS Special Topics

Writing Decision Papers

Designing a system, analyzing a client's operations, planning a computer center upgradedecisions, decisions! Software developers, consultants, and managers at AMS often need to evaluate alternative courses of action and make a recommendation. It has been a time-honored tradition at AMS to write a paper when confronted with an issue that doesn't have an obvious solution. Once called an issue paper, now more commonly referred to as a decision paper, it serves to clarify the issue, document the analysis, and provide a sound basis for making a decision.

This is a paper on how to write decision papers. It originated with a study that the author, Milt Hess, performed in 1981 for a large AMS development project. Tasked to evaluate alternative database management systems for the project, he formulated the approach that this paper describes. Pleased with the results, he wrote the initial version of this paper and circulated it to a few people, who found it useful. The original version has been revised several times to clarify the concepts and to provide specifics on how to execute the analysis steps, and these earlier versions have been distributed and used widely within AMS.

Writing Decision Papers

Perhaps your client has asked for a strategy to cut the cost of running the inventory system. Or, your project manager has asked for a recommendation on a software package for the new financial system. How will you organize the study, identify and analyze the options, and present your recommendations? You know that you will have to write a paper of some sort. Such papers are variously called issue papers, working papers, or decision papers. I prefer the term decision paper because it is more results oriented. Based on my experience reading and writing decision papers, I have found a strategy that consistently produces good results. The strategy is based on a few key ideas: Identify a rich set of options by exploring the design space, Evaluate all options against the same criteria, Identify the discriminatorsthe criteria that most sharply differentiate the options, and Reduce the analysis to a clear choice among the options based on the discriminators. Following a discussion of the defects that decision papers often contain, I describe the recommended strategy for performing the analysis and writing the report. reader with the uncomfortable feeling that some important issue has either been omitted or given short shrift. His conclusions sound like a comparison of apples and oranges, and they are usually expressed in vague, unquantified terms. Another common approach, also unsatisfactory, goes to the opposite extreme. The analyst identifies a large number of criteria and assigns a numerical weight to each to signify its importance in the analysis. He then evaluates the options and assigns a numerical rating for how well each option meets each criterion. He then multiplies the ratings and the weights, and sums them to derive a score for each option. The best score wins. Although rigorous, this is in fact an arbitrary approach that replaces analysis by arithmetic. It is based on two assumptions that are rarely true and, in any case, impossible to validate: 1) That the weights accurately reflect the relative importance of the criteria, 2) That the weighted sum of the ratings is a meaningful measure of the option. These strategies are subject to yet another deficiency. They fail to address the sensitivity of the results to the assumptions. For example, what happens in a cost/benefit analysis if the estimated rate of inflation is wrong? The best choice may be the option that minimizes the downside exposure, not the one that offers the most benefits if everything works perfectly.

The Usual Approaches

Sometimes, issues are analyzed and documented using a straightforward three-step approach: 1. Define the options, 2. Describe the strengths and weaknesses of each option, and 3. Draw conclusions and present a recommendation. This approach has a serious defect which derives from the failure to explicitly define the evaluation criteria. When the analyst focuses only on the highlights for each option, he often leaves the

The Goal of a Decision Paper

The goal of a decision paper is not necessarily a single answer. After all, if you are asked to write a decision paper you are probably not the person who has to make the decision. The goal of your paper is to provide the decision maker with a sound basis for the decision. The paper is successful if the decision maker understands the trade-offs among the most attractive optionswhat is gained and what is lost when the decision is implemented. Consider an everyday example, the so-called cost/benefit analysis. A cost/benefit analysis is simply a decision paper where one criterion is cost and the others are the potential benefits. Who is to say whether the benefits of an expensive option are

worth the cost? The decision maker, that's who. As the analyst, your job is to enable him or her to make the decision with confidence: That there is not yet another option with an even better cost/benefit profile, That all the costs and key benefits are represented accurately. If you have done this, you have done your job well. The same is true for any decision paper. Your goal is to identify all the best options and the applicable criteria, and to accurately assess the options. If one of the options is obviously better than the others, that makes the decision maker's job easier. Your mettle as an analyst will only be proven, however, when you have to do a decision paper where the options present a difficult choice. If you can define the issue clearly enough to support an informed decision, you will have met the test.

It is tempting to perform this step mentally, without writing the information down. Don't. The reason for this step is to make sure you have the issue defined correctly. If it is only in your head, you can't be sure.

Identify the Options

Prepare a list of the possible solutions to the problem. Do not evaluate them yet or even describe them in detail; that comes later. Generating optionspotential solutions to the problemis the most creative and important activity in the analysis. To be successful in resolving an issue, you have to include the "right" answers among your options. If you overlook them, you will offer only inferior options to the decision maker. Brainstorming is a popular technique for generating options, but it is unreliable. How can you be certain that the "right" answers will surface in a brainstorming session? You can't. A more reliable method is needed. Imagine a "design space" of several dimensions. For example, suppose that you have to evaluate design strategies for a new inventory system. The design space has at least three dimensions: How much of the existing system to retain - All - Some - None Degree of centralization vs. distribution - Full}- centralized - Centralized database with remote workstations - Fully decentralized Hardware vendors - IBM - DEC - Tandem

Doing the Analysis

Before you can write the paper, you have to do the analysis. This is a seven step process: Define the issue, Identify the options, Identify the evaluation criteria, Describe each option, Compare and score the options, Identify the discriminators, Summarize the trade-offs among the options.

Only rarely will you perform these steps in sequence. Unless you are intimately familiar with the issue at hand, you will have to revisit some of these steps as you proceed with the analysis. For example, the process of describing and comparing the options often discloses additional criteria that should be considered. Nevertheless, it is important to perform each of these steps consciously. They should not be skipped or blurred together.

Define the Issue

Document your understanding of the problem as precisely as you can. Describe the background of the issuehow it arose, who will be affected by the decision, who will make the decision, any constraints on the solution, and anything else of importance.

Options can be generated by selecting one choice from each dimension, for example: Fully centralized IBM system retaining some of the existing system, Fully decentralized Tandem system retaining none of the existing system.

This systematic method of generating options is called "exploring the design space." First, identify the dimensions of the design space; then select "points" along each dimension; finally, define all the feasible combinations. The richness of the options derived in this way may surprise you. Still more options can be generated by forming hybridscreative combinations of other options. For example, instead of choosing between buying a Porsche or buying a limousine, one could buy a Porsche and rent a limousine when necessary. After you have identified all the feasible options, the next step is usually to eliminate some. To keep the analysis manageable, you should evaluate no more than three or four options in detail. There are two reasons for eliminating an option from further consideration: It fails to satisfy a constraint. It is clearly inferior to the other options. A constraint is a filter for eliminating unacceptable options, not merely a very important criterion. It is useless to consider further any option that has a fatal defect. If, for example, an accounting model fails to satisfy your auditor's requirements, or a software package does not run on your hardware reject it immediately. The rest of the analysis will be shorter and therefore easier to understand. But do ensure that the requirement is really a requirement. Sometimes an option is eliminated because someone thinks the decision maker won't accept it. Don't constrain the options unnecessarily. Although they satisfy the constraints, some options may be clearly inferior to others for a variety of reasonstechnology, cost, risk, etc. Eliminate them as well. Document briefly the options that you have eliminated, in case any of your assumptions change later.

you to perform a completeness proof on the criteria. It is difficult at best to demonstrate that an unclustered set of dozens of individual criteria defines the goal of the decision. Include as criteria the factors that will influence the decision. If a factor will not influence the decision, leave it out of the analysis. For example, although "number of current users" is often a factor in a software package evaluation, who really makes such a decision based on this criterion? Vendor credibility and stability are real concerns, however, and would be valid criteria. There is a straightforward technique for identifying the "right" criteriathe ones that will influence the decision. Considering each option in turn, define what is good and what is bad about it. If this proves to be difficult, consider the options in pairs why A is better than B, and why B is better than A. Consolidate the resulting list of candidate criteria by eliminating duplicates. The final step is to ensure that the criteria are independent and at an appropriate level of abstraction. Criteria are independent if they describe completely different characteristics of the options. Review the list of candidate criteria and combine any that are not independent into a more general criterion. For example, if an analysis of motor vehicles generated "number of seats" and "passenger capacity" as criteria, you would have to combine them into a single criterion; they are not independent. As you combine criteria in this way, you formulate more abstract criteria. Be careful that the final list comprises criteria that are at a comparable, useful level of abstraction. For example, "number of seats" and "luggage capacity" are at the right level of abstraction; each addresses a major issue in buying a car, and they are independent. The next higher level of abstraction, perhaps "capacity," is too broad to be useful. Do not try to assign weights to the criteria, even subjectively. Just write them down and define them as precisely as you can, then proceed to the next step.

Identify the Criteria

Define clear and unambiguous criteria for evaluating the options. To keep the rest of the analysis manageable, include no more than five to seven major criteria. If there are many more than this, as is sometimes the case in a technical evaluation, group the criteria into categories. For example, in a DBMS evaluation there may be a dozen or more specific questions related to restart/recovery and concurrent update; group them into a category called "Data Integrity." Besides simplifying the analysis and the presentation, this approach helps

Describe Each Option

Having identified a set of viable options and significant criteria, you are now ready to evaluate how well each option satisfies each criterion. If you think of the analysis in terms of a matrixoptions versus criteriathis step addresses each column separately (see Figure 1A on next page).

Options

Options

Criteria

Criteria

Figure 1A Describe Each Option

Your purpose in this step is to describe each option as objectively as possible in terms of the criteria. Write down all of the important facts for each "cell" in the matrix. Use numerical values where possible: for example, cost ($), speed (MIPS), service life (years). Most evaluations, unfortunately, result in subjective ratings such as "excellent," "good," "fair," and "poor." There would be no problem if everyone agreed on the meanings of such ratings, but that is rarely the case. To solve this dilemma, you need to define the ratings as explicitly as possible. Once again, there is a strategy that will lead you to your goal in an orderly way. For each criterion that generates subjective ratings, define a set of descriptors that span the range from best to worst. For example, in evaluating the features of a software package, the following set of descriptors might be appropriate: Best Includes all required features, plus additional useful features Includes all required features, but little more Includes most required features, with minor exceptions Excludes some important required features

Figure IB Compare and Score the Options

For an automobile evaluation, seating comfort might be evaluated with the following descriptors: Best Good support, good cushioning, good position relative to controlsno discomfort after 1 hour Good support, cushioning, position, but some discomfort after 1/2 hour Some discomfort apparent after 5-10 minutes Uncomfortable immediately

t
Worst

To simplify the subsequent comparison of the options and presentation of the results, limit the number of descriptors to four. These examples demonstrate two important points. First, it is important to make the descriptors mutually exclusive. In the software package example, what happens if a package has most of the required features and some additional features? My descriptors don't address this possibility explicitly. One approach is to refine the descriptors to include this case. Another is to recognize that we really have two criteria hererequired features and additional featuresand to separate them. Each situation is unique, so you have to use the results of the analysis to tailor the criteria and the descriptors to the situation.

\
Worst

Second, you can sometimes transform a set of qualitative descriptors into a numerical value. After I created the seating comfort descriptors for this example, I realized that they could be discarded in favor of the objective criterion "elapsed time until noticeable discomfort." This is a striking example of how an ambiguous criterion can be transformed into one that can be measured and that everyone will understand. Do not hesitate to iterate the previous steps. If you discover that you have included a criterion that doesn't matter, or omitted one that does, go back and revise the list of criteria. When you have finished gathering and documenting this information, you are ready to do the comparisons.

perfectly satisfactory options so that the worst appears to be awful. The discussion of discriminators in the next section will help you to see when to emphasize the differences and when not to. A graphic format often is the best way to present the results of the analysis. Figure 2 on the next page, an example from the system concept for a municipal financial system, shows how incisive this format can be.

Identify the Discriminators

Recall that you were advised not to assign weights to the criteria. Now you will learn why. Perhaps surprisingly, the importance of each criterion depends on the results of the analysis. If a criterion does not discriminate the options, it is not important to the decision even though it may be critical to the success of the project. For example, although data integrity is crucial in most DBMS applications, it will not affect the selection decision if all the options satisfy all the data integrity objectives. The decision will be based on the discriminatorsthe criteria that most sharply differentiate the options. In this phase of the analysis your task is to identify the discriminators and to review the descriptions of the options to make certain that each discriminator is discussed fully. Start by eliminating the criteria that have similar, acceptable scores for all options (e.g., criterion 6, "Flexibility to Change" in Figure 2). Next, eliminate any options that are clearly inferior. In Figure 2, option 3 dominates option 1 for every criterion except 4 and 5, where option 3 is at least good; option 1 is clearly inferior to option 3. Finally, identify the criteria that differentiate the remaining options. In Figure 2, these are primarily criteria 4 and 5; the differences among the others are interesting but won't drive the decision. If there are truly no discriminators, don't fabricate one. "I don't care" is a valid conclusion for the analysis.

Compare and Score the Options

Your goal in this step is to identify the relative strengths and weaknesses of the options for each of the criteria. In terms of the matrix, this step addresses each row separately (see Figure IB). If you were able to quantify the results for any of the criteria, use the values directly. Otherwise, use the descriptors as the basis for the evaluation. If you have done a thorough job of describing the options, it will be easy to compare and score them. If not, you may have to do more research to gather missing data. First, define the scale that you will use to present the results. A simple scale with a few clearly defined gradations is best (e.g., 0-3). In an effort to add precision, analysts sometimes make the mistake of using too fine a rating scale. Using a scale of 0-10, for example, implies that there is a meaningful difference between a 6 and a 7. This is why you should try to limit the number of descriptors to four. If you need more than this to describe the options accurately, then you should cluster them into four categories for comparison and presentation. Likewise, if you have a criterion with numerical scores (e.g., cost ($)), define four ranges for comparison and presentation. If the differences among the options are small, define the ranges and the descriptors so that your evaluation will highlight the differences. Do not, however, exaggerate minor differences among

Summarize the Trade-offs Among the Options

Having identified the discriminators, and having clear distinctions among the options in terms of the discriminators, you are now ready to frame the issue for the decision maker. Mavbe the differ-

Criteria

Option 1 (Decentralized)

Option 2 (Central)

Option 3 (Mixed)

1. Auditability 2. Control 3. Reporting 4. Management Incentive 5. Agency Needs 6. Flexibility to Change 7. Training 8. Cost

Figure 2

ences are technically complex, or the benefits of the more costly approach are subtle. You have to summarize the trade-offs in terms that the decision maker will understand. Here are a few of the things that you do not do. You do not add the scores to find the winner. You do not automatically recommend the option with the most high scores (because just one poor score may be fatal). Instead, imagine yourself preparing for a briefing where you have five minutes to explain the issue to the decision maker. Write your briefing notes. If you are analyzing the financial system options shown in Figure 2, point out the fatal flaws in options 1 and 2. Focus on the criteria that highlight those flaws. If you are performing a classic cost/benefit analysis, focus on the benefits that might justify the higher cost option. Finally, if your analysis is based on assumptions about future events or other uncertainties (e.g., the cost savings to be realized by a course of action), perform a sensitivity analysis. What is the effect if your assumptions are wrong by a little (or a lot)? What is the crossover point, when an option is no longer attractive? How likely is the contingency to occur?

Writing the Paper

Presumably you have documented the analysis as you performed it. To write the decision paper, you now have to organize the material in the best format for its intended audience. Here is a good general purpose outline for a decision paper: Management Summary Background Options Evaluation Criteria Evaluation of the Options Comparison of the Options Summary and Recommendations

This outline mirrors the analysis steps, with two exceptions. It adds a Management Summary to provide an introductory overview of the entire analysis, and it lumps the last two analysis steps into the Summary and Recommendations. The Management Summary explains the issue and the key features of the analysis. It presents the advantages and disadvantages of the optionsthe discriminatorsand succinctly defines the basis for the decision. If a table such as Figure 2 has been prepared, it should be included in the Management Summary. Remember that you are not writing a novel; the reader wants to know the ending up front.

The Background section documents the information assembled in the first step of the analysis, namely, define the issue. If there are any technical, political, or economic constraints on the resolution of the issue, identify them here. If the issue has been considered previously, include a brief description of its history and the reasons for reopening it. In the Options section, identify the options you analyzed and explain how you selected them. If any other options were dropped from consideration, describe them briefly along with the reasons for eliminating them. In the Evaluation Criteria section, describe the criteria and why each is important. Include a discussion of what is good, what is bad, and why. Present the descriptorsthe definitions of the ratingsthat you used for the subjective criteria. One of the reasons for defining the criteria explicitly is to enable others to review the analysis. Without the rationale for each criterion, the reader is left to guess why the author included it in the study. If a large number of features were grouped into categories, present the details in an appendix; don't clutter the body of the paper with them. In the Evaluation of the Options section, describe how well each option satisfies the criteria. Group the presentation by option. If there is a lot of detail, put the detail in an appendix and summarize it in the text. In presenting the Comparison of the Options, group the presentation by criterion. Highlight the key differences among the options for each criterion, using graphics wherever possible. The Summary and Recommendations section consolidates the information developed in the last two analysis steps, namely, identify the discriminators and summarize the tradeoffs. If you have based the analysis on any assumptions, you should present the results in a way that helps the decision maker judge the validity of the assumptions.

The typical approach is to state the assumption, derive and present the results, and then evaluate the sensitivity of the results to the assumption. For example, if a buy/lease analysis assumes a cost of money of 10 percent, you would present the results and then show what happens if the cost is lower or higher than 10 percent. Although straightforward, this approach has a defect. The decision maker may immediately question the assumption and then be skeptical of the rest of the analysis. An alternative is to use a goal-seeking presentation format. State the desired result, then derive the value of the assumed variable that produces this result. For the buy/lease analysis, for example, you would derive the value of the cost of money that generates a break-even resultsay 18 percent. Then you can tell the decision maker "if the cost of money will be under 18 percent, it's better to buy; over 18 percent, it's better to lease." In effect, this allows the decision maker to make the assumptions, and it does it in a way that eliminates the skepticism of the typical approach. If appropriate, conclude the paper with a recommended course of action.

Conclusion
The approach described in this paper encourages a rigorous thought process in analyzing issues. Adherence to the approach prevents armwaving and other forms of muddy analysis, presumably improving the effectiveness of the analyst. In a rational world, it will also lead to better decisions.

About the Author

Milton S. Hess
Milton Hess is an AMS Vice President. Since joining AMS in 1977, he has served as a technical specialist, manager, and senior reviewer on large information systems projects for both commercial and government clients. Prior to joining AMS, Milt spent 10 years at Bell Laboratories as a research engineer and computer systems manager. Throughout his career, Milt has been a problem solver. He has sought strategies that would help him, and others, find good solutions to complex problems in an orderly, repeatable fashion. He has published and presented papers on strategies for developing large information systems, he was a key contributor to LPS Methodology, AMS's system development methodology, and he has developed and presented a number of seminars within AMS on system design and analysis techniques. Among his project roles, Milt has been the lead architect on system design projects for the Army Materiel Command, Army Corps of Engineers, and U.S. State Department. He has prepared software development and testing methodologies for GTE and the Army Information Systems Command, and a database design methodology for Standard Oil Company of Indiana. Milt is currently Technical Director for information Technology for AMS's defense and aerospace business, where he focuses on the application of proven engineering strategies to system design and development.

CORPORATE HEADQUARTERS American Management Systems, Inc. 1777 North Kent Street Arlington, Virginia 22209 (703) 841-6000 (703) 841-5584 FAX

EUROPEAN HEADQUARTERS AMS Management Systems Deutschland GmbH Querstrasse 8-10 6000 Frankfurt 1 GERMANY 49-69-2455110 49-69-24 55 11 99 FAX

NORTH AMERICAN SUBSIDIARIES Data Base Management, Inc. AMS Courseware Developers 300 Chapel Road Manchester, Connecticut 06040 (203) 646-3264 (800) 232-3264 (203) 643-9629 FAX AMS Management Systems Canada Inc. 180 Elgin Street Ottawa, Ontario CANADA K2P 2K3 (613) 232-7400 (613) 232-0324 FAX AMS Operations Corporation 66 South Van Gordon Street Lakewood, Colorado 80228 (303) 989-7065 (303) 232-4618 FAX AMS Technical Systems, Inc. 1525 Wilson Boulevard Arlington, Virginia 22209 (703) 841-2000 (703) 276-3295 FAX EUROPEAN SUBSIDIARIES AMS Management Systems Deutschland GmbH Querstrasse 8-10 6000 Frankfurt 1 GERMANY 49-69-2455110 49-69-24 55 11 99 FAX AMS Management Systems Europe, S.A./N.V. Avenue Louise 65 1050 Brussels BELGIUM 32-2-535-7831 32-2-535-7731 FAX AMS Management Systems U.K. Ltd. Garden House 18 Finsbury Circus London EC2M 7BP ENGLAND 44-71-638-3232 44-71-382-9776 FAX
1992 American Management Systems, Inc. Printed in the USA. All rights reserved.

NORTH AMERICAN OFFICES Alexandria, VA Arlington, VA Boston, MA Bremerton, WA Chesapeake, VA Chicago, IL Dallas, TX Detroit, MI Emeryville, CA Houston, TX Lakewood, CO Los Angeles, CA Manchester, CT Montreal, Canada New York, NY Ottawa, Canada Port Hueneme, CA Portsmouth, NH Redwood City, CA Richmond, VA Roseland, NJ San Diego, CA San Francisco, CA Sarasota, FL Toronto, Canada EUROPEAN OFFICES Brussels, Belgium Dusseldorf, Germany Frankfurt, Germany London, England