Air University Review, May-June 1967
Too much has been written by too many on how to do analysis. But too little has actually been accomplished by too few. At the risk of being placed in the first category, I will offer some remarks in the hope of enhancing the state of understanding as to how to go about achieving good analysis.
Simply said, the purpose of an analysis is to provide illumination and visibility—to expose some problem in terms that are as simple as possible. This exposé is used as one of a number of inputs by some “decision-maker.” Contrary to popular practice, the primary output of an analysis is not conclusions and recommendations. Most studies by analysts do have conclusions and recommendations even though they should not, since invariably whether or not some particular course of action should be followed depends on factors quite beyond those that have been quantified by the analyst. A “summary” is fine and allowable, but “conclusions” and “recommendations” by the analyst are, for the most part, neither appropriate nor useful. Drawing conclusions and making recommendations (regarding these types of decisions) are the responsibility of the decision-maker and should not be pre-empted by the analyst.
Under the heading of “summary” one can write quite perceptively, stating that, within the factors we have been able to quantify, if such and such is true, then this is the outcome. But, most important, one is not required to go beyond those factors that have been “analyzed” and make a recommendation which surely is based in part on factors that have not. Of course there are the non useful recommendations. A common one of this type is something like “The subject requires further study.” Not only are such statements of little import, but such a conclusion is usually quite obvious without being stated.
So, to repeat, the job of the analyst is to provide illumination and visibility—to expose the problem. This is so obvious that it hardly seems necessary to make the point. Still there is ample evidence to show that many analysts were surely not focused in that direction when they went about their work. In fact I have reason to believe that most card-carrying analysts would look upon the idea of omitting conclusions and recommendations as sheer heresy.
How can one go about exposing a problem? He thinks in the following terms: I am going to make a simple “analogue computer” of this problem. Now, this analogue computer is not an electronic marvel; it is nothing more than a curve (or curves) on graph paper. One can easily handle four variables: the dependent variable on the ordinate (usually the measure of merit) and three independent variables—one on the abscissa, another on a family of curves on each graph, and still another by having a family of graphs (actually one can handle more independent variables, since variables can be combined or aggregated—a “Reynolds number,” for example).
So the idea is to construct an analogue computer. It is a computer since it tells the outcome (on the ordinate) for given values of the independent variables. Such a computer allows one to look at trends as shown by the slope and placement of lines. One should as a matter of course make all possible cross plots. That is, now make the variable that was once on the family of curves the running variable on the abscissa, and so on. Not all the cross plots will be useful, of course. But plot all possible combinations, and use those that provide the greatest visibility. The analogue computer should be exercised. It will tell what input factors drive whatever is being measured and how sensitive the answers are to these factors. Digital computers are quite useful in calculating points in constructing the analogue computer. But the print-outs of digital computers are not terribly useful for presentation. Our analogue computer is the way to present the results so as to expose the problem. The digital computer should be regarded as simply a means of relieving oneself of the drudgery of calculation.
This brings me to the next point. Many people who like to call themselves “analysts” are really “calculators.” They spend more time having calculations made on a digital computer than they spend in analyzing the results. They are “expanders” rather than “distillers” and can be identified easily by the pride which they exude when they present some “decisionmaker” with a “five-foot” study and announce how many hours it took to generate all this material on some high-speed computer.
If an analyst is asked what is the effect of halving the circular probable error of a missile in attacking hard targets, he will derive simple statements such as: “If the CEP is halved, it takes only one-fourth as many missiles to have a certain assurance (damage expectancy) of killing a certain number of targets.” Further, he will add that the ratio of four to one is independent of the hardness of the targets being attacked, the absolute value of the CEP, the assurance desired, and the number of targets. The “calculator” will do a number of war games and, if he is persistent, may discover that the ratio of missiles is about 3.948 for some particular set of circumstances. But rarely will calculations (particularly single valued war games as such) expose the universal truths. If at all possible, reduce (collapse) the problem to a simple formula or formulas and then show the solutions to these formulas by graphs or tables. Actually these graphs are the analogue computers we talked of earlier. The idea is the same, but I have described it another way for emphasis, in the forlorn hope that more and more people will believe in this approach and actually try it.
There are many examples where problems have been “collapsed” in an elegant and simple way. They will not be described in detail here. But the final results of two examples will be alluded to in order to whet the appetite of the curious and to demonstrate that complex problems can be made simple if “analysts” think about them for weeks rather than have “calculators” quickly call on a “programmer” to turn the problem over to an unimaginative electronic marvel.
Example 1: There is an optimum-sized missile for the
Ps = e exp — (1-x / x)*
where x is the exponent in the formula that relates the cost of missile and its size as measured in payload. Collapsing the problem for finding the optimum-sized missile starts with the formula
C = (K) (Wx)
where C is the ten-year system cost in millions of dollars, K is some constant (about $10 M), W is the payload of the missile in thousands of pounds, and x is an empirically derived exponent with a value of around .5. This means that, in the face of an enemy attack, the P8 of the optimum-sized missile will be about .368
(e exp — (1-.5 /.5) = .368). Note that the Ps
of the optimum-sized missile is independent of the amount of payload we desire and the size and effectiveness of the Soviet attack, just so long as the restraint given in the footnote is met. No “calculator,” particularly one armed with a digital computer, would ever stumble on this fact. You don’t understand and believe it? Well, you never will if you insist on engaging in a profusion of calculations as a substitute for some deep thinking and elementary mathematics.
The problem can only be this simple if one accepts that the effectiveness of an intercontinental ballistic missile can be measured adequately by the number of pounds of throw-weight (payload) it can deliver. That in itself was something of a breakthrough. This seems simple and straightforward now, but it was very strong medicine for the “calculators” to swallow in 1958 that five surviving missiles with a 1000-pound payload each were equal in effectiveness to one surviving missile with a 5000-pound payload. The reaction invariably was, “It couldn’t possibly be that simple”—but it was.
Example 2: One can do elaborate war games in air defense. It turns out that the results of these war games can be approximated closely by the formula
Ps = e exp ( — p I / B)
where P8 is the probability that a bomber survives the area bomber defenses and reaches the target, I is the number of interceptors in the game, B is the number of bombers, and p is an empirical constant that depends on the geography wherein the encounters take place, on the radar cross section of the bomber, the effectiveness of the interceptor radar, and the relative speeds of the interceptor and bomber. Whether or not the formula P8 = e exp (- p I/B) adequately represents the actual world of continental air defense is a moot question. Unfortunately, or fortunately, there is no actual experience for testing. Suffice to say, the formula does approximate the results of accepted war games. So if one believes the war games (actually they are “computer simulations,” but the term “war games” adds a note of realism), he has to give some credence to the formula.
Let us see what an “analyst” can do with the formula. He can tell you that, when the expenditures on “terminal defense” are balanced with expenditures on area defense, the probability (P8) of the bombers’ penetrating the area defense is given by
Ps = C / pWT
where C is the cost of an interceptor and its associated control environment, p has to do with the effectiveness of the area bomber defense as described above, W is the number of short-range attack missiles (SRAM’s) carried by the bomber, and T is the cost of negating a SRAM with terminal defense. The idea here is that “area defense” and “terminal defense” are both operating at the same “marginal return”—a term that has been in the vernacular of the economists for a long time. An analyst can derive additional simple truths: that, for every additional attacking bomber, the amount the defense must allocate to terminal defenses is an amount of money equal to C/p, and the allocation to area defenses is an amount of money
equal to C / p ln pWT / C.
(This is for the case where the defender is “balancing” his expenditures so as to negate the bomber at least cost by either shooting it down with area defense or negating with terminal defense the SRAM’s launched from the bombers that survive the area defenses.)
These two examples are intended to show what is meant by “collapsing” problems. This is the real payoff in one’s effort to provide visibility and illumination. This is where the analyst succeeds and the calculator fails. Notice that the word “analyst” was used rather than mathematician. Granted that a knowledge of differential calculus is useful if not necessary. But the big task is figuring out how to set the problem up so as to have something to differentiate in the first place. Mathematicians who can manipulate the formulas in a mechanical sense are as easy to come by as the calculators; but analysts aren’t. As a matter of fact I have come to the conclusion that the makings of a good analyst are more apt to be found in a lawyer who has a smattering of mathematics than in a mathematician who is a calculator rather than a thinker. Since lawyers are not particularly well schooled in calculating, they are forced to think and reason, and this is a very good thing.
The best education for an analyst is in the school of doing. This presupposes that the person involved is alert, curious, and eager to work. Further, he should feel somewhat at home with integral and differential calculus. But, given this background, the best way to become an analyst—if there is indeed such a type as distinct from other people—is to work on problems. Guidance and assistance from someone who has been through similar studies are quite helpful. But, ultimately, good studies are produced by hard and earnest work. They are the result of going over and over and over and over some problem with a view to reducing and collapsing it on the one hand and providing illumination and visibility on the other.
Probably the best procedure for a student who is preparing to embark upon studies called analysis is to review carefully the analytical techniques that were used to good effect in analyses already accomplished. By luck, one of these techniques might apply to the problem at hand. In my view the courses on analysis now being conducted at various places have far too much emphasis on statistical theory and the like, along with instruction in mathematical manipulations, and too little on case histories. The emphasis should be on how to think about problems so as to simplify them. I know of no better way to do this than to review what has been demonstrated in the past. Unfortunately, the textbook I am talking about has yet to be written, but a noble beginning would be for someone to publish a compendium demonstrating the better techniques that have been used to date.
Too many times the results of what was potentially a good analysis go down the drain because of poor presentation. This goes for both oral and written efforts. I have a theory that each listener or reader has a threshold for “naggers.” “Naggers” are things that he does not understand. When his threshold is exceeded, he quits listening or reading. The “naggers” can come in several forms, all of them used by presenters at some time or another, for one reason or another. A common practice is to fail to delineate clearly how a particular curve was derived. Now, the ingredients for deriving the curve are almost always contained (submerged) somewhere in the prose —a little clue here and another clue there—and a determined sleuth can finally piece the whole thing together. The trouble is that most readers are not that determined, and they give up. The credibility of a curve will not be established with those who count unless they can reproduce, at least in concept, the points on the curve. Without establishing credibility, one has little or no chance of making any of the points he may have had in mind. The day has long since passed when one could get away with “Since the bar for System A is longer than the bar for System B, we should buy System A.” The fact that the bar for one system is longer is of little import unless the decision-maker “believes” the analysis; and this belief can only be established by the clearest exposition. Sometimes the lack of clear exposition is purposeful in order to submerge some awkward or shaky input. To think that such a practice can possibly payoff borders on idiocy.
Other times the lack of clear exposition in an oral presentation stems simply from a well—known and prosaic disease: the briefer doesn’t know his subject. The curves were provided to him by someone else. He thought he understood them, and ostensibly did, until someone asked a question that wasn’t in the script. Oral presentations also suffer many times from a plethora of charts and a paucity of message. The best illumination stems from a few charts that are well explained.
What are the fixes for these ills? The fixes can be summed up in one word: discipline. Air Force personnel should apply the same rigid discipline to analysis that they do to flying an airplane. The accident rate for analysis is quite high. However, these “accidents” are for the most part not as dramatic and personal as aircraft accidents, and consequently there is no concerted campaign to reduce the rate.
If nothing else, poor analysis efforts reflect adversely upon our professional image. But how do you apply discipline? You go over and over and over each bit of logic and each calculation. By “you,” I mean you. If it is your study, then you should be able to reproduce, when called upon, any number in the study in a reasonable time without too much fumbling. You only really understand something after you have made the calculations yourself. If the study is so complex that you feel you simply can’t master the calculations, then one of two things (or both) is wrong: either the study is too complex or you are a poor analyst and should take up another pursuit. A rule of thumb regarding simplicity is that “even generals must understand it.” Many of the top people in the Department of Defense make it a point to understand important analyses in considerable detail. Rather awkward situations are created when the analyst and intervening echelons do not do likewise in advance.
After all, simplicity, in the interest of illumination, is what we are after. If you are asked to explain something and in lieu of a direct answer you start out with “Well, it’s rather complicated,” you are losing altitude fast. Ambiguous answers to oral questions have the same fleeting value as the air above you and the runway behind you.
So the first part of discipline is to keep it simple. The second part of discipline is to explain fully. In a written text, for each graph or table, one should have a facing page (or pages) with three sections: (1) a section that describes the purpose of the graph; (2) a section that describes the basis for computations, including all values for inputs and assumptions; and (3) a section that tells the reader what message is to be gotten out of the graph or table. Now, if you find it trying and difficult to write section 3, then you might give serious consideration to omitting the graph in the first place. Exercising this discipline in the written report also helps any oral presentation, particularly if the writer is also the presenter—and he should be. At the risk of being repetitious: you learn the details only by getting your hands dirty in the actual derivation of the report. A deep-tanned colonel with a resonant voice is no substitute for a pale-skinned major who has not had much sunshine because he is the one who has been doing the dirty work.
In closing I would like to go back to the matter of whether or not to include conclusions and recommendations in analyses. Decision-makers, with good reason, often feel that their responsibilities are being eroded in some fashion or another by the analysts. This concern sometimes takes the form of, “These analysis studies will never take the place of military judgment.” The rejoinder by the analyst to this charge should be, “Sir, my hope is that a decision by you, based on your excellent judgment aided by my elegant analysis, will be better than a decision based on your judgment alone. I can hardly believe the aid afforded by my analysis could be counterproductive.” But to be confident that analysis is not “counterproductive” is sometimes most difficult, particularly if conclusions and recommendations are included. Besides that, the analyst can’t make his statement in the first place unless he has been careful not to pre-empt the decision-maker.
As stated earlier, the prime purpose of an analysis has to do with providing illumination on the utility of a particular weapon system or piece of equipment. This illumination provides the basis for the Air Force proposing (or not proposing) that the system should be developed and procured; that is, its utility is such that the Department of Defense should (or should not) spend money and resources to acquire it. Said another way, analysis provides a basis for decision on whether or not certain equipment is to be introduced into the operational forces. Action and decision-making center around proposals. To paraphrase Shakespeare slightly, “The proposal’s the thing wherewith we’ll tap the coffers of the king.” The central question is whether or not the proposal is worthwhile. Analysis, hopefully, provides added insight on this all-important question.
Hq
Air Force Systems Command
Major General Glenn A. Kent (M.S., California Institute of
Technology; M.S.,
Disclaimer
The conclusions and opinions expressed in this
document are those of the author cultivated in the freedom of expression,
academic environment of
Home Page | Feedback? Email the Editor