Decision Theory and Weather Forecasts A Union with Promise

An adaptation of a popular advertising slogan might be stated like this: “Weathermen try harder.” Despite the effort, better weather forecasts are difficult to get because of such chronic problems as too few weather observations and the gaps in scientific knowledge. Since these difficulties are well known and may sometimes be gleaned from the muttering of frustrated weathermen, enough has already been said about them. Instead, this article advances ideas for getting more effective weather service from present weather forecasting skill.

The proposal, which stems from elementary decision theory, is based on three interlocking procedures for weathermen to follow: express weather forecasts in probability terms; know how weather affects operations; and, through decision-making aids, use the first two steps to recommend a best choice to the decision-maker. Although each step is distinct and complex, there is no need to treat each step separately in this article. Instead, the proposal is developed through these topics: background—how weather forecasts are presently stated; decision-making under risk—what it means, two examples, and problems of application; and, finally, possible changes in weather services—implications for the Air Force.

Linear programming and the theory of games are well-known decision-making aids. Perhaps less well known is the work of some meteorologists in using decision theory to improve the usefulness of weather forecasts. These meteorologists think that better use of present weather forecasting skill will enhance the reliability of weather service and contribute to sound operational decisions. This idea can be appreciated better by first reviewing present practices.

At present, Air Force weather forecasters generally provide categorical statements of predicted weather, e.g., “Rain this afternoon, clearing tonight.” The forecaster often keeps this additional thought to himself: “Although there are several ways the atmosphere may behave, I’ll forecast this one because it has better than a 50 percent chance of being correct.” This practice can deprive the customer of a better understanding of what may happen. For example, whether the probability of rain ending is 90 or 51 percent could make a great deal of difference to some rain-sensitive operations. But such information is known only when volunteered by the forecaster or when drawn out under skillful questioning by the operator.

On the other hand, operators may not be fully aware of how weather affects their operation. Such clichés as “Flying safety is paramount” do not adequately define the need for weather service. Unless the operator knows how weather affects his operation, he is unable to profit fully from available weather service; i.e., even if provided a complete statement of weather probabilities, some operators may not know enough about their operation to make the best decision.

This brief account of today’s practices is a starting point for understanding how improvements might be made. Emphasis now shifts to the ideas being advocated.

This heading clearly means that the operator acts on the basis of weather forecasts that are not 100 percent reliable and hence there is risk in making a decision based on an imperfect forecast. The central theme, then, is to minimize the risk to the operator. To illustrate, consider the strategies and possible outcomes facing a golfing “duffer.” This example, which uses only expected value, is deliberately simplified. The scene: first tee and narrow fairway, bounded on the right by a deep ditch and a dense rough on the left. Our frustrated, right-handed duffer frequently imparts a large “slice” to his drive. His problem is clear—Where should he aim his drive? The problem is expressed in the accompanying table.

Possible Strategies		Possible Outcomes
	Ball hooks (0.05)	Ball straight (0.20)	Ball slices (0.75)
Aim left	3	6	8
Aim straight	6	10	1
Abandon game	This strategy is unacceptable.

The probability of each outcome is in parentheses; these probabilities are analogous to a probabilistic weather forecast because a probability is assigned to each possible outcome. Arbitrary values of satisfaction or utility are entered in the table. These values are subjective but logical. For example, a drive that is aimed straight and goes straight is rated as 10 because it will remain in the fairway and should have good distance. A drive that is aimed left and slices is awarded 8 points because the ball should stop on the fairway but with less distance than a straight drive. Similar considerations about distance, penalty strokes, etc., were used to get the remaining values. The expected value, in arbitrary units, is 7.35 for aiming left and 3.05 for aiming straight. Obviously, the duffer’s best strategy is to aim left if he uses expected value as his guide. But he still aims his drive under risk, for he is not completely sure of the direction of his drive.

In this example either a categorical or a probabilistic prediction would prompt the duffer to aim left. But suppose on the eighteenth tee the match may be won by a long, straight drive. If the duffer relies solely on a categorical outlook, he may lose his chance to win. On the other hand, he may assign such a high value to a straight drive that his best strategy is changed to aiming straight even though he still has only a 20 percent chance of hitting a straight drive. Hopefully, this example shows what decision-making under risk is about:

(a) Choose the best strategy for the situation at hand by considering the consequences and the probability of the consequences.

(b) Realize that the next decision may be incorrect although in the long run the decision-maker should maximize his gains or minimize his losses.

Although weather-sensitive operational problems may be approached in a similar manner, the methods are more refined and complex. As expected, situations involving several strategies and outcomes are harder to solve and, apparently, less well developed than the case of only two strategies and two outcomes. ^{1, 2, 3}Yet these methods still offer operators a means for choosing optimum strategies. Moreover, further developments in technique may make solutions to the multiple strategy-outcome problem as easy and useful as the two-strategy-outcome case. Available methods of the latter, simpler case can indicate whether available weather forecasting skill can help the operator or whether an alternative decision, such as always protecting against a hazardous event, is better.⁴More important, the methods can show how existing skill may be used to improve the value of weather forecasts. And, finally, analysis can show how the operator would profit from perfect weather forecasts; this information can guide meteorologists to study those problems offering the greatest potential gain to the operator.

In a hypothetical example of a two-strategy-outcome problem, the operator’s problem is whether or not to protect aircraft from severe weather. The example supports a key point: if the operator takes protective action only when the probability of an event exceeds a critical value, the value of the forecasts changes from unprofitable to profitable.

The setting is a small air base harboring nine aircraft. A safe refuge base may be reached in 30 minutes’ flying time. Any required readiness posture may be maintained at the safe-haven base. The flying training program is on schedule, so there is no intangible benefit in evacuating the aircraft. These additional assumptions are also needed:

(a) Experience shows that a particular degree of severe weather—say gusts to 65K and one-inch hail—will inflict damage of 0.1 percent of the cost of a single aircraft on one-ninth of the unprotected aircraft. If each aircraft is worth $3,844,000, damage of $3844 may be expected.

(b) Since the climatological frequency of severe weather is 4 percent, assume that forecasts of nonoccurrence and occurrence are issued with a frequency of 96 and 4 percent, respectively. Three views of forecaster skill are noted: forecasts of nonoccurrence are 98.5 percent correct; forecasts of occurrence are 62.5 percent accurate; and overall accuracy is 97 percent. Overall accuracy is a measure of the number of correct forecasts—of both non-occurrence and occurrence—compared to the number issued. The assumptions about forecast skill are fairly realistic but are deliberately and slightly biased to show good forecast skill.

(c) The cost of evacuating and returning the nine aircraft is $3141, based on per diem costs of $14 per aircrew and flying-hour costs of $335 per aircraft. A likely damage loss of $3844 has already been noted. A comparison of these costs indicates that a decision to evacuate should be taken only when the probability of occurrence equals or exceeds 82 percent.⁵The present typical practice is for the forecaster to issue a warning if he believes there is more than a 50 percent chance of occurrence. But this practice requires the forecaster to be 98.6 percent accurate if he is to provide economically meaningful service to the operator.⁶Although forecast accuracy is 97 percent in this example, available skill is inadequate to provide economically meaningful service.

One final assumption is needed to complete this example: if the forecaster issues warnings only when the probability of occurrence appears greater than 82 percent, forecasts of occurrence and nonoccurrence will be issued with a frequency of 1.2 and 98.8 percent, respectively. Forecasts of nonoccurrence are 97.2 percent correct; forecasts of occurrence are 91.7 percent accurate. Overall accuracy remains constant at 97 percent. Using these assumptions and available techniques, four possible annual costs to the operator may be derived, as follows:⁷

(a) If perfect forecasts were available, the operator would pay about $45,800 to evacuate the aircraft only when necessary.

(b) If the operator evacuated only when the probability of occurrence exceeded 82 percent, annual costs would be approximately $54,500. The additional cost over that of acting on perfect forecasts is due to two factors: aircraft are sometimes evacuated unnecessarily, and aircraft may sustain damages from unpredicted occurrences of severe weather.

(d) If the operator evacuated whenever the probability of severe weather exceeded 50 percent, annual costs would be approximately $66,680.

A hypothetical example involving severe weather warning has been developed. Analysis revealed the costs to the operator for several situations, ranging from perfect knowledge of future weather to present-day practices and limitations. The example indicates that the operator would save about $1650 per year if he acted only when the probability of an important weather event exceeded a value critical to his operation. Since each real-life situation depends on climatology, forecasting skill, and operational factors that will differ significantly from this example, the reader is cautioned not to apply this example to his operation.

Despite the promise of a better decision-making rationale, as advanced by this article, there are several important obstacles:

First, some operators may refuse to accept weather forecasts worded in probabilistic terms; they may insist that the forecaster say “It will rain” or “It won’t rain.” Unfortunately, this attitude makes the forecaster—not the operator—the decision-maker. Yet the same operator may willingly accept climatological data worded in probabilistic terms. Consider the loss of information if an operator is told that the climatological outlook of a site is VFR (visual flight rules) while the detailed facts are omitted: VFR—65 percent; IFR (instrument flight rules)—33percent; below minimums—2 percent.

Second, assignment of awards and penalties for various strategies and outcomes can be troublesome. Recall the duffer. It is not too hard to assign dollar values to the possible outcome if the duffer has a wager on the game. If the duffer seeks only personal satisfaction from a well-executed shot, utility values may be assigned to indicate the personal value of a good drive. ⁸ The problem arises when the golfer tries to mix both monetary and utility values. Unless the relative importance of these two values can be defined, there may be no way to approach the problem. This difficulty would limit application of the proposed techniques. For example, a commander maybe concerned about the costs of evacuating his aircraft because of an advancing typhoon. His problem could be compounded by operational aspects, such as the ability to maintain an assigned alert in support of a contingency plan. In such a case, application of rational decision-making aids would be extremely difficult, if not impossible. But for combat situations where concern for effectiveness dwarfs monetary considerations, numerical values with meaning for the commander could readily be substituted for cost values.

Third, neither Air Weather Service nor the units it serves have adequate experience in the methods. Moreover, the assignment of probabilities to all possible weather categories is a significant obstacle.

Fourth, the operator may have difficulty in sorting out the many facets of his operation, and unless he can do so, these techniques offer only limited help. It should be clear, too, that the successful application of the techniques requires a complete, frank dialogue between the meteorologist and the operator. Each has well-defined responsibilities.

A number of ideas have been presented. Rather than probe these in greater depth, I shall consider an application of these concepts in the next section.

· Some customary forecasts might be withdrawn after the required evaluation. This evaluation could conceivably evolve to a process similar to that of validating manpower spaces; if the requisite forecast skill is unavailable, the validator would suggest an optimum strategy to the operator in lieu of routinely provided forecasts. Generally speaking, the remaining forecasts would result in significant savings or tangible benefits to weather-sensitive operations.

· In those cases where the techniques are not applicable, the customary service would still be provided. “Customary service” means weather forecasts without decision-making aids. The definition is also amplified to mean use of probabilistic forecasts whenever the operator is agreeable.

· Meteorologists would have an additional means of identifying those problems for which better forecasting skill would be most helpful to the operator.

Two basic ideas have been advanced. First, the present-day use of weather forecasts may not always result in advantages to the operator. Indeed, use of some forecasts may hinder the operation. Second, available “off-the-shelf” techniques can enhance the application of weather forecasts to many operational problems, resulting in economies and greater effectiveness.

This hopeful outlook will require considerable effort to be achieved, yet the overall result would be as meaningful as a significant increase in weather forecasting skill. The Air Force could realize two benefits for some weather-sensitive operations: increased combat effectiveness and lower operational costs. Although the Air Weather Service is interested in the possibilities, means of applying these techniques must be evolved. This will take time. Meanwhile, weathermen will try harder.

1. L. E. Borgman, “Weather Forecasting Profitability from a Client’s Viewpoint,” Bulletin of the American Meteorological Society, Vol. 44 No. 7 (July 1960), pp. 347-56, esp. p. 353.

2. T. A. Gleeson, “A Prediction and Decision Method for Applied Meteorology, Based Partly on the Theory of Games,” Journal of Meteorology, Vol. 17, No. 2 (April 1960), pp. 116-21, esp. p. 116.

3. E. E. Epstein, “A Bayesian Approach to Decision Making in Applied Meteorology,” Journal of Applied Meteorology, Vol. 1, No. 2 (June1962), pp. 169-77, esp. p. 169.

5. J. C. Thompson, “On the Operational Deficiencies in Categorical Weather Forecasts, ” Bulletin of the American Meteorological Society, Vol. 33, No. 6 (June 1952), pp. 223-26, esp. p. 223.

7. J. C. Thompson, “Economic Gains from Scientific Advances and Operational Improvement in Meteorological Prediction,” Journal of Applied Meteorology, Vol. 1, No. 1 (March 1962), pp. 13-17, esp. p. 14.

Major Frank P. Scruggs, Jr. (M. S., Florida State University; M. S., George Washington University) is Forecasting Evaluation Officer, Hq Air Weather Service, Military Airlift Command, Scott AFB, Illinois. He enlisted in the Air Force in 1945 and served as a weather observer in Germany and as weather forecaster at Godman Air Field, Kentucky, at France Field, C. Z., at Albrook AFB, C. Z., and at Patrick AFB, Florida. Since his commissioning in 1954, he has served as Weather Officer, Patrick AFB; as commander of weather detachment, Aberdeen Proving Ground, Maryland; as Commander of Weather Collection Squadron (Provisional), Hickman AFB, Hawaii; as Weather Officer at Eglin AFB, Florida, and Tachikawa AB, Japan; and as Forecasting Services Officer, Hq 1 Weather Wing, Fuchu AS, Japan.

The conclusions and opinions expressed in this document are those of the author cultivated in the freedom of expression, academic environment of Air University. They do not reflect the official position of the U.S. Government, Department of Defense, the United States Air Force or the Air University.