Document created: 24 August 04
Air University Review, September-October 1970

Expanding the Scope of
Performance Evaluation

Colonel Vincent J. Klaus, U.S. Army

A new Air Force Regulation 178-4 has recently been published that establishes policy for output measurement/management indicator systems.¹ The regulation, which implements a Department of Defense directive, merits the attention of managers of all services. Its implementation should promote the spread of urgently needed output information for the Air Staff and the Department of Defense.

Developing the Defense budget requires the tailoring of worldwide military commitments to the parameters of available resources. The Congress is faced with an even greater responsibility. That body must weigh each military request against those developed by the other federal departments to insure that the nation receives maximum benefit for its tax dollars. The most popular form of analysis today is to divide the budget into component parts in order to permit a closer examination of the resources required.

Such subdivisions, however, rarely provide better visibility into the program objectives. One of the results of the new regulation will be to assist budget review at all levels up to and including the Congress, by providing better information on the output side of the equation. If all the implications of a trade-off decision are to be appreciated, detailed explanations of costs must be matched by comparable explanations of objectives or output.

The Department of Defense, in general, enjoys an excellent flow of input information. By all indications it is the output side of the input/output equation that urgently requires improvement. To support this statement a news release dated June 26, 1969, might be offered as evidence. It originated with the Congressional Subcommittee on Economy in Government, of which Senator William Proxmire is Chairman, and contains the following reference to output information:

. . . with Federal spending accounting for nearly 20 percent of total national production, we must focus on major outputs and objectives. We must begin asking what our expenditures will accomplish and whether or not the accomplishment is worth the cost.

For most of the appropriation decisions which the Congress makes, there is little or no indication of who gains and who loses and even less knowledge of whether or not the expenditure program is producing more than it is costing. A budgetary process which operates with so little information and in such a haphazard fashion is not a rational system.

Other evidence of the need for improved output information can be found in modern accounting literature. In a recent article Dr. Lennis M. Knighton² first establishes his purpose: 

. . . One of the most fundamental and important concepts of accounting is the “matching” concept, for it is precisely from this concept that the whole system and practice of accrual accounting derives its justification and importance. In commercial accounting this concept is generally interpreted to require a matching of revenues with the expenses incurred to produce the revenues.

Dr. Knighton then clearly describes the problem in government accounting:

. . . Determining the benefits to be matched with current expenses is undoubtedly the most difficult problem in governmental accounting; for the ideal measures of benefits are not the services or products produced but the results of those services or products.

In the same article he provides an excellent example of the type of output information he is recommending:

. . . For example, effectiveness in highway construction would be difficult to assess with anything other than engineering measures and statistics; and the effectiveness of highway programs must necessarily include statistics reflecting changes in the number of accidents, dollar losses of property, personal injuries, deaths, average speed of vehicle movement, number of vehicle miles traveled, etc. Likewise it is impossible to evaluate the effectiveness of health programs in the absence of medical statistics and measures, or of welfare programs in the absence of sociological, psychological, and economic data. Certainly not all of this information can or ought to be accumulated through the accounting information system, but it all must be related to the information that is available (or should be) through the accounting process. Some information may be gathered in a systematic way through other processes; some may be available only through special surveys or statistical testing; and some may be available only as the product of an expert opinion or professional judgment, as is often the case in health matters, for instance. But until and unless such standards are developed and such information is available, it will be impossible either to articulate objectives, to exercise meaningful control over operations, to measure the results of operations, or to evaluate the effectiveness of performance.

This type of information, its collection and utilization, is the subject matter of Air Force Regulation 178-4. The task of collecting this information is a formidable one and will require all the assistance the new regulation can offer. An example is the difficulty encountered in measuring the output of Defense organizations. They perform a multitude of tasks, and each type of performance has its peculiar output. The difficulty in measuring the output of a unit is indicated by some of the complicating factors:

―An organization’s output can be measured by actual performance and by the maintenance of a capability to perform.
―An organization may have only a few major outputs but several “hidden outputs.”
―Current output measures are nonstandard, vary greatly in quality and usefulness from user to user, and have no established basis for comparisons.

An examination of the impact of the new Air Force Regulation on these factors confirms the opinion that a good deal of progress toward improvement may result from that regulation.

The distinction between actual performance and maintenance of a capability to perform was established by DOD Directive 7000.4, “Output Measurement Systems,” 13 April 1968, and is also contained in the new AF Regulation. Accordingly the first complicating factor is more widely accepted than the others. Perhaps a useful example may be found in the major outputs of a supply depot and of a Minuteman squadron. Most of the outputs of the former are tangible, quantifiable, and easily adaptable to managerial manipulations such as quality control, trend analysis, cost analysis, and many more sophisticated analytical methods. The major output of a Minuteman squadron is the maintenance of a capability to perform, within parameters of time and effectiveness. One instance of measuring the capability to perform would seem to be the Joint Staff’s FORSTAT ratings of C-1, C-2, C-3, or C-4. Superficially, these ratings would appear to reflect a unit’s capability to perform, but a more detailed analysis by air staffs will be required before they can be accepted.

The new regulation provides a workable set of criteria for evaluation of existing measures and development of new measures. Paragraph 5 states that output measures

. . . must therefore possess the following characteristics:
a. A plausible relationship must exist between activity as measured by the indicator and the resources consumed;
b. The output measure must not be synonymous with resources consumed;
c. The output measure must be meaningful to managers and stated in simple, direct terms;
d. The output measure must be programmable and available from bonafide data sources;
e. Output measures selected must permit aggregation of data to program element level with reliability and meaningfulness.

The second complicating factor, “hidden outputs,” does not mean literally hidden, but rather obscured, overshadowed by the more obvious outputs of a unit. These obscured outputs may consume a sizable portion of an organization’s resources but are omitted or, at best, rarely included as output information flowing from the organization. Their existence explains many of the inconsistencies that appear when cost is matched with output over a period of time. Increases in the visible efficiency of an organization may in fact be due to increased productivity, reduction of waste, etc., but may also be due to a reallocation of resources. When a unit is pressed by higher managerial staffs to decrease its costs of production in relation to its output, it may accomplish this by diverting hidden or unreported outputs to the major output under analysis. As an example, a military unit can improve the ratio of cost to output of “short tons shipped” by reallocating resources from hidden outputs as well as by increased efficiency. Upon careful analysis it may be found that maintenance, or material surveillance, or some other hidden output was cut back or held in abeyance until the pressure was off.

Such diversion of resources may well be justified by the emergency and therefore may represent sound managerial action to insure accomplishment of an important task. The flaw exists in the fact that senior staffs, well removed from the organization, are not aware of the diversion and tend to credit the increase in output (without an increase in input) to improved efficiency or, most likely, to elimination of “fat” in the organization’s budget.

When measures developed in response to the new regulation are matched with accrued costs, these diversions should become obvious. The measures defined in paragraph 6a of the regulation, “Organizational Products,” should indicate that the increase in one product was accomplished at the expense of another. However, complete visibility does require accrued costs for the matching process. Will they be available? An article by Carl W. Tiller of the Bureau of the Budget promises that accrual accounting will be a reality in this fiscal year.

[To] the [same] extent to which this matching of accrued costs with organizational products occurs, identification and observation of hidden outputs will also be possible. Whether or not such detailed matching will occur and at what levels of command it will be used is, at this point, highly speculative. Forecasting is a hazardous pastime but certain observations can be made with a comfortable degree of assurance. Given, that accrued costs become available and that organizational products are identified, the matching process appears inevitable. Managers will, by the nature of their responsibilities, seek ratios such as cost per unit of output. This increased visibility of a unit’s performance will just as naturally lead to analysis of production and of changes in product/resource ratios thus leading to identification and observation of a much more complete list of outputs.³

This should permit identification and observation of hidden outputs in the near future.

Thus the accrued costs combined with the output measures developed as a result of the new regulation will make possible the matching concept described by Dr. Knighton.

As to the third complicating factor, that a unit’s output often changes with its assignment, Department of Defense organizations may be assigned tasks never envisioned by the staff that designed them. In adapting themselves to unusual missions, the units curtail or lose other capabilities. Dependent upon the time spent in an unusual configuration, it may take days or weeks of retraining before the unit regains all the capabilities it was designed to possess. This should not be construed as criticism of extraordinary assignments. Indeed, the flexibility of a military unit is one of its most valuable attributes. However, when such assignments are made, senior staffs must receive usable information concerning the effects of the assignment on the unit’s ability to perform all of its designated tasks. How can a staff, separated geographically from the unit, determine what capabilities the unit has or to what extent degradation will occur? It appears that three things are required: an easily referenceable listing of what the unit is expected to be able to do; a means of assessing and expressing the unit’s present capability in relation to the listing; and a means of transmitting this information.

At the time of this writing, the Air Staff has under development a Unit Capability Measurement System that promises to meet all these requirements and more. The development of output measures in response to the new regulation will provide needed groundwork for the Unit Capability Measurement System or whatever follow-on system is selected by the Air Staff.

As to the last factor listed (that the measures in use vary from user to user), in a good deal of the output information being collected within the department today enough variation exists to confuse the issue whenever output measurement is discussed. A portion of the OSD staff that has been faced with this problem for some time informally developed the classification system shown in accompanying tabulation. The system serves to orient and explain objectives and to form a base for discussion. It was perhaps the best tool at hand to use in explaining what was available and what was wanted.

With the publication of Air Force Regulation 178-4, a new spectrum of definitions becomes available, as shown in the following excerpt:

Classifications of Output Measures. The types of output described and recorded by various organizations fall at different points along a spectrum. The output of an organization may be described by:

a. Organizational Products which describe what is produced by an organization for external use or effect. For example, number of engines repaired, amount of ordnance delivered.

b. Benefits to Another Organization, i.e., products of one organization expressed in terms of benefits received by other organizations. For example, adequacy and quality of repaired engines as received by operating units, tactical assistance resulting from ordnance delivered.

c. Benefits to Society or Contributions to the National Objective which are at the top of the scale and translate organizational output into social or national goals. For example, program element output measure, unit capability, economic indicators.

NOTE: Different users describe outputs at different levels on this spectrum according to their requirements. In general, outputs at the lower end of the spectrum are more readily expressed in precise or quantified terms; those at the higher end may be difficult or wholly impracticable to describe in precise terms.

Although there are similarities, the set of classifications in use by the Air Force is the only approved and documented one. Accordingly, it provides stabilization and standardization to the classification problem. The importance of standardized categories will be most appreciated by people working to develop output measures.

The Air Force is the first service to respond formally to Department of Defense Directive 7000.4, and this response promises to make a major contribution to the general state of the art. The regulation should initiate actions that will not only improve Air Force managerial systems but also contribute, by knowledge and example, to all other services, indeed to the federal government as a whole.

Office of the Assistant Secretary of Defense

Notes

1. Air Force Regulation 178-4, “An Force Output Measurement/Management Indicator Systems,” 11 September 1969.

2. Lennis M. Knighton, “Performance Evaluation and the Matching Concept in Governmental Accounting, “The Federal Accountant,” September 1969, pp. 95-103.

3. Carl W. Tiller, “Why Accrual Accounting?” Defense Management Journal, Summer 1969, pp. 29-31.

Acknowledgment

For this article’s lead-in photograph the Review is indebted to Colonel Clifford Eugene Raisor.

Contributor

Colonel Vincent J. Klaus, USA, (M.S., Georgia Institute of Technology) is a Systems Analyst, Office of the Assistant Secretary of Defense (Comptroller). He served as an infantry company commander in the Korea War and later as a special weapons officer in Europe. A graduate of the Command and General Staff College, Colonel Klaus has been an instructor at the Army Chemical School and commander of the Army Chemical and Biological Briefing Team.

Disclaimer

The conclusions and opinions expressed in this document are those of the author cultivated in the freedom of expression, academic environment of Air University. They do not reflect the official position of the U.S. Government, Department of Defense, the United States Air Force or the Air University.